The kernel correlation filtering algorithm (KCF) is robust to target rotation and partial occlusion. However, it can not meet the requirements of realtime tracking in the actual project. Based on DSP embedded platform, this paper proposes an implementation scheme of real time target tracking system. Firstly, based on the hardware resources of TMS320C6678 processor, the algorithm is optimized for twodimensional Fast Fourier Transform(FFT2D). Secondly, optimize the KCF algorithm by C languagelevel code, including loop expansion, the use of inline functions, the use of const keywords and so on. The results show that when the input image’s size is 640×480 pixels and the tracking target’s size is 32×64 pixels,the size of bomen is 64×128. It meets the tracking effect, meanwhile the speed up to 25 ms per frame, which meets the realtime requirements.