In the process of infrared small object tracking, the tracking accuracy and real-time performance are poor. Therefore, the algorithm of infrared over-sampling scanning image small object tracking based on deep learning is proposed. Firstly, the infrared oversampling scanning image model is constructed, used to filter the background and external clutter, then add the design feature fusion module and area selection module to improve the twin network, generate the fusion feature map input target area, and improve the feature representation ability and tracking accuracy through classification and regression calculation. Finally, the loss function is established to train the twin network and output the small target tracking results of infrared oversampled scanning images. The experimental results show that the proposed algorithm can be up to 35dB, the proposed algorithm has high regional overlap rate, high tracking accuracy, strong real-time algorithm, the frame rate reaches more than 200 fps, and the overall tracking effect is good.