To achieve highly accurate infringement retrieval for reference images in digital publications, an improved SIFT (scale invariant feature transformation) algorithm for image retrieval has been proposed in this paper. To increase the reliability of featurepoints matching and image retrieval, the neighborhood color features of SIFT featurepoints are innovatively added to the image feature extraction process and a doublematching process is designed in our paper. In this paper, our improved algorithm is divided into three steps to describe: double scale space generation based on grayscale space and HSV color space respectively, new feature descriptors generation by combing traditional descriptors of featurepoints and color feature vectors around feature points, featurepoints searching and double matching. Experimental results show that, compared with the original SIFT algorithm, our improved algorithm has higher retrieval accuracy, and at the same time, the retrieval speed is not significantly decreased.