According to a large number of users in a given image annotation based on image is segmented into a series of single semantic integrity area, and at the same time to the area to realize the semantic annotation problem, proposes a segmentation algorithm of semantic image two clustering based on weakly supervised learning. The combination of spectral clustering and discriminant clustering is used to guide the discriminant clustering and the latent data structure of learning features. The proposed method can make full use of the regional context information, select discriminative features for each class, and output robust multi class classifier. The effectiveness of the proposed method is demonstrated by experiments on a common data set.