A two-stage multi-scale generative adversarial network restoration model is proposed to solve the problem of insufficient use of image scale information and incorrect glasses structure restoration in face image restoration tasks. In the first stage, the model introduces U-Net coarse reconstruction network with perceptual loss, fuses image features at different scales with jump connections, introduces three different loss functions to improve the reconstruction ability of the generator, adopts double discriminator to consider global information and local information, and proposes a mixed-domain attention mechanism to focus on spatial and channel information of the image. In the second stage, a new feature enhancement module is constructed using the fine repair network to enhance the ability of the network to extract details and express structures. A relative discriminator is introduced to focus on the relative authenticity between the generated samples and the real samples, which improves the generation quality and training stability. The experimental results show that this method has excellent performance in all kinds of missing images, and has good performance in face restoration with glasses. Compared with other methods, the peak signal-to-noise ratio, structural similarity and perceived similarity evaluation indexes of the proposed method are improved by 3.81%, 2.65% and 0.45%, respectively.