Aiming at the situation that the quality inspection of bottled liquor packaging has low detection accuracy and high overlap of small targets, resulting in false detection and missed detection, this paper proposes a target detection optimization algorithm based on RetinaNet, which mainly uses the defect dataset of liquor bottle caps for detection. In this method, the network backbone is replaced by Swin Transformer, which contains window attention mechanism calculation, which effectively improves the accuracy of cap defect detection, reduces complexity, and saves calculation. In the neck stage, neural architecture is used to search for FPN instead of FPN, automatic architecture search is used to select the best feature fusion layer to provide a higher quality model for subsequent detection, and finally Soft-NMS is used to reduce the confidence of the detection frame and retain a certain real frame, which effectively prevents the leakage of cap defects caused by too close or overlapping. Experiments show that the improved algorithm in this paper can accurately identify various bottle cap defects, and the detection accuracy reaches 93.53% in the liquor bottle cap defect dataset, which is 8.02% higher than the original network.