Aiming at the problems of low efficiency and high missed detection rate of the rail workers' cross-track safety actions supervision method, an improved human pose estimation algorithm YOLOv8n-Pose is introduced to detect and supervise the cross-track safety actions. The improvement method of YOLOv8n-Pose algorithm is to add an attention mechanism to the network and lighten the network structure, and improved the bbox loss function and the keypoint loss function of the network in order to improve the network's recognition accuracy and speed. The self-made dataset is enhanced by Gaussian filtering and ColorJitter algorithm. Genetic algorithm is used to adaptively adjust the training hyperparameters before training, and migration learning and knowledge distillation methods are used during training to improve the network training speed, recognition accuracy and generalization ability. The trained model is used to detect the images of the workers, which can successfully recognize the workers' keypoints and identify the safety actions based on the keypoints. The human keypoints recognition accuracy is 94.3% at the speed of 238.1 FPS, which verifies that the model improvement research has achieved beneficial effects and improved recognition accuracy, recognition speed and robustness of the model.