Distracted driving behavior is one of the key factors contributing to traffic accidents. Existing detection methods encounter challenges in recognizing typical distracted driving behaviors. Behaviors such as drinking water, smoking, and using mobile phones are often characterized by significant individual variability and occlusion in image feature recognition, which substantially increases the computational load when traditional convolutional neural networks are used to extract features. This leads to the loss of important feature information and negatively impacts recognition accuracy. To enhance the accuracy of distracted driving behavior detection and reduce background interference, this paper improves the YOLOv5 detection model by incorporating a CA attention mechanism for model optimization, extracting distracted features using regional feature extraction methods, applying a lightweight network structure by removing the large-scale detection head, and refining the loss function. Experiments were performed on both publicly available and self-constructed datasets, which contain 22,424 images of driver distracted behaviors, focused primarily on three critical behaviors: drinking water, smoking, and using mobile phones. The experimental results indicate that the YOLOv5-CA model achieves a distracted behavior detection accuracy of 93.1% on the experimental set, with an accuracy of 89.2% under occlusion interference, which represents a 12.4 percentage point improvement in mAP on occluded targets compared to the original YOLOv5s algorithm (76.8% mAP). Additionally, the model demonstrates faster response times while maintaining accuracy, showcasing its superiority. This study holds significant research implications for reducing traffic accidents through the effective recognition of distracted behaviors.