Currently, deep learning-based research in fruit recognition primarily involves the extraction of characteristics from individual static images of agricultural products captured through static photography. However, in real-world grading scenarios on assembly lines, dynamic image sequences are captured, posing a challenge for these methods to effectively extract information from such sequences. To address this issue, this paper introduces a dynamic classification and recognition approach for Wan Zhou tangerines in multi-direc-tional visual environments, leveraging Region Proposal Networks (RPN). Specifically, this approach employs a multi-directional Wan Zhou tangerine image acquisition system comprising three CCD cameras. The system collects three different images of Wan Zhou tangerines, and their color characteristics are fused to extract essential information. Fur-thermore, a custom RPN feature extraction network is developed by optimizing the net-work structure based on ResNets. This network helps in discriminatively grading Wan Zhou tangerines. The results demonstrate the effectiveness of this method, with an average accuracy rate of 93% for tangerine color grading and the hue mean between the grades is 15. This suggests that the grading approach, which utilizes the mean chromaticity value, is not only feasible but also applicable in practical production scenarios.