Traditional engineering drawings are widely used but not easily digitally accessible or searchable. This paper presents a novel method for digital recognition of engineering drawings understanding. We investigated two tasks that determine the performance and accuracy of a recognition method: drawing classification and character sequence recognition. Engineering drawings consist of three types, and each type contains different geometric features. First, we propose a new method combining random sample consensus and geometric features to address the classification problem. The classification error of this method is less than 5%, and we designed a strategy that enables users to correct misclassifications. After precise classification of drawings, the feature information extractor can be applied effectively. Second, we use an end-to-end neural network combining the convolutional neural network (CNN) and recurrent neural network to recognize sequence labels. In contrast to traditional character recognition methods such as those that use support vector machine and CNN technology, the proposed end-to-end neural network architecture integrates character segmentation, feature extraction, and character recognition. The performance of this character recognition method on real-world engineering drawings was shown to be robust and competitive.