One-third of the world population suffers from diseases related to gastrointestinal (GI) tract. Capsule endoscopy (CE) is a non-sedative, hygienic, non-invasive and patientfriendly and particularly child-friendly technology to scan the entire GI tract. How- ever, CE generates nearly 60000 images, which make the diagnosis process time consuming and tiresome for physicians. Also, the diagnosis is highly subjective and varies from person to person. Thus, a computer-aided diagnosis (CAD) system is a must. This study addresses a multi-class medical image analysis problem using image processing and machine learning techniques. It presents a CAD system based on the hybrid confluence of transfer learning and conventional machine learning technique for automatic abnormality detection in the GI tract. The system performs with an accuracy of 96.89%. The rigorous performance evaluation shows that the system is capable of fast and accurate diagnosis of GI tract abnormalities. Such a system can be beneficial to physicians and with the advancement of smart devices and IoT, such a system can prove to be a handy remote diagnosis tool for geographically distant locations where an expert of the subject may not be available.