In drug production, defect detection is usually performed manually, often leading to low efficiency and accuracy. In this paper, we propose a system that detects defects using computer vision and then grasps defective drugs using a robotic manipulator. The proposed system adopts an eye-to-hand architecture with the camera fixed above the conveyor belt and uses a two-point correction method to convert pixel coordinates in the image into real-world coordinates to determine the target position. This system combines the Taguchi-based You Only Look Once (TYOLO) model for object detection and the SORT algorithm to track the location of the drugs. When defects are detected, the manipulator removes the defective drugs. Experimental results indicate that the mean accuracy, precision, recall, and F1-score of the TYOLO model reached 98.67%, 98%, 98%, and 98%, respectively. The success rates of static and dynamic grasping tasks were 98% and 93%, respectively, showing that the proposed system performs well in terms of both accuracy and stability.