According to a literature review, breast arterial calcification (BAC) in mammograms can be used to predict the risk of cardiovascular disease, including coronary artery disease, atherosclerotic cardiovascular disease, and arrhythmia. This study applied a deep Q network and a state-action-reward-state-action learning algorithm combined with a deep reinforcement learning (RL) network to construct a computer-aided diagnosis system for detecting BAC in mammograms. The proposed system has an artificial agent that automatically learns the strategy and can iteratively modify the focus of attention from an initial bounding box to a smaller bounding box containing the BAC area. Then, the agent constructs a deep learning feature representation within the bounding box that is used to allow deep RL to determine the next action, such as transforming or scaling the current bounding box or triggering the end of the search process. The experimental results prove that the deep RL network with numerous training samples is significantly better than the regional growth method. The precision, recall, and F-measure of the proposed system are 0.9498, 0.9575, and 0.9536, respectively. For 50 ground truth samples, the average Intersection over Union (IoU) of the proposed system is 0.9355, minimum IoU is 0.9010, maximum IoU is 0.9591, median IoU is 0.9363, and standard deviation of the IoU is 0.0132. Thus, the proposed computer-aided diagnosis system can assist radiologists to make preliminary auxiliary judgments for detecting BAC in mammograms.