In this paper, a novel two-stage noise removal algorithm to deal with salt-pepper impulse noise is proposed. In the first stage, the decision-based recursive adaptive noise- exclusive median filter is applied to remove the noise cleanly and to keep the uncor-rupted information as well as possible. In the second stage, the fuzzy decision rules in-spired by human visual system (HVS) are proposed to classify image pixels into human perception sensitive class and non-sensitive class. A neural network is proposed to com-pensate the sensitive regions for image quality enhancement. According to the experi-mental results, the proposed method is superior to conventional methods in perceptual image quality as well as the clarity and the smoothness in edge regions of the resultant images.