This paper proposes a novel exemplar-based inpianting algorithm based upon the color distribution analysis. A novel priority scheme is designed to assign priority and guide the filling-in order, which is a crucial step for the performance of the inpainted results. Color distribution is proposed to determine the confidence of the target patches situated on the boundary of the missing region by measuring the variation of variances of neighboring source patches. Our experiments on the natural images and the comparisons with the representative existing approaches show that our proposed scheme can more robustly assign high priority to the structures and enables the salient structures, to which human visual system is sensitive, to be consistent with the surrounding regions.