This paper proposes a perceptual-aware mode decision algorithm to reduce the coding time of 3D video coding (3DVC) with 2D plus depth maps. The algorithm utilizes a factor called “interesting activity” to determine the interesting region and its candidate coding modes of the depth map. The interesting activity is defined based on the characteristic of visual perception for 3D video viewers, where the characteristics are presented by the existence of MBs with object feature in foreground, or with motion information. In interesting regions, the quality of constructed depth map is maintained; in the non-interesting regions, the computation time of the depth map coding is reduced. In the final step of the algorithm, we use the third dimensional search to improve the quality of the re-constructed depth map. Furthermore, the proposed algorithm is implemented in FPGA to achieve a hardware design. Experimental results show that the entire encoding time is reduced by at least 50.6% compared with the exhaustive mode decision approach, and the structural similarity (SSIM) index is increased by at least 0.0291 and the Peak Signal to Noise Ratio (PSNR) is increased by at least 4.67dB compared with the motion vector (MV) sharing approach. The logic utilization only occupies 8% of the total resource on the Altera DE3-260 FPGA demonstration board.