In the H.264/AVC coding standard, seven motion estimation modes from 4 × 4 to 16 × 16 are used to find the minimum motion compensation error for each macroblock. However, the high computation cost of the full search method in the reference software JM-9.3 makes the encoding process inefficient. Therefore, the methods of applying the SAD (sum of absolute difference), homogeneous region analysis, and edge detection are developed to determine the optimum motion estimation mode. Nevertheless, the additional computation cost of image processing still reduces the efficiency of the motion compensation process. In this paper, the spatial-temporal correlations between the current frame and reference frame are analyzed to develop a fast mode decision method in which no extra image processes are used. Furthermore, the concept of drift compensation is adopted to avoid the error accumulation phenomenon during the mode decision process. The experimental results show that the computation cost may be reduced above 60% and the average PSNR is only dropped about 0.04db.