In this paper, we present a novel video coding system, where the motion estimation, residual image coding and initial/refresh frame coding are integrated using an HVS-based segmentation (region) technique. The Region-based Motion estimation Algorithm (RMA) estimates motion vectors by matching regions rather than blocks. Region-based Residual image Coding (RRC) approximates the residual contents with the regions* mean intensities. To obtain faithful initial/refresh frames while keeping the bit-rate low, a Region-based Two-pass initial frame Coding technique (RTC) is demonstrated: 0.6 bpp for SALESMAN and 0.3 bpp for CLAIRE with good image quality. In addition to encoding the Region match predicted Residual Images (RRI), the proposed residual image coding algorithm is also applied to BMA predicted residual images (BRI), and their performances are compared. It is found that the proposed RRC performs better with RMA residuals for image sequences with simpler contents such as CLAIRE; and that RRC performs better with BMA residuals for image sequences with more complex contents like SALESMAN. In any case, the proposed segmentation based residual coding technique is proven to be effective on both BMA and RMA residuals.