Digital video recording (DVR) systems are widely used in our daily life because of cost-down of capturing devices. Developing an automatic and intelligent system to detect, track, recognize, and analyze moving objects could save human power in monitoring centers. In this study, the color features of an employee’s uniform were extracted to identify the entrance legality in a restricted area of an open space. First of all, a background subtraction technique was used to detect moving objects in image sequences. Three key object features, the position, the size and the color, were extracted to track the detected entrants. After that, the body of an entrant was segmented into three parts for locating the region of interest (ROI) using a watershed transform. Dominant color features extracted from the ROI were classified for preventing the illegal entrance. Some experiments were conducted to show the feasibility and validity of the proposed system. In the final part of the paper, conclusions are drawn and future work is suggested.