Thousands of people are killed or injured in the vehicle accidents which are mostly caused by the carelessness of humans. Therefore, if there exists a warning system which can notify the driver of the approaching vehicles and the lane departure, the terrible tragedies can be greatly avoided. Moreover, if the driving status, such as the situation around the vehicle and the drivers’ behaviors, can be recorded in videos, not only in texts, the police officers can reconstruct the scene easily. Furthermore, the increasing numbers of stolen cars is an important issue so the protection of the vehicles is essential. In this paper, a real-time DSP-based driving assistance and surveillance system is proposed. It can manipulate the images to detect the lane and the vehicles in both front and rear directions and the full-time driving status can also be recorded in H.264. After the car is parked, the monitor images inside the vehicle can be transmitted to users’ mobile phones. The contribution of this paper is to provide an embedded system to improve the security at all-time, and the drivers, the passengers, and the vehicles can be much safer. The proposed system has been successfully implemented and fully tested by the real road environment in Taiwan so the stability and the reliability can be ensured.