This paper presents a system for extracting houses and buildings from high resolution aerial photographs. The problems of initial level selection, merging, grouping, and threshold selection in the well-known split-and-merge (SM) algorithm are discussed. A modified SM segwentation algorithm is thus presented to solve these problems. Some features, such as elongation, orthogonality, minimum bounding rectangle (MBR) and so on, are calculated for those segmented regions. In order to extract the regions of houses and buildings, the regions belonging to them are identified incrementally. First, residential areas, where many houses are located , are extracted from the segmented image. Next, the regions which satisfy the strictest constraints of being houses and buildings are extracted. Some other regions are further extracted by using the properties of shadow, shadow-making, direction and so on. Finally, an effective texture analysis algorithm is used to extract missing houses. Some experimental results show that the approach has very high applicability.