Skin color segmentation by a block histogram-based neural fuzzy network is proposed in this paper. The Hue-Saturation (HS) color model is used. Color information is represented by a block histogram in an HS space image. Several non-uniform quantization approaches on HS space are proposed to represent histogram information as accurately as possible. The neural fuzzy network used is the self-constructing neural fuzzy inference network (SONFIN). Block histogram information from images under different environments is used to train SONFIN to make the method as robust as possible. Experiments on skin color segmentation are performed to verify the performance of the proposed method. For comparison, three other segmentation methods, including principal component transformation (PCT), histogram-based skin classifier (HSC), and mixture of Gaussian classifier (MGC) are applied to the same problem. Comparisons show that the proposed approach achieves the best segmentation results. In addition, the proposed non-uniform HS partition approach also improves segmentation performance.