In this paper, we propose a new scheme that combines the semi-continuous hidden Markov model (SCHMM) and modular neural networks (MNN) to recognize isolated Mandarin syllables. The SCHMM formulation has been proven successful in modeling the temporal arrangement of speech signals, and MNN is also suitable for performing static pattern classification. In the scheme described here, SCHMM outputs establish the sequence of observation vectors to be inputs of the MNN. Experimental results show that by combining both the discriminative power of MNN and the capability of modeling the temporal variations of an SCHMM into a hybrid model, speech recognition performance is significantly improved.