Feature extraction plays an important role in facial expression recognition. Canonical correlation analysis (CCA), which studies the correlation between two random vectors, is a major linear feature extraction method based on feature fusion. Recent studies have shown that facial expression images often reside on a latent nonlinear manifold. However, either CCA or its kernel version KCCA, which is globally linear or nonlinear, cannot effectively utilize the local structure information to discover the low-dimensional manifold embedded in the original data. Inspired by the successful application of spectral graph theory in classification, we proposed spectral supervised canonical correlation analysis (SSCCA) to overcome the shortcomings of CCA and KCCA. In SSCCA, we construct an affinity matrix, which incorporates both the class information and local structure information of the data points, as the supervised matrix. The spectral feature of covariance matrices is used to extract a new combined feature with more discriminative information, and it can reveal the nonlinear manifold structure of the data. Furthermore, we proposed a unified framework for CCA to offer an effective methodology for nonempirical structural comparison of different forms of CCA as well as providing a way to extend the CCA algorithm. The correlation feature extraction power is then proposed to evaluate the effectiveness of our method. Experimental results on two facial expression databases validate the effectiveness of our method.