Dictionary learning (DL) for sparse coding based classification has been widely re-searched in pattern recognition in recent years. Most of the DL approaches focused on the reconstruction performance and the discriminative capability of the learned dictionary. This paper proposes a new method for learning discriminative dictionary for sparse rep-resentation based classification, called Incoherent Fisher Discrimination Dictionary Lear- ning (IFDDL). IFDDL combines the Fisher Discrimination Dictionary Learning (FDDL) method, which learns a structured dictionary where the class labels and the discrimina-tion criterion are exploited, and the Incoherent Dictionary Learning (IDL) method, which learns a dictionary where the mutual incoherence between pairs of atoms is exploited. In the combination, instead of considering the incoherence between atoms in a single shared dictionary as in IDL, we propose to incorporate the incoherence between pairs of atoms within each sub-dictionary, which represent a specific object class. This aims to increase discrimination capacity between basic atoms in sub-dictionaries. The combination allows one to exploit the advantages of both methods and the discrimination capacity of the en-tire dictionary. Extensive experiments have been conducted on benchmark image data sets for Face recognition (ORL database, Extended Yale B database, AR database) and Digit recognition (the USPS database). The experimental results show that our proposed method outperforms most of state-of-the-art methods for sparse coding and DL based classification, meanwhile maintaining similar complexity.