In this paper, a novel local texture descriptor called Local Block Difference Pattern (LBDP) is proposed. In conventional LBP, the problem of sensitivity to intensity change usually constrains its practicality due to its pixel-based comparison in the encoding mechanism. Different from LBP, the proposed LBDP describes the local texture information by extending the encoding mechanism from pixel-based comparison to blockbased comparison so as to extracting more detailed information. The discrimination capability of LBDP is thus enhanced because the difference of local structures and the similarity of neighboring blocks are both considered in the proposed encoding mechanism. Moreover, the proposed LBDP can decrease the influence resulting from intensity change because of the expanding of encoding range. The validity and excel performance of the proposed LBDP is demonstrated in the application of gait-based gender classification. In the experiments, CASIA dataset B is adopted for performance evaluation and the results demonstrate that the proposed LBDP outperforms the other local texture descriptors.