In traditional fractal image compression, the encoding procedure is time-consuming due to the full search mechanism. In order to speedup the encoder, we adopt particle swarm optimization method performed under classification and Dihedral transformation to further decrease the amount of MSE computations. The classifier partitions all of the blocks in domain pool and range pool into three classes according to the third level wavelet coefficients. Each range block searches the most similar block only from the blocks of the same class. Furthermore, according to the property of Dihedral transformation, only four transformations for each domain block are considered so as to reduce the encoding time. Experimental results show that, the encoding time of the proposed method is faster than that of the full search method. Experimental results show that the proposed method is about 178 times faster with only 1.46 dB decay in image quality.