Compared with affine transformation, projection transformation represents the process of imaging objects more accurately. This paper proposes a novel object tracking method using particle filtering with dual manifold models. One is the covariance manifold used for the object observation model, and the other is the geometric deformation on SL(3) group, where the rank of projection transformation matrix equals 1, adapted to utilize for object dynamic model. Our main contribution is to utilize both the geometry of SL(3) group and covariance manifolds in developing a general particle filtering-based tracking algorithm. Extensive experiments prove that the proposed method can realize stable and accurate tracking of object with significant geometric deformation, even with illumination changes and when an object is obscured.