Sparse representation scheme is very influential in visual tracking field. These L1 trackers obtain robustness by finding the target with the minimum reconstruction error via L1 norm minimization problem. However, the high computational burden of L1 minimization and absence of effective model for appearance changes may hamper its application in real world sceneries. In this research, we present a fast and robust tracking method that exploits a fast memory gradient pursuit algorithm (FMGP) with sparse representation scheme in a Bayesian framework to accelerate the L1 minimization process. For tracking, our approach adopts a non-overlapping covariance descriptor and uses a new similarity metric with scaled unscented transform. In order to reduce the problem of drift tracking, we construct a different template dictionary including benchmark template with different scales, adaptive background templates and stable templates. We test the proposed tracking method on the challenging image sequences. Both quantitative and qualitative results demonstrate the excellent performances of the proposed algorithm compared with several state of the art tracking algorithms.