Most of the existing tracking algorithms are challenged for the deficiency of handling non-stationary target appearance such as the drastic scale and perspective change of a moving pedestrian in the PTZ surveillance record. We propose a novel pedestrian tracking algorithm to cope with this problem by integrating incrementally learned representation and classification model. In the representation model, besides the widely used intensity template, the contour template with several sets of profiles from different perspectives is also employed to cope with the change of pedestrian contour. Both templates are updated incrementally during the tracking process to deal with the non-stationary appearance of the pedestrian. In the classification model, a multiple instance classier based on an incremental support vector machine is trained on-line as new observation becomes available. The learned classifier keeps the evolving representation model from drifting and enables reinitialization of the tracker once a failure occurs in the tracking process. The effectiveness of our algorithm is tested over several surveillance records captured from PTZ. The experiment results show that our algorithm can track the pedestrian more robustly than the other two compared cutting edge tracking algorithms.