Tracking of maneuvering target in infrared images is a challenging problem, especially when the target under tracking experiences large appearance variations or disappears temporarily during some periods. The difficulties lie in the uncertainties in both target motion mode and target appearance, and the nonlinearity in the observation process when the images are treated as measurements directly. This paper presents a robust tracking algorithm to cope with these difficulties. We propose a mixture observation model, which can describe both the gradual intensity variation and sudden disappearance of target pixels, and use an online EM algorithm to update the model parameters. Target is tracked with the interacting multiple model particle filter (IMM-PF), where the proposed adaptive observation model is used to assign weights to the particles based on current measurement. The problems of the simultaneous update of target state and observation model and calculation of motion model likelihood are investigated. Moreover, particle number adaptation is introduced to improve the efficiency and robustness of the algorithm. Finally, we extend the algorithm to multiple targets tracking by introducing a likelihood function based on the probabilistic exclusion principle. Experimental and simulation results demonstrate the robustness of our algorithm.