In this paper, we present a new method to find a moving object trajectory from an image sequence by using the motion smoothness constraint. Most motion analysis methods find the correspondence of the tokens (feature points or line segments) or their trajectories from image sequemces based on the smooth variation of the path of each moving token. Here, we addthe local and global shage coherence constraints that make token correspondence finding much easier. However, a correspondence is not essentially one-to-one because two sets of tokens extracted from two consecutive frames are not the same. Here, we use the interpolation correspondence to find many-to-many mapping of tokens (line segments). In the experiments, we find that our method has a faster converging rate and higher corresponding accuracy than do previous ones, and that it also can find most of the trajectories of occluded and shape varivant moving objects.