The motion track is an important feature to show the spatio-temporal relationship of a video object [2, 5, 7, 8]. In this paper, we propose a novel motion track representation to represent the motion track in the X-Y plane and the trend of velocity changes. Moreover, a new similarity measure for comparing two motion tracks based on the representation is proposed. Furthermore, the motion track segmentation method is proposed to handle a complicated motion behavior and the relevance feedback is used to improve the query results.