The video content management has attracted increasing attention in recent years. We have proposed a new spatio-temporal knowledge structure, called 3D C-string, to represent the spatio-temporal relations between the objects in a video and to keep track of the motions and size changes of the objects. In this paper, we propose a video algebra to infer the spatio-temporal relations between the objects in a video represented by the 3D C-string. The algebra contains four kinds of rules, namely, transitive, distributive, manipulation, and integration rules. By using those rules, all the binary relations between the objects in a video can be derived from a given 3D C-string. The algebra provides the theoretic basis for spatio-temporal reasoning and video query inference.