Local image features in space-time or spatio-temporal interest points provide compact and abstract representations of patterns in a video sequence. In this paper, we present a novel human action recognition method based on multi-velocity spatio-temporal interest points (MVSTIPs) and a novel local descriptor called motion energy (ME) orientation histogram (MEOH). The MVSTIP detection includes three steps: first, filtering video frames with multi-direction ME filters at different speeds to detect significant changes at the pixel level; thereafter, a surround suppression model is employed to rectify the ME deviation caused by the camera motion and complicated backgrounds (e.g., dynamic texture); finally, MVSTIPs are obtained with local maximum filters at multispeeds. After detection, we develop MEOH descriptor to capture the motion features in local regions around interest points. The performance of the proposed method is evaluated on KTH, Weizmann, and UCF sports human action datasets. Results show that our method is robust to both simple and complex backgrounds and the method is superior to other methods that are based on local features.