Pattern recognition and object tracking play very important roles in various applications, such as motion capture, object detection/recognition, video surveillance, and human computer interface. One very useful method that is rarely mentioned in literature is performing recognition from the motion cue. In many situations, the motion of an object is very representative and informative; therefore, it is possible to identify the object and its behavior from its motion. In this paper, we propose an original method to both identify and track an object in dynamic scenes. The method works on the situations with occlusions, appearance changes and global camera motions. It does not require prior segmentation or initialization. We test this method on a video database containing 18 World Cup soccer videos recorded from TV to detect and track the soccer ball. The results are satisfying. The results are also integrated into a video indexing system and the improvement on video retrieval is described.