We shall present some results of our research on band tracking and gesture recognition in the last 5 years. This research is motivated by applications in human-computer interaction such as display control in virtual environments and the manipulation of virtual objects. Although we have studied both Appearance-Based and 3D Model-Based approaches, this talk will concentrate on the latter. At any given time instant, the hand configuration - 6 parameters for the global hand "pose" and 21 finger joint angles (the hand "posture") - is a point the 27-dimensional configuration space. We track the trajectory of this point over time using a 3D model-based and analysis-by-synthesis approach. The challenge is to represent the constraints on band posture and finger movement in a compact way and use this representation to speed up the search in the 21-dimensional space. The tracking results can then be used to do gesture recognition. Some preliminary results will be shown on constraint representation and its use in tracking.