Gender recognition is important for many applications including human computer interaction (HCI). This paper shows that gender recognition accuracy is affected significantly by the age of the person. Our empirical studies on a large face database of 8,000 images with ages from 0 to 93 years show that gender classification accuracy on adult faces can be 10% higher than that on young or senior faces, evaluated using one of the state-of-the-art methods. We examine aging effects on human faces, which motivates us to investigate which features can incorporate shape and texture variations on faces together with gender encoding. Based on the aging effects, the local binary pattern (LBP) and histograms of oriented gradients (HOG) methods are evaluated for gender characterization with age variation. We also investigate a biologically-inspired method for gender recognition. Overall, no matter what methods are used, the accuracies on adult faces are consistently higher than on young or senior faces. This new finding suggests new efforts in both psychological studies and computational visual recognition for the purpose of HCI applications.