What is the shelf life of a test? The effect of time on the psychometrics of a cognitive ability test battery

The psychometric stability of the Armed Services Vocational Aptitude Battery was studied with data collected at 5 points over a 16-year period using item response theory (IRT) methods. Although 25 of the 200 items changed significantly over the years across 3 different gender-ethnic groups (i.e., White men, White women, and Black men), the overall characteristics of the tests were not severely affected by item-level changes. Items from tests that were more semantically laden were found to be more susceptible to the effects of time compared with those that focused on skills and principles. The findings are discussed in the context of the effects of time on the effectiveness of psychological measures. A call is made to test developers and test users to pay attention to the shelf life of their tests. The use of IRT methods for studying the effects of time on psychometrics is also discussed.