Monitoring the structural health of rolling stock is critical to ensuring safe and efficient railroad operation. The structural integrity of freight cars depends on the health of certain structural components within their underframes. These structural components serve two principal functions: supporting the car body and lading, and transmitting longitudinal buff and draft forces. Although railcars are engineered to withstand large static, dynamic and cyclical loads, various structural defects can still develop within their underframes. As a result, both the United States Department of Transportation (USDOT) Federal Railroad Administration (FRA) regulations and individual railroad mechanical department practices require periodic inspection of railcars to detect structural damage and defects. These inspections are conducted manually and rely heavily on the acuity, knowledge and endurance of qualified inspection personnel. Enhancements are possible through machine-vision technology, which uses computer algorithms to convert digital image data into useful information. This paper describes research investigating the feasibility of an automated inspection system capable of detecting structural defects in freight car underframes and presents an inspection approach using machine-vision techniques, including multi-scale image segmentation. Using field data from a preliminary image acquisition system, algorithms were developed to analyze images of railcar underframes and assess the condition of certain structural components. This technology, in conjunction with additional preventive maintenance systems, has the potential to provide more objective information on railcar condition, improve utilization of railcar inspection and repair resources, increase train and employee safety, and improve overall railroad network efficiency.