Ballast is an essential layer of railroad substructure that facilitates drainage and load distribution and is constantly subjected repeated train loadings. Both size distribution and morphological properties of the ballast aggregates therefore impact significantly the overall performance of railway track substructure in terms of strength, modulus and permanent deformation. It is crucial to have reliable inspection methods that can quantitatively assess the ballast condition with respect to the aggregate size and shape properties as part of ballasted track design and maintenance strategies. This paper introduces a prototype of non-intrusive track ballast inspection system based on smartphone platform such that engineer or inspector can take photos of ballast aggregates with a smartphone-like mobile device to assess the ballast condition. The analyses are performed based on cloud-based photogrammetry processing, which is enabled by (1) cloud computing server (for image processing and analyses of the aggregate photos) and (2) smartphone (to take photos of ballast aggregate and communicate with the server by sending the photos and receiving the analysis results) or similar mobile device. The system utilizes mobile device hardware, i.e., built-in camera and global positioning system (GPS), to add location and time information into fetched ballast aggregate images, analyzes size and shape properties and evaluates ballast condition. All collected data are entered in a database in the cloud computing server such that the engineers can exploit the spatial-Temporal information as well as the analyzed ballast aggregate condition whenever needed. The initial results with the ballast inspection system are presented in this paper to demonstrate its certain significant advantages over the commonly used ballast inspection techniques.