Current pavement condition assessment methods are predominantly manual and time consuming. Existing pothole recognition and assessment methods rely on 3D surface reconstruction that requires high equipment and computational costs or relies on acceleration data which provides preliminary results. This paper presents an inexpensive solution that automatically detects and assesses the severity of potholes using vision-based data for both 2D recognition and for 3D reconstruction. The combination of these two techniques is used to improve recognition results by using visual and spatial characteristics of potholes and measure properties (width, number, and depth) that are used to assess severity of potholes. The number of potholes is deduced with 2D recognition whereas the width and depth of the potholes is obtained with 3D reconstruction. The proposed method is validated on several actual potholes. The results show that the proposed inexpensive and visual method holds promise to improve automated pothole detection and severity assessment.