Breast cancer is a common disease in women. Current imaging and diagnostic methods for breast cancer confront several limitations, like time-consuming, invasive and with a high cost. Alternative strategies are in high demand to alleviate patients' trauma and lower medical expenses. Coherent anti-Stokes Raman scattering (CARS) imaging technique offers many advantages, including label-free, sub-wavelength spatial resolution and video-rate imaging speed. Therefore, it has been demonstrated as a powerful tool for various biomedical applications. In this study, we present a label-free fast imaging method to identify breast cancer and its subtypes using CARS microscopy. Human breast tissues, including normal, benign and invasive carcinomas, were imaged ex vivo using a custom-built CARS microscope. Compared with results from corresponding hematoxylin and eosin (H&E) stains, the CARS technique has demonstrated its capability in identifying morphological features in a similar way as in H&E stain. These features can be used to distinguish breast cancer from normal and benign tissues, and further separate cancer subtypes from each other. Our pilot study suggests that CARS microscopy could be used as a routine examination tool to characterize breast cancer ex vivo. Moreover, its label-free and fast imaging properties render this technique as a promising approach for in vivo and real-time imaging and diagnosis of breast cancer.