Breast-conserving surgery is a frequent option for women with stage I and II breast cancer, and with radiation treatment, can be as effective as a mastectomy. However, adequate margin detection remains a challenge, and too often additional surgeries are required. Optical coherence tomography (OCT) provides a potential method for real-time, high-resolution imaging of breast tissue during surgery. Intra-operative OCT imaging of excised breast tissues has been previously demonstrated by several groups. In this study, a novel handheld surgical probe-based OCT system is introduced, which was used by the surgeon to image in vivo, within the tumor cavity, and immediately following tumor removal in order to detect the presence of any remaining cancer. Following resection, study investigators imaged the excised tissue with the same probe for comparison. We present OCT images obtained from over 15 patients during lumpectomy and mastectomy surgeries. Images were compared to post-operative histopathology for diagnosis. OCT images with micron scale resolution show areas of heterogeneity and disorganized features indicative of malignancy, compared to more uniform regions of normal tissue. Video-rate acquisition shows the inside of the tumor cavity as the surgeon sweeps the probe along the walls of the surgical cavity. This demonstrates the potential of OCT for real-time assessment of surgical tumor margins and for reducing the unacceptably high re-operation rate for breast cancer patients.