Analysis and quantification of magnetic resonance spectroscopic imaging (MRSI) data is complicated by the presence of lipid nuisance signals in the spectra. These signals typically appear as peaks with amplitudes much larger than those of the metabolites of interest and, in the case of lipids, present broad, distorted lineshapes. Although the problem of lipid signal removal by postprocessing has been addressed in the past both in the context of single voxel spectroscopy (SVS) and MRSI, existing approaches use either spatial or spectral constraints to determine the lipid component in the signal. This paper introduces a method that incorporates both types of constraints for improved removal of lipid signals. Specifically, this method uses an anatomical image of the lipid locations to spatially constrain the lipid estimate as well as a field inhomogeneity map to improve spectral fitting of the lipid lineshape. Experimental results are shown to demonstrate the performance of the proposed method.