Relighting from multiple color and depth images using matrix factorization

In this paper, we propose a novel relighting approach that takes advantage of the 3D shape information acquired from a depth sensor. Assuming distant illumination and Lambertian reflectance, we model the reflected light field in terms of spherical harmonic coefficients of the Bi-directional Reflectance Distribution Function (BRDF) and lighting. To estimate both the reflectance and illumination, different illumination samples can be generated through moving the object of interest in space while keeping the light source unchanged. The samples are registered onto the base view's coordinate frame using camera pose estimated from multiple depth maps. Our results indicate that we can successfully recover both illumination and diffuse BRDF (up to a global scaling ambiguity). Our method can be used to estimate complex illumination in indoor environments for applications such as lighting transfer.