Inverse Rendering and Relighting from Multiple Color Plus Depth Images

Siying Liu, Minh N. Do

Research output: Contribution to journalArticlepeer-review


We propose a novel relighting approach that takes advantage of multiple color plus depth images acquired from a consumer camera. Assuming distant illumination and Lambertian reflectance, we model the reflected light field in terms of spherical harmonic coefficients of the bi-directional reflectance distribution function and lighting. We make use of the noisy depth information together with color images taken under different illumination conditions to refine surface normals inferred from depth. We first perform refinement on the surface normals using the first order spherical harmonics. We initialize this non-linear optimization with a linear approximation to greatly reduce computation time. With surface normals refined, we formulate the recovery of albedo and lighting in a matrix factorization setting, involving second order spherical harmonics. Albedo and lighting coefficients are recovered up to a global scaling ambiguity. We demonstrate our method on both simulated and real data, and show that it can successfully recover both illumination and albedo to produce realistic relighting results.

Original languageEnglish (US)
Article number7983410
Pages (from-to)4951-4961
Number of pages11
JournalIEEE Transactions on Image Processing
Issue number10
StatePublished - Oct 2017


  • Inverse rendering
  • illumination
  • reflectance
  • relighting

ASJC Scopus subject areas

  • Software
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Inverse Rendering and Relighting from Multiple Color Plus Depth Images'. Together they form a unique fingerprint.

Cite this