Interactive 3D model acquisition and tracking of building block structures

We present a prototype system for interactive construction and modification of 3D physical models using building blocks.Our system uses a depth sensing camera and a novel algorithm for acquiring and tracking the physical models. The algorithm,LatticeFirst, is based on the fact that building block structures can be arranged in a 3D point lattice where the smallest block unitis a basis in which to derive all the pieces of the model. The algorithm also makes it possible for users to interact naturally with thephysical model as it is acquired, using their bare hands to add and remove pieces. We present the details of our algorithm, along withexamples of the models we can acquire using the interactive system. We also show the results of an experiment where participantsmodify a block structure in the absence of visual feedback. Finally, we discuss two proofofconcept applications: a collaborativeguided assembly system where one user is interactively guided to build a structure based on another users design, and a gamewhere the player must build a structure that matches an onscreen silhouette.