Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model

So far the Non-Rigid Structure-from-Motion problem has been tackled using a batch approach. All the frames are processed at once after the video acquisition takes place. In this paper we propose an incremental approach to the estimation of deformable models. Image frames are processed online in a sequential fashion. The shape is initialised to a rigid model from the first few frames. Subsequently, the problem is formulated as a model based camera tracking problem, where the pose of the camera and the mixing coefficients are updated every frame. New modes are added incrementally when the current model cannot model the current frame well enough. We define a criterion based on image reprojection error to decide whether or not the model must be updated after the arrival of a new frame. The new mode is estimated performing bundle adjustment on a window of frames. To represent the shape, we depart from the traditional explicit low-rank shape model and propose a variant that we call the 3D-implicit low-rank shape model. This alternative model results in a simpler formulation of the motion matrix and provides the ability to represent degenerate deformation modes. We illustrate our approach with experiments on motion capture sequences with ground truth 3D data and with real video sequences.

Code

We made the MATLAB code for this method available. If you use this code for a scientific publication, please reference the original paper:


Download:
zip archive
tar.gz archive

Videos

Video: Model built incrementally, the rank of the decomposition increases when needed. Camera rotations are recovered correctly.

Video: Test on Motion Capture sequence. Model is built incrementally, when new deformations appear, the rank is increased. 3D shape recovered with only 2.9% 3D error against ground truth MOCAP data

Reference: "Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model"
Marco Paladini, Adrien Bartoli, Lourdes Agapito
ECCV 2010, September 5-11 Crete, Greece

This work was partially funded by the European Research Council under ERC Starting Grant agreement 204871-HUMANIS and the British Council/Alliance Research Programme. A. Bartoli was funded by ANR through the HFIBMR Project.