Millimetric Human Surface Capture in Minutes

Our reconstruction algorithm is designed from the ground up to achieve fast training and real-time volumetric rendering out-of-the-box, without relying on a post processing step. This is especially desirable in the context of human acquisition in which hundreds of frames must be processed consistently and as quickly as possible. Our contribution is twofold:

• A novel sparse representation for lightweight memory usage and fast empty-space skipping. Voxels can be allocated or pruned in parallel on the GPU at training time, to maintain sparsity as the implicit surface evolves. A single SDF value is stored into each voxel while the features are factorized into 3 orthogonal planes at a coarser level.
• A reordered volume rendering pipeline for real-time rendering and fast training.

More specifically, three rendering options have been considered in the neural radiance field community:

1. The classic grid-based NeRF pipeline: render(MLP(interpolate(features)))
2. The deferred rendering pipeline: MLP(render(interpolate(features)))
3. The precomputed rendering pipeline: render(interpolate(MLP(features)))

We advocate to use the third option in which the view-dependent color is predicted for all the voxels at once, followed by the volume rendering of the linearly interpolated predicted colors. This allows several key optimizations:

1. On-the-fly volume rendering: We can fuse the volume rendering and color interpolation in a single GPU kernel, that dynamically skips empty space and only writes the final accumulated color. The number of samples per ray thus becomes unlimited, enabling to render a complete image all at once regardless of the number of rays.
2. Fused color prediction: Most previous works predict colors from interpolated features stored in coarse voxels. Our insight is that there is no need to interpolate features as long as the voxels are small enough, which is possible thanks to an efficient sparse structure. Consequently, the color prediction MLP only needs to encode the angular dependency while the spatial dependency is fully explained by the voxel grid, hinting that further reducing the size of the MLP is possible. We opt for an MLP with 32 neurons and 2 layers that proves to be sufficient for human acquisition. For additional memory and compute savings, we fuse the entire color prediction in a single GPU kernel so that only the per-voxel predicted colors need to be stored.

We train the implicit surface in a coarse-to-fine manner, starting from a coarse visual hull. We iteratively subdivide the sparse grid that we supervise with images of increasing resolution. Each voxel has a side length of 2mm at the final level-of-detail, projecting to an area comparable to that of a pixel.

Our dataset contains 8 female and 6 male mannequins. A high quality hand-scanned mesh and corresponding multi-view images are provided in each case. This allows to benchmark reconstruction algorithms on real data with a smaller domain gap in regards to human appearance compared to other popular object datasets. The mannequins are fitted with a diverse selection of clothes displaying complex folds and textured patterns.

We evaluate our approach on a representative set of multi-view reconstruction methods: Voxurf, NeuS2, Colmap, 3D Gaussian Splatting (3DGS), 2D Gaussian Splatting (2DGS) and Gaussian Opacity Fields (GOF). Figure 5 presents a side-by-side comparison of renderings from the novel view synthesis methods and shows the benefits of encoding colors at a millimetric scale. Figure 6 shows the corresponding meshes recovered by each baseline.

The multiview data used for the human reconstruction examples presented here is from the 4DHumanOutfit dataset.