Towards Modeling 3D Dense Shape Correspondence from Category-Specific Multi-View Images

We present Neural Radiance Fields (NeRF) with Template, dubbed Template-NeRF, for modeling 3D appearance and geometry and generating dense shape correspondence simultaneously among objects of the same category from only multi-view posed images. No 3D supervision or ground-truth correspondence knowledge is required. The learned dense correspondence can be directly used for various image-based tasks such as keypoint detection, part segmentation, and texture transfer that previously required specific model designs. Our method can also accommodate annotation transfer in a one or few-shot manner. Given only one or a few annotated instances of the category, our model can transfer to many others. We introduce deep implicit templates on 3D data into the 3D-aware image synthesis pipeline NeRF using periodic activation and feature-wise linear modulation (FiLM) conditioning. By representing object instances within the same category as shape and appearance variation of a shared NeRF template, our proposed method can achieve dense shape correspondence reasoning on images for a wide range of object classes.