Deep 3D Representations

We introduce a new implicit shape representation called Primary Ray-based Implicit Function (PRIF). In contrast to most existing approaches based on the signed distance function (SDF) which handles spatial locations, our representation operates on oriented rays. Specifically, PRIF is formulated to directly produce the surface hit point of a given input ray, without the expensive sphere-tracing operations, hence enabling efficient shape extraction and differentiable rendering. We demonstrate that neural networks trained to encode PRIF achieve successes in various tasks including single shape representation, category-wise shape generation, shape completion from sparse or noisy observations, inverse rendering for camera pose estimation, and neural rendering with color.
We introduce Multiresolution Deep Implicit Functions (MDIF), a hierarchical representation that can recover fine geometry detail, while being able to perform global operations such as shape completion. Our model represents a complex 3D shape with a hierarchy of latent grids, which can be decoded into different levels of detail and also achieve better accuracy. For shape completion, we propose latent grid dropout to simulate partial data in the latent space and therefore defer the completing functionality to the decoder side. This along with our multires design significantly improves the shape completion quality under decoder-only latent optimization. To the best of our knowledge, MDIF is the first deep implicit function model that can at the same time (1) represent different levels of detail and allow progressive decoding; (2) support both encoder-decoder inference and decoder-only latent optimization, and fulfill multiple applications; (3) perform detailed decoder-only shape completion. Experiments demonstrate its superior performance against prior art in various 3D reconstruction tasks.

Publications

teaser image of PRIF: Primary Ray-based Implicit Function

PRIF: Primary Ray-based Implicit Function

European Conference on Computer Vision (ECCV), 2022.
Keywords: deep implicit functions, neural representation, signed distance function, interactive perception, interactive graphics
teaser image of Multiresolution Deep Implicit Functions for 3D Shape Representation

Multiresolution Deep Implicit Functions for 3D Shape Representation

2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Keywords: deep implicit functions, neural representation, compression, levels of detail, MDIF, interactive perception

Videos

Talks

Cited By

  • BACON: Band-limited Coordinate Networks for Multiscale Scene Representation. https://arxiv.org/pdf/2112.04645.pdf.David B. Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. source | cite | search
  • UNIST: Unpaired Neural Implicit Shape Translation Network. https://arxiv.org/pdf/2112.05381.pdf.Qimin Chen, Johannes Merz, Aditya Sanghi, Hooman Shayani, Ali Mahdavi-Amiri, and Hao Zhang. source | cite | search
  • PINs: Progressive Implicit Networks for Multi-Scale Neural Representations. https://arxiv.org/pdf/2202.04713.pdf.Zoe Landgraf, Alexander Sorkine Hornung, and Ricardo Silveira Cabral. source | cite | search
  • 3DILG: Irregular Latent Grids for 3D Generative Modeling. arXiv.2205.13914.Biao Zhang, Matthias Nießner, and Peter Wonka. source | cite | search
  • PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories. arXiv.2206.04916.Yuchen Rao, Yinyu Nie, and Angela Dai. source | cite | search
  • Preprocessing Enhanced Image Compression for Machine Vision. arXiv.2206.05650.Guo Lu, Xingtong Ge, Tianxiong Zhong, Jing Geng, and Qiang Hu. source | cite | search
  • PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation. arXiv.2207.11790.Bo Sun, Vladimir G. Kim, Noam Aigerman, Qixing Huang, and Siddhartha Chaudhuri. source | cite | search
  • Progressive Multi-scale Light Field Networks. arXiv.2208.06710.David Li and Amitabh Varshney. source | cite | search
  • Learning Deep Implicit Functions for 3D Shapes With Dynamic Code Clouds. https://arxiv.org/abs/2203.14048.Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. source | cite | search
  • Towards Implicit Text-Guided 3D Shape Generation. https://arxiv.org/abs/2203.14622.Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu. source | cite | search
  • Supplementary for “Learning Deep Implicit Functions for 3D Shapes With Dynamic Code Clouds”. https://arxiv.org/abs/2203.14048.Tianyang Li, Xin Wen, Yu-Shen Liu, Hua Su, and Zhizhong Han. source | cite | search
  • ShapeFormer: Transformer-based Shape Completion Via Sparse Representation. arXiv.2201.10326.Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. source | cite | search
  • NeuRBF: A Neural Fields Representation With Adaptive Radial Basis Functions. arXiv.2309.15426.Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. source | cite | search
  • Controllable Mesh Generation Through Sparse Latent Point Diffusion Models. arXiv.2303.07938.Zhaoyang Lyu, Jinyi Wang, Yuwei An, Ya Zhang, Dahua Lin, and Bo Dai. source | cite | search
  • MARF: The Medial Atom Ray Field Object Representation. Computers & Graphics.Peder Bergebakken Sundt and Theoharis Theoharis. source | cite | search
  • TMSDNet: Transformer With Multi-scale Dense Network for Single and Multi-view 3D Reconstruction. Computer Animation and Virtual Worlds.Xiaoqiang Zhu, Xinsheng Yao, Junjie Zhang, Mengyao Zhu, Lihua You, Xiaosong Yang, Jianjun Zhang, He Zhao, and Dan Zeng. source | cite | search
  • Grid-guided Neural Radiance Fields for Large Urban Scenes. arXiv.2303.14001.Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. source | cite | search
  • Text2shape Deep Retrieval Model: Generating Initial Cases for Mechanical Part Redesign Under the Context of Case-Based Reasoning. arXiv.2302.06341.Tianshuo Zang, Maolin Yang, Wentao Yong, and Pingyu Jiang. source | cite | search
  • Ponder : Point Cloud Pre-training Via Neural Rendering. arXiv.2301.00157.Di Huang, Sida Peng, Tong He, Xiaowei Zhou, and Wanli Ouyang. source | cite | search
  • OReX: Object Reconstruction From Planner Cross-sections Using Neural Fields. arXiv.2211.12886.Haim Sawdayee, Amir Vaxman, and Amit H. Bermano. source | cite | search
  • FPM-INR: Fourier Ptychographic Microscopy Image Stack Reconstruction Using Implicit Neural Representations. arXiv.2310.18529.Haowen Zhou, Brandon Feng, Haiyun Guo, Siyu Lin, Mingshu Liang, Christopher A. Metzler, and Changhuei Yang. source | cite | search
  • Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations. arXiv.2310.17880.Tristan Aumentado-Armstrong, Ashkan Mirzaei, Marcus A. Brubaker, Jonathan Kelly, Alex Levinshtein, Konstantinos G. Derpanis, and Igor Gilitschenski. source | cite | search
  • RayDF: Neural Ray-surface Distance Fields With Multi-view Consistency. arXiv.2310.19629.Zhuoman Liu and Bo Yang. source | cite | search
  • ConVRT: Consistent Video Restoration Through Turbulence With Test-time Optimization of Neural Video Representations. arXiv.2312.04679.Haoming Cai, Jingxi Chen, Brandon Feng, Weiyun Jiang, Mingyang Xie, Kevin Zhang, Ashok Veeraraghavan, and Christopher Metzler. source | cite | search
  • Ponder : Point Cloud Pre-training Via Neural Rendering. arXiv.2301.00157.Di Huang, Sida Peng, Tong He, Xiaowei Zhou, and Wanli Ouyang. source | cite | search
  • 3D Motion Magnification: Visualizing Subtle Motions With Time Varying Radiance Fields. arXiv.2308.03757.Brandon Feng, Hadi Alzayer, Michael Rubinstein, William Freeman, and Jia-Bin Huang. source | cite | search
  • Stay In Touch