Foveated Rendering and Streaming for 3D Graphics

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this paper, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47\times-7.28\times speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

Publications

teaser image of A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming

A Log-Rectilinear Transformation for Foveated 360-degree Video StreamingTVCG Honorable Mentions

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table, eye tracking, digital world

teaser image of 3D-Kernel Foveated Rendering for Light Fields

3D-Kernel Foveated Rendering for Light Fields

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization, eye tracking, interactive graphics
teaser image of Eye-Dominance-guided Foveated Rendering

Eye-Dominance-guided Foveated Rendering

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.
Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking, interactive graphics
teaser image of Kernel Foveated Rendering

Kernel Foveated RenderingMost read in PACMCGIT

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.
Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays, eye tracking, interactive graphics

Videos

A Log-Rectilinear Transformation for Foveated 360-degree Video Streaming


Eye-Dominance-guided Foveated Rendering


3D-Kernel Foveated Rendering for Light Fields


Talks

Cited By

  • Foveated Real-Time Path Tracing in Visual-Polar Space. Eurographics Symposium on Rendering - DL-only and Industry Track.Matias Koskela, Atro Lotvonen, Markku Mäkitalo, Petrus Kivi, Timo Viitanen, and Pekka Jääskeläinen. source | cite | search
  • Perception-Driven Hybrid Foveated Depth of Field Rendering for Head-Mounted Displays. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).Jingyu Liu, Claire Mantel, and Soren Forchhammer. source | cite | search
  • Pistol: Pupil Invisible Supportive Tool to Extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D As Well As 3D Gaze. https://arxiv.org/abs/2201.06799.Wolfgang Fuhl, Daniel Weber, and Enkelejda Kasneci. source | cite | search
  • Foveated Depth-of-Field Filtering in Head-Mounted Displays. ACM Transactions on Applied Perception.Martin Weier, Thorsten Roth, André Hinkenjann, and Philipp Slusallek. source | cite | search
  • Foveated Streaming of Real-time Graphics. Proceedings of the 12th ACM Multimedia Systems Conference.Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. source | cite | search
  • Selective Foveated Ray Tracing for Head-Mounted Displays. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).Youngwook Kim, Yunmin Ko, and Insung Ihm. source | cite | search
  • Noise-based Enhancement for Foveated Rendering. ACM Transactions on Graphics.Taimoor Tariq, Cara Tursun, and Piotr Didyk. source | cite | search
  • Rectangular Mapping-based Foveated Rendering. The IEEE Conference on Virtual Reality and 3D User Interfaces 2022 (IEEE VR).Jiannan Ye, Anqi Xie, Susmjia Jabbireddy, Yunchuan Li, Xubo Yang, and Xiaoxu Meng. source | cite | search
  • An Integrative View of Foveated Rendering. 0.Bipul Mohanto, ABM Tariqul Islam, Enrico Gobbetti, and Oliver Staadt. source | cite | search
  • A Simulation System for Scene Synthesis in Virtual Reality. Virtual Reality and Mixed Reality.Jingyu Liu, Claire Mantel, Florian Schweiger, and S{\o}ren Forchhammer. source | cite | search
  • A Cybersickness Review: Causes, Strategies, and Classification Methods. Journal on Interactive Systems.Thiago Porcino, Daniela Trevisan, and Esteban Clua. source | cite | search
  • Building Virtual Scene Construction and Environmental Impact Analysis Based on Image Processing. Scientific Programming.Fan Ying and Zhou Bo. source | cite | search
  • Dynamic Viewport Selection-Based Prioritized Bitrate Adaptation for Tile-Based 360\textdegree Video Streaming. IEEE Access.Abid Yaqoob, Mohammed Amine Togou, and Gabriel-Miro Muntean. source | cite | search
  • Practical Saccade Prediction for Head-Mounted Displays: Towards a Comprehensive Model. ACM Transactions on Applied Perception.Elena Arabadzhiyska, Cara Tursun, Hans-Peter Seidel, and Piotr Didyk. source | cite | search
  • Foveated Rendering: Motivation, Taxonomy, and Research Directions. arXiv.2205.04529.Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. source | cite | search
  • A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry. Proceedings of the ACM on Human-Computer Interaction.Johannes Meyer, Thomas Schlebusch, and Enkelejda Kasneci. source | cite | search
  • Visual Perception in Virtual Reality and the Application in Sports. Universität Magdeburg.Stefan Pastel. source | cite | search
  • Modeling and Optimizing Human-in-the-Loop Visual Perception. SID Symposium Digest of Technical Papers.Qi Sun, Budmonde Duinkharjav, and Anjul Patney. source | cite | search
  • Deep Light Field Spatial Super-Resolution Using Heterogeneous Imaging. IEEE Transactions on Visualization and Computer Graphics.Yeyao Chen, Gangyi Jiang, Mei Yu, Haiyong Xu, and Yo-Sung Ho. source | cite | search
  • A Real 3D Scene Rendering Optimization Method Based on Region of Interest and Viewing Frustum Prediction in Virtual Reality. International Journal of Digital Earth.Pei Dang, Jun Zhu, Jianlin Wu, Weilian Li, Jigang You, Lin Fu, Yiqun Shi, and Yuhang Gong. source | cite | search
  • Application of Eye-tracking Systems Integrated Into Immersive Virtual Reality and Possible Transfer to the Sports Sector - a Systematic Review. Multimedia Tools and Applications.Stefan Pastel, Josua Marlok, Nicole Bandow, and Kerstin Witte. source | cite | search
  • Progressive Multi-scale Light Field Networks. arXiv.2208.06710.David Li and Amitabh Varshney. source | cite | search
  • Wavelet-Based Fast Decoding of 360° Videos. arXiv.2208.10859.Colin Groth, Sascha Fricke, Susana Castillo, and Marcus Magnor. source | cite | search
  • Bullet Comments for 360\textdegreeVideo. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).Yi-Jun Li, Jinchuan Shi, Fang-Lue Zhang, and Miao Wang. source | cite | search
  • Look Over There! Investigating Saliency Modulation for Visual Guidance With Augmented Reality Glasses. The 35th Annual ACM Symposium on User Interface Software and Technology.Jonathan Sutton, Tobias Langlotz, Alexander Plopski, Stefanie Zollmann, Yuta Itoh, and Holger Regenbrecht. source | cite | search
  • Rega-Net:Retina Gabor Attention for Deep Convolutional Neural Networks. arXiv.2211.12698.Chun Bao, Jie Cao, Yaqian Ning, Yang Cheng, and Qun Hao. source | cite | search
  • Foveated Rendering: A State-of-the-Art Survey. arXiv.2211.07969.Lili Wang, Xuehuai Shi, and Yi Liu. source | cite | search
  • Towards Energy Efficient Mobile Eye Tracking for AR Glasses Through Optical Sensor Technology. arXiv.2212.03189.Johannes Meyer. source | cite | search
  • Beyond Flicker, Beyond Blur: View-coherent Metameric Light Fields for Foveated Display. SIGGRAPH Posters.Prithvi Kohli, David R. Walton, Rafael Kuffner dos Anjos, Anthony Steed, and Tobias Ritschel. source | cite | search
  • Foveated Rendering Technique in Modern Graphics. University of Maryland, College Park.Xiaoxu Meng. source | cite | search
  • Time-Warped Foveated Rendering for Virtual Reality Headsets. Computer Graphics Forum.Linus Franke, Laura Fink, Jana Martschinke, Kai Selgrad, and Marc Stamminger. source | cite | search
  • Foveated Instant Radiosity. 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).Lili Wang, Runze Li, Xuehuai Shi, Ling-Qi Yan, and Zhichao Li. source | cite | search
  • Foveated Light Culling. Computers & Graphics.Qinqi Yang, Zhuxin Chen, Yanli Liu, Guanyu Xing, and Yanci Zhang. source | cite | search
  • A Unified Deep Learning Approach for Foveated Rendering \& Novel View Synthesis From Sparse RGB-D Light Fields. 2020 International Conference on 3D Immersion (IC3D).Vineet Thumuluri and Mansi Sharma. source | cite | search
  • On the Interplay of Foveated Rendering and Video Encoding. 26th ACM Symposium on Virtual Reality Software and Technology.Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. source | cite | search
  • Modelling Foveated Depth-of-field Blur for Improving Depth Perception in Virtual Reality. 2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS).Razeen Hussain, Manuela Chessa, and Fabio Solari. source | cite | search
  • Toward Standardized Classification of Foveated Displays. IEEE Transactions on Visualization and Computer Graphics.Josef Spjut, Ben Boudaoud, Jonghyun Kim, Trey Greer, Rachel Albert, Michael Stengel, Kaan Aksit, and David Luebke. source | cite | search
  • Learning Foveated Reconstruction to Preserve Perceived Image Statistics. https://arxiv.org/abs/2108.03499.Luca Surace, Marek Wernikowski, Okan Tursun, Karol Myszkowski, Rados{\l}aw Mantiuk, and Piotr Didyk. source | cite | search
  • State of the Art in Perceptual VR Displays. Real VR –Immersive Digital Reality.Gordon Wetzstein, Anjul Patney, and Qi Sun. source | cite | search
  • How the Presence and Size of Static Peripheral Blur Affects Cybersickness in Virtual Reality. ACM Transactions on Applied Perception.Yun-Xuan Lin, Rohith Venkatakrishnan, Roshan Venkatakrishnan, Elham Ebrahimi, Wen-Chieh Lin, and Sabarish V. Babu. source | cite | search
  • Cybersickness Analysis Using Symbolic Machine Learning Algorithms. Universidade Federal Fluminense.Thiago Porcina. source | cite | search
  • Eye Tracking Data Collection Protocol for VR for Remotely Located Subjects Using Blockchain and Smart Contracts. 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).Efe Bozkir, Shahram Eivazi, Mete Akgun, and Enkelejda Kasneci. source | cite | search
  • A Privacy-preserving Approach to Streaming Eye-tracking Data. IEEE Transactions on Visualization and Computer Graphics.Brendan David-John, Diane Hosfelt, Kevin Butler, and Eakta Jain. source | cite | search
  • FAVR - Accelerating Direct Volume Rendering for Virtual RealitySystems. 2020 IEEE Visualization Conference (VIS).Andre Waschk and Jens Kruger. source | cite | search
  • Focus Guided Light Field Saliency Estimation. 2021 13th International Conference on Quality of Multimedia Experience (QoMEX).Ailbhe Gill, Emin Zerman, Martin Alain, Mikael Le Pendu, and Aljosa Smolic. source | cite | search
  • Foveated Photon Mapping. IEEE Transactions on Visualization and Computer Graphics.Xuehuai Shi, Lili Wang, Xiaoheng Wei, and Ling-Qi Yan. source | cite | search
  • Exploiting Object-of-Interest Information to Understand Attention in VR Classrooms. 2021 IEEE Virtual Reality and 3D User Interfaces (VR).Efe Bozkir, Philipp Stark, Hong Gao, Lisa Hasenbein, Jens-Uwe Hahn, Enkelejda Kasneci, and Richard Gollner. source | cite | search
  • Rendering Optimizations for Virtual Reality Using Eye-Tracking. 2020 22nd Symposium on Virtual and Augmented Reality (SVR).Shawn Matthews, Alvaro Uribe-Quevedo, and Alexander Theodorou. source | cite | search
  • Towards Everyday Virtual Reality Through Eye Tracking. der Eberhard Karls Universität Tübingen.Efe Bozkir. source | cite | search
  • HPCGen: Hierarchical K-Means Clustering and Level Based Principal Components for Scan Path Genaration. .Wolfgang Fuhl. source | cite | search
  • Perceptual Rasterization for Head-mounted Display Image Synthesis. ACM Transactions on Graphics.Sebastian Friston, Tobias Ritschel, and Anthony Steed. source | cite | search
  • A Review of Resolution Losses for AR/VR Foveated Imaging Applications. 2018 IEEE Games, Entertainment, Media Conference (GEM).Timothee E. Cognard, Alexander Goncharov, Nicholas Devaney, Chris Dainty, and Peter Corcoran. source | cite | search
  • Impact of Foveated Rendering on Procedural Task Training. Virtual, Augmented and Mixed Reality. Multimodal Interaction.Rafael Radkowski and Supriya Raul. source | cite | search
  • Masked360: Enabling Robust 360-degree Video Streaming With Ultra Low Bandwidth Consumption. IEEE Transactions on Visualization and Computer Graphics.Zhenxiao Luo, Baili Chai, Zelong Wang, Miao Hu, and Di Wu. source | cite | search
  • Eye-tracked Virtual Reality: A Comprehensive Survey on Methods and Privacy Challenges. arXiv.2305.14080.Efe Bozkir, Süleyman Özdel, Mengdi Wang, Brendan David-John, Hong Gao, Kevin Butler, Eakta Jain, and Enkelejda Kasneci. source | cite | search
  • LFACon: Introducing Anglewise Attention to No-Reference Quality Assessment in Light Field Space. IEEE Transactions on Visualization and Computer Graphics.Qiang Qu, Xiaoming Chen, Yuk Ying Chung, and Weidong Cai. source | cite | search
  • Quality-Constrained Encoding Optimization for Omnidirectional Video Streaming. IEEE Transactions on Circuits and Systems for Video Technology.Chaofan He, Roberto G. de A. Azevedo, Jiacheng Chen, Shuyuan Zhu, Bing Zeng, and Pascal Frossard. source | cite | search
  • Providing Privacy for Eye-Tracking Data With Applications in XR. University of Florida.Brendan David-John. source | cite | search
  • Polar Coordinate Fourier Single-pixel Imaging. Optics Letters.Guan Wang, Huaxia Deng, Mengchao Ma, and Xiang Zhong. source | cite | search
  • Foveated Rendering: A State-of-the-art Survey. Computational Visual Media.Lili Wang, Xuehuai Shi, and Yi Liu. source | cite | search
  • Data and Model Hybrid-driven Virtual Reality Robot Operating System. Frontiers in Energy Research.Xinyu Liu, Lin Nan, Yuexin Lin, Jiatong Han, Jinxin Liu, and Tao Ku. source | cite | search
  • Area of Interest Adaption Using Feature Importance. arXiv.2303.12744.Wolfgang Fuhl, Susanne Zabel, Theresa Harbig, Julia Astrid Moldt, Teresa Festl Wiete, Anne Herrmann Werner, and Kay Nieselt. source | cite | search
  • Software-Defined Imaging: A Survey. Proceedings of the IEEE.Suren Jayasuriya, Odrika Iqbal, Venkatesh Kodukula, Victor Torres, Robert LiKamWa, and Andreas Spanias. source | cite | search
  • Locomotion-aware Foveated Rendering. 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR).Xuehuai Shi, Lili Wang, Jian Wu, Wei Ke, and Chan-Tong Lam. source | cite | search
  • Design Guidelines for Limiting and Eliminating Virtual Reality-induced Symptoms and Effects at Work: A Comprehensive, Factor-oriented Review. Frontiers in Psychology.Alexis D. Souchet, Domitile Lourdeaux, Jean-Marie Burkhardt, and Peter A. Hancock. source | cite | search
  • Non-homogeneous Denoising for Virtual Reality in Real-time Path Tracing Rendering. Graphical Models.Victor Peres, Esteban Clua, Thiago Porcino, and Anselmo Montenegro. source | cite | search
  • Compact Near-eye Display for Firefighter\textquotesingles Self-contained Breathing Apparatus. ETRI Journal.Ungyeon Yang. source | cite | search
  • Individualized Foveated Rendering With Eye-tracking Head-mounted Display. Virtual Reality.Jihwan Kim, Jejoong Kim, Myeongul Jung, Taesoo Kwon, and Kwanguk Kenny Kim. source | cite | search
  • Virtual Reality Telepresence: 360-Degree Video Streaming With Edge-Compute Assisted Static Foveated Compression. IEEE Transactions on Visualization and Computer Graphics.Xincheng Huang, James Riddell, and Robert Xiao. source | cite | search
  • Stay In Touch