Foveated Rendering and Streaming for 3D Graphics

Light fields capture both the spatial and angular rays, thus enabling free-viewpoint rendering and custom selection of the focal plane. Scientists can interactively explore pre-recorded microscopic light fields of organs, microbes, and neurons using virtual reality headsets. However, rendering high-resolution light fields at interactive frame rates requires a very high rate of texture sampling, which is challenging as the resolutions of light fields and displays continue to increase. In this paper, we present an efficient algorithm to visualize 4D light fields with 3D-kernel foveated rendering (3D-KFR). The 3D-KFR scheme coupled with eye-tracking has the potential to accelerate the rendering of 4D depth-cued light fields dramatically. We have developed a perceptual model for foveated light fields by extending the KFR for the rendering of 3D meshes. On datasets of high-resolution microscopic light fields, we observe 3.47\times-7.28\times speedup in light field rendering with minimal perceptual loss of detail. We envision that 3D-KFR will reconcile the mutually conflicting goals of visual fidelity and rendering speed for interactive visualization of light fields.

Publications

teaser image of A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2021.
Keywords: 360° video, foveation, virtual reality, live video stream-ing, log-rectilinear, summed-area table

teaser image of 3D-Kernel Foveated Rendering for Light Fields

3D-Kernel Foveated Rendering for Light Fields

IEEE Transactions on Visualization and Computer Graphics (TVCG), 2020.
Keywords: light field, foveated rendering, microscopic light field, eye tracking, visualization
teaser image of Eye-Dominance-Guided Foveated Rendering

Eye-Dominance-Guided Foveated Rendering

IEEE Transactions on Visualization and Computer Graphics (TVCG, Special Issue of IEEE Conference on Virtual Reality and 3D User Interfaces), 2020.
Keywords: virtual reality, foveated rendering, perception, gaze-contingent rendering, ocular dominance, eye tracking
teaser image of Kernel Foveated Rendering

Kernel Foveated Rendering

Proceedings of the ACM on Computer Graphics and Interactive Techniques (I3D), 2018.
Keywords: foveated rendering, perception, log-polar mapping, eye-tracking, virtual reality, head-mounted displays

Videos

A Log-Rectilinear Transformation for Foveated 360-Degree Video Streaming


Eye-Dominance-Guided Foveated Rendering


3D-Kernel Foveated Rendering for Light Fields


Talks

Cited By

  • Perceptual Rasterization for Head-Mounted Display Image Synthesis. ACM Transactions on Graphics.Sebastian Friston, Tobias Ritschel, and Anthony Steed. source | cite
  • A Review of Resolution Losses for AR/VR Foveated Imaging Applications. 2018 IEEE Games, Entertainment, Media Conference (GEM). Timothee E. Cognard, Alexander Goncharov, Nicholas Devaney, Chris Dainty, and Peter Corcoran. source | cite
  • Impact of Foveated Rendering on Procedural Task Training. Virtual, Augmented and Mixed Reality. Multimodal Interaction. Rafael Radkowski and Supriya Raul. source | cite
  • Foveated Real-Time Path Tracing in Visual-Polar Space. Eurographics Symposium on Rendering - DL-only and Industry Track. Matias Koskela, Atro Lotvonen, Markku Mäkitalo, Petrus Kivi, Timo Viitanen, and Pekka Jääskeläinen. source | cite
  • Perception-Driven Hybrid Foveated Depth of Field Rendering for Head-Mounted Displays. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Jingyu Liu, Claire Mantel, and Soren Forchhammer. source | cite
  • Pistol: Pupil Invisible Supportive Tool to Extract Pupil, Iris, Eye Opening, Eye Movements, Pupil and Iris Gaze Vector, and 2D As Well As 3D Gaze. https://arxiv.org/abs/2201.06799. Wolfgang Fuhl, Daniel Weber, and Enkelejda Kasneci. source | cite
  • Foveated Depth-of-Field Filtering in Head-Mounted Displays. ACM Transactions on Applied Perception. Martin Weier, Thorsten Roth, André Hinkenjann, and Philipp Slusallek. source | cite
  • Foveated Streaming of Real-Time Graphics. Proceedings of the 12th ACM Multimedia Systems Conference. Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. source | cite
  • Selective Foveated Ray Tracing for Head-Mounted Displays. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Youngwook Kim, Yunmin Ko, and Insung Ihm. source | cite
  • Rectangular Mapping-Based Foveated Rendering. The IEEE Conference on Virtual Reality and 3D User Interfaces 2022 (IEEE VR). Jiannan Ye, Anqi Xie, Susmjia Jabbireddy, Yunchuan Li, Xubo Yang, and Xiaoxu Meng. source | cite
  • An Integrative View of Foveated Rendering. 0. Bipul Mohanto, ABM Tariqul Islam, Enrico Gobbetti, and Oliver Staadt. source | cite
  • A Simulation System for Scene Synthesis in Virtual Reality. Virtual Reality and Mixed Reality. Jingyu Liu, Claire Mantel, Florian Schweiger, and S{\o}ren Forchhammer. source | cite
  • A Cybersickness Review: Causes, Strategies, and Classification Methods. Journal on Interactive Systems. Thiago Porcino, Daniela Trevisan, and Esteban Clua. source | cite
  • Building Virtual Scene Construction and Environmental Impact Analysis Based on Image Processing. Scientific Programming. Fan Ying and Zhou Bo. source | cite
  • Foveated Rendering Technique in Modern Graphics. University of Maryland, College Park.Xiaoxu Meng. source | cite
  • Time-Warped Foveated Rendering for Virtual Reality Headsets. Computer Graphics Forum. Linus Franke, Laura Fink, Jana Martschinke, Kai Selgrad, and Marc Stamminger. source | cite
  • Foveated Instant Radiosity. 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). Lili Wang, Runze Li, Xuehuai Shi, Ling-Qi Yan, and Zhichao Li. source | cite
  • Foveated Light Culling. Computers & Graphics. Qinqi Yang, Zhuxin Chen, Yanli Liu, Guanyu Xing, and Yanci Zhang. source | cite
  • A Unified Deep Learning Approach for Foveated Rendering \& Novel View Synthesis From Sparse RGB-D Light Fields. 2020 International Conference on 3D Immersion (IC3D). Vineet Thumuluri and Mansi Sharma. source | cite
  • On the Interplay of Foveated Rendering and Video Encoding. 26th ACM Symposium on Virtual Reality Software and Technology. Gazi Karam Illahi, Matti Siekkinen, Teemu Kämäräinen, and Antti Ylä-Jääski. source | cite
  • Modelling Foveated Depth-of-Field Blur for Improving Depth Perception in Virtual Reality. 2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS). Razeen Hussain, Manuela Chessa, and Fabio Solari. source | cite
  • Toward Standardized Classification of Foveated Displays. IEEE Transactions on Visualization and Computer Graphics. Josef Spjut, Ben Boudaoud, Jonghyun Kim, Trey Greer, Rachel Albert, Michael Stengel, Kaan Aksit, and David Luebke. source | cite
  • Learning Foveated Reconstruction to Preserve Perceived Image Statistics. https://arxiv.org/abs/2108.03499. Luca Surace, Marek Wernikowski, Okan Tursun, Karol Myszkowski, Rados{\l}aw Mantiuk, and Piotr Didyk. source | cite
  • State of the Art in Perceptual VR Displays. Real VR –Immersive Digital Reality. Gordon Wetzstein, Anjul Patney, and Qi Sun. source | cite
  • How the Presence and Size of Static Peripheral Blur Affects Cybersickness in Virtual Reality. ACM Transactions on Applied Perception. Yun-Xuan Lin, Rohith Venkatakrishnan, Roshan Venkatakrishnan, Elham Ebrahimi, Wen-Chieh Lin, and Sabarish V. Babu. source | cite
  • Cybersickness Analysis Using Symbolic Machine Learning Algorithms. Universidade Federal Fluminense. Thiago Porcina. source | cite
  • Eye Tracking Data Collection Protocol for VR for Remotely Located Subjects Using Blockchain and Smart Contracts. 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR). Efe Bozkir, Shahram Eivazi, Mete Akgun, and Enkelejda Kasneci. source | cite
  • A Privacy-Preserving Approach to Streaming Eye-Tracking Data. IEEE Transactions on Visualization and Computer Graphics. Brendan David-John, Diane Hosfelt, Kevin Butler, and Eakta Jain. source | cite
  • FAVR - Accelerating Direct Volume Rendering for Virtual RealitySystems. 2020 IEEE Visualization Conference (VIS). Andre Waschk and Jens Kruger. source | cite
  • Focus Guided Light Field Saliency Estimation. 2021 13th International Conference on Quality of Multimedia Experience (QoMEX). Ailbhe Gill, Emin Zerman, Martin Alain, Mikael Le Pendu, and Aljosa Smolic. source | cite
  • Foveated Photon Mapping. IEEE Transactions on Visualization and Computer Graphics. Xuehuai Shi, Lili Wang, Xiaoheng Wei, and Ling-Qi Yan. source | cite
  • Exploiting Object-of-Interest Information to Understand Attention in VR Classrooms. 2021 IEEE Virtual Reality and 3D User Interfaces (VR). Efe Bozkir, Philipp Stark, Hong Gao, Lisa Hasenbein, Jens-Uwe Hahn, Enkelejda Kasneci, and Richard Gollner. source | cite
  • Rendering Optimizations for Virtual Reality Using Eye-Tracking. 2020 22nd Symposium on Virtual and Augmented Reality (SVR). Shawn Matthews, Alvaro Uribe-Quevedo, and Alexander Theodorou. source | cite
  • HPCGen: Hierarchical K-Means Clustering and Level Based Principal Components for Scan Path Genaration. . Wolfgang Fuhl. source | cite
  • Dynamic Viewport Selection-Based Prioritized Bitrate Adaptation for Tile-Based 360\textdegree Video Streaming. IEEE Access. Abid Yaqoob, Mohammed Amine Togou, and Gabriel-Miro Muntean. source | cite
  • Beyond Flicker, Beyond Blur: View-Coherent Metameric Light Fields for Foveated Display. SIGGRAPH Posters. Prithvi Kohli, David R. Walton, Rafael Kuffner dos Anjos, Anthony Steed, and Tobias Ritschel. source | cite
  • Towards Everyday Virtual Reality Through Eye Tracking. der Eberhard Karls Universität Tübingen. Efe Bozkir. source | cite
  • Noise-Based Enhancement for Foveated Rendering. arXiv.2204.04455. Taimoor Tariq, Cara Tursun, and Piotr Didyk. source | cite
  • Practical Saccade Prediction for Head-Mounted Displays: Towards a Comprehensive Model. arXiv.2205.01624. Elena Arabadzhiyska, Cara Tursun, Hans-Peter Seidel, and Piotr Didyk. source | cite
  • Foveated Rendering: Motivation, Taxonomy, and Research Directions. arXiv.2205.04529.Susmija Jabbireddy, Xuetong Sun, Xiaoxu Meng, and Amitabh Varshney. source | cite
  • A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry. Proceedings of the ACM on Human-Computer Interaction. Johannes Meyer, Thomas Schlebusch, and Enkelejda Kasneci. source | cite
  • Visual Perception in Virtual Reality and the Application in Sports. Universität Magdeburg. Stefan Pastel. source | cite
  • Modeling and Optimizing Human-in-the-Loop Visual Perception. immersive computing lab.Qi Sun, Budmonde Duinkharjav, and Anjul Patney. source | cite
  • Stay In Touch