Tuesday , April 20 2021

DeepFovea: Neural Reconstruction for Foveated Rendering and Video Compression using Learned Statistics of Natural Videos – Facebook Research, Hacker News


Abstract

In order to provide an immersive visual experience, modern displays require head mounting, high image resolution, low latency, as well as high refresh rate. This poses a challenging computational problem. On the other hand, the human visual system can consume only a tiny fraction of this video stream due to the drastic acuity loss in the peripheral vision. Foveated rendering and compression can save computations by reducing the image quality in the peripheral vision. However, this can cause noticeable artifacts in the periphery, or, if done conservatively, would provide only modest savings. In this work, we explore a novel foveated reconstruction method that employs the recent advances in generative adversarial neural networks. We reconstruct a plausible peripheral video from a small fraction of pixels provided every frame. The reconstruction is done by finding the closest matching video to this sparse input stream of pixels on the learned manifold of natural videos. Our method is more efficient than the state-of-the-art foveated rendering, while providing the visual experience with no noticeable quality degradation. We conducted a user study to validate our reconstruction method and compare it against existing foveated rendering and video compression techniques. Our method is fast enough to drive gaze-contingent head-mounted displays in real time on modern hardware. We plan to publish the trained network to establish a new quality bar for foveated rendering and compression as well as encourage follow-up research.

Brave Browser
Read More
Payeer

About admin

Check Also

A neural network picks promising antibiotics out of a library of chemicals, Ars Technica

A neural network picks promising antibiotics out of a library of chemicals, Ars Technica

AI on drugs — Evaluating over a million molecules takes four days of computer time. John Timmer - Feb 22, 2020 6:49 pm UTC Enlarge / Here, have some antibiotics. Biochemists have had some success at designing drugs to match specific targets. But much of drug development remains an arduous grind, screening hundreds to thousands…

Leave a Reply

Your email address will not be published. Required fields are marked *