Nvidia Instant NeRF: A Tool that Turns 2D Snapshots into a 3D-Rendered Scene

Nvidia Instant NeRF uses neural networks to render realistic 3D scenes based on an input collection of 2D images

Nvidia's AI model is pretty impressive: a tool that quickly turns a collection of 2D snapshots into a 3D-rendered scene. The tool is called Instant NeRF, referring to "neural radiance fields". 

Known as inverse rendering, the process uses AI to approximate how light behaves in the real world, enabling researchers to reconstruct a 3D scene from a handful of 2D images taken at different angles. The Nvidia research team has developed an approach that accomplishes this task almost instantly making it one of the first models of its kind to combine ultra-fast neural network training and rapid rendering.

NeRFs use neural networks to represent and render realistic 3D scenes based on an input collection of 2D images.

Collecting data to feed a NeRF is a bit like being a red carpet photographer trying to capture a celebrity's outfit from every angle. The neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of those shots.

In a scene that includes people or other moving elements, the quicker these shots are captured, the better. If there's too much motion during the 2D image capture process, the AI-generated 3D scene will be blurry.

From there, a NeRF essentially fills in the blanks, training a small neural network to reconstruct the scene by predicting the color of light radiating in any direction, from any point in 3D space. The technique can even work around occlusions when objects seen in some images are blocked by obstructions such as pillars in other images.

Nvidia Instant NeRF, however, cuts rendering time by several orders of magnitude. It relies on a technique developed by Nvidia called multi-resolution hash grid encoding, which is optimized to run efficiently on Nvidia GPUs. Using a new input encoding method, researchers can achieve high-quality results using a tiny neural network that runs rapidly.

The model was developed using the Nvidia CUDA Toolkit and the Tiny CUDA Neural Networks library. Since it's a lightweight neural network, it can be trained and run on a single Nvidia GPU running fastest on cards with Nvidia Tensor Cores.