Humans have a way to perceive the world in 3D, despite the eyes can only understand 2D.
This happens because the brain is able to learn how the eyes see the depth of things. By having the eyes at the front, humans have learned depth perception by putting together several 2D images together to create extrapolate depth.
This is called stereoscopic vision.
Computers don't have eyes. They learn vision from their sensors, and learn their surroundings about what's what, from data it has been trained with.
Nvidia is the company known for designing graphics processing units (GPUs) for the gaming and professional markets, as well as system on a chip units (SoCs) for the mobile computing and automotive market.
This time, Nvidia has created an AI that can literally mimic the human brain's ability to understand depth.
The method used in called NeRF, which is an acronym for Neural Radiance Field.
Modern NeRF models are quick at rendering, but require a lot of training time
Traditionally, depending on the depth required, computers usually take hours or days to compute the results.
But Nvidia's version of NeRF here, can do that in mere seconds.
This is why Nvidia dubbed it the "instant NeRF."
To do this, the technology uses artificial intelligence (AI) to create a 3D scene from a set of still images, in an approach called "reverse rendering."
All the AI needs, is a brief moment to learn from a dozen of still images taken from multiple angles, and a few more tens of milliseconds to render a 3D rendered scene.
Since the process is the exact opposite of taking a Polaroid, which instantly converts a 3D scene into a 2D image, Nvidia recreated an image of Andy Warhol using Polaroid.
Warhol is known to have produced silkscreens using photographs taken mostly from a specific model of Polaroid camera.
This approach in creating his snapshots had a great effect on artistic photography.
Nvidia here, is paying s a homage to the famous photographer and artist, using a would-be Warhol in a video to showcase its NeRF.
Read: This '3D MoMa' AI From Nvidia Is Able To Turn 2D Images Into 3D, Efficiently
According to Nvidia in a blog post:
"Collecting data to feed a NeRF is a bit like being a red carpet photographer trying to capture a celebrity’s outfit from every angle — the neural network requires a few dozen images taken from multiple positions around the scene, as well as the camera position of each of those shots."
Nvidia's Instant NeRF can be so quick because it uses a technique called the 'multi-resolution hash grid encoding'.
This technique creates a number of small neural networks that can be trained in a shorter window. Nvidia said that the whole process can work with a single GPU, which means it's not a particularly resource-intensive and can be applied on products that can run on consumer-grade PCs.
This kind of technology can be used to train robots to better understand depth and spatial awareness, improving the AI in self-driving cars, help turn 2D assets into 3D objects for virtual worlds, and lots more.
Furthermore, what the AI does that other methods couldn't do quickly, is filling the blanks where necessary.
NeFR creates a 3D image from these dozens of different angles, and can even make up for blockages.
Like for example, NeRF can complete a scene when an object is obscured in one of the photos. the AI can even fill that angle even if it cannot see the subject well or at all.
Its only major weakness, is applying its technique to moving objects.
"In a scene that includes people or other moving elements, the faster these shots are, the better," explained Nvidia. "If there is a lot of movement during the process of capturing the 2D image, the 3D scene generated by AI will be blurry."