How Researchers Use AI To Give A Classic Game A Photorealistic Makeover

AI GTA V

Video games are becoming better and better, that reality is like the only step to go. But no, graphics in video games are far from what the human eyes can perceive.

And that is because game graphics are made by polygons, which are then rendered so the colors can be calculated to be filled onto them. This is done pixel by pixel, at every frame. Among other things, the whole process, while providing increasingly good results, they don't really resemble the real world.

In other words, graphics in video games can really look convincing, but they are far from photorealism.

To counter that fact, researchers from Intel Labs conducted a research to try to recreate existing graphics from video games, and make them look closer to the real thing, using AI.

The researchers let the AI scan and analyze the game Grand Theft Auto V, to then revamp the graphics of the classic Rockstar game.

Intel research scientist Stephen Richter said that:

"For games, simulations, and films, there is a tremendous amount of work that needs to go into modeling objects, materials, etc. to make them look realistic. Set up the right way, convolutional networks can just learn these things directly from real-world photo collections automatically."

Convolutional networks are well-suited for the job because the deep learning architecture is already commonly used by the researchers for image processing.

As a result, the result is strikingly realistic.

While eyes can be deceiving, and the appearance may seem to resemble a dashcam than a high-quality game, but the the AI managed to tweak the colors and add a number of elements that weren't present in the original graphics.

For example, while the colors are less crisp, but they closer to photorealism. Then there are the reflections that are added to the windows of cars, the introduction of smoother roads and pavements, and lusher textures at the surrounding vegetations.

So how does the AI managed to understand how the real world looks like, and make Grand Theft Auto V look more realistic?

The answer is based on its training datasets.

In one of the tries, the researchers trained the AI using Cityscapes dataset, which is a collection of images recorded primarily in Germany. As a result, the game's vegetations mimic German climate. This is unlike San Andreas, which has a more grey hue.

In another try, the researchers trained the network using a more diverse Mapillary Vistas dataset, and this resulted in an environment that was a lot more brighter and more vibrant.

"Cityscapes was recorded with an automotive-grade camera, which has this characteristic green tint. Consequently, images enhanced to look like Cityscapes also get this green tint. Vistas was recorded with a diverse set of cameras, including, for example, smartphone cameras. Images from Vistas are much more vibrant and you can see this in the results by our method," said Richter.

This was where the researchers made tweaks and adjustments.

The researchers admitted that their method still requires further optimization for real-time applications. However, they envisioned the technology being integrated into game engines to advance their level of realism.

Published: 
31/05/2021