Sony Created AI That Can Beat Top Human Drivers In Gran Turismo Racing Game

In the past, researchers have managed to create various AIs that best humans in a number of games.

From the classic game of Go and Chess, to more complex games like StarCraft II., among others.

This time, machines can claim a new high score on the popular racing video game series Gran Turismo.

Sony announced that its researchers have partnered with Gran Turismo's developer Polyphony Digital, and created an AI driver named 'GT Sophy' that is considered "reliably superhuman."

According to the company, the AI is able to defeat some of the best human drivers in Gran Turismo Sport in back-to-back laps.

Racing in motorsport involves driving a vehicle as fast as possible, with the best reaction time and knowledge of the track and the vehicle being driven.

Beyond that, there are tactics, strategy, and the somewhat nebulous concept of etiquette.

As described in a paper published in Nature, the researchers said that racing is not easy.

"Automobile racing is a domain that poses exactly these challenges; it requires real-time control of vehicles with complex, non-linear dynamics while operating within inches of opponents."

Some drivers might have limited success through aggression and going for every overtaking opportunity they see. But knowing where to pass and when to wait for a better opportunity, is also important. And knowing all that is needed to defeat rivals.

This is why the popularity of automotive racing didn't come only by the performance of the cars themselves, but also from the skills and the daring displayed by the drivers as they push those high-performance cars to the limit.

And the game Gran Turismo is a simulation style racing game that simply replicates all of that.

Using vehicular behavior physics as its key factor in the experience, drivers (players) need to know how to properly corner and maneuver the car they are driving. With the many cars to choose from, the experience and the knowledge to be learned, are vast.

In other words, winning a race is much more than following the race line.

And top drivers in the game, are people with high grade of driving skills.

To beat those 'professional' racers. GT Sophy was trained using a method known as reinforcement learning.

This is a form of trial-and-error training, in which the AI agent is thrown into an environment with no instructions and rewarded when it managed to accomplish certain goals.

For example, the AI was rewarded according to how well it progressed along the track since the last observation, and it was given a bonus for passing another car. Penalties were given to the AI whenever it cut corners, hit walls, or slip the car's tires. It also had to avoid rear-ending other cars or have any other collisions.

The technology was developed to employ Sony’s cloud gaming infrastructure for the training, where it focuses on three areas: Physical Realism, Realtime Race Tactics, and Sports Etiquette

The researchers then fine-tuned the criteria, in order to shape a driving style that is aggressive enough to win but not to bully others.

The so-called "AI" in games, like non-playable characters, or in this case, opponent racers, aren't real AIs.

Those are only computer programs that know everything in advance, and has everything programmed, even before they race. GT Sophy on the other hand, didn't know how to drive to begin with.

Just like normal players in Gran Turismo, it needed to learn the tracks, how to control the car it is given, and master how to drive properly in order to win.

To eliminate some innate advantages of computers, like having perfect reaction times, the researchers at Sony had to cap GT Sophy's input by 10Hz, compared to a theoretical maximum human input of 60Hz. The researchers also added artificial delay, which makes GT Sophy's reaction time to average at a few hundred milliseconds, which is comparable to human's.

Using reinforcement learning, GT Sophy was able to navigate round a race track with just a few hours of training, and "within a day or two," it was faster than 95% of all drivers in its training dataset.

After some 45,000 total hours of training, GT Sophy was able to achieve superhuman performance on three tracks in Gran Turismo: Dragon Trail Seaside, Lago Maggiore GP, and Circuit de la Sarthe.

GT Sophy was tested against a trio of top e-sport drivers: Emily Jones, Valerio Gallo, and Igor Fraga.

“It was really interesting seeing the lines where the AI would go, there were certain corners where I was going out wide and then cutting back in, and the AI was going in all the way around, so I learned a lot about the lines,” said Jones. “Going into turn 1, for example, I was braking later than the AI, but the AI would get a much better exit than me and beat me to the next corner. I didn’t notice that until I saw the AI and was like, ‘Okay, I should do that instead.’”

"We tend to sacrifice a little bit of the entry to make the car be in a better position for the exit, but the AI seems to be able to carry more speed into the corner but still be able to have the same kind of exit, or even a faster exit," said Fraga. "The AI can create this type of line a lot quicker than us... It was not a possibility before because we never realized it. But the AI was able to find it for us."

"The ghost is always a reference. Even when I train, I always use someone else’s ghost to improve," said Gallo.

Sony and Polyphony Digital also held head-to-head races between four instances of GT Sophy and four top-level Japanese GT: Sport racers: Takuma Miyazono, Tomoaki Yamanaka, Ryota Kokubun, and Shotaro Ryu, at the developer's headquarters in Tokyo.

On this occasion, the humans won 86 to 70.

Later, after some tweaks and modifications to the training engine, a rematch between GT Sophy and the Japanese racers reversed the outcome, with the AI easily defeated the human team 104 to 52.

The researchers said that, "as long as it is exposed to the right conditions, the agent learns to do the right thing by trial and error."

They also pointed out that GT Sophy's strategic decision-making is still far from perfect, and that there is room for improvement.

With this breakthrough, experts in both video game racing and AI said that GT Sophy’s success is a significant.

The AI managed to not only know what human players have overlooked, but also mastered tactics and strategy needed to win.

“We are pursuing beauty, and we want a positive effect on society like a bright light shining,” said Gran Turismo creator Kazunori Yamauchi during the presentation. “What kind of AI can we design so it can give joy to people – it has to drive fast, and it has to drive simple. The AI ​​agency always has to handle the various environments, and it has to appear natural. With conventional rule -based AI, this could not be done – it would only react to a certain condition and follow the same rules, so people could detect it’s a machine and not a human.

The researchers do noted that GT Sophy does have some advantages compared to human players, like a precise knowledge of the map with coordinates of track boundaries and "precise information about the load on each tire, slip angle of each tire, and other vehicle state."

Realizing the potential of this AI, Sony said that it is planning to integrate GT Sophy into future Gran Turismo titles.

It's worth noting that GT Sophy was not created to be faster or better than human players. Instead, Kazunori Yamauchi, President of Polyphony Digital, explained that, "The goal with Gran Turismo Sophy is ultimately to entertain people."

More or less, GT Sophy is meant to be a way to push human players to drive faster, or help them learn different race tracks or new techniques to be faster for specific sections of tracks

Previously, researchers managed to create an AI that drives faster than 50,000 human drivers in Gran Turismo Sport