Humans Have Almost Reached The Limit Of Modern Computers, Said AI Researchers

Modern computers are powerful, and there is no doubt about it. But when it comes to AI, their capacity is almost reaching the maximum potential for creating state-of-the-art deep learning technology.

According to a group of MIT researchers who conducted an audit of more than 1,000 pre-print papers on arXiv, humans are basically running out of compute.

The researchers claimed that humans will soon reach to a point where it's no longer economically or environmentally feasible to continue scaling deep learning systems.

According to the team's paper:

"Progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods."
Implications of achieving performance benchmarks on the computation, carbon emissions, and economic costs from deep learning based on projections from polynomial and exponential models
Implications of achieving performance benchmarks on the computation, carbon emissions, and economic costs from deep learning based on projections from polynomial and exponential models. (Credit: MIT)

While researchers can indeed use AI frameworks to create and run neural networks on GPUs and ordinary PCs, but training large-scale models is what the researchers are concerning.

To train large-scale AI models, researchers need a much more powerful systems. These hardware is power hungry and expensive. Despite tweaked algorithms and dedicated hardware can help, but they still have limits.

For example, researchers that use OpenAI's GPT-2 text generator to create a sophisticated AI, they'll be spending a lot of money, and will create so much carbon footprint that they will cause serious damage to the environment.

The above chart, a screenshot from the MIT team’s research paper, shows what popular deep learning systems like ImageNet cost us in terms of environmental, computational, and financial expenditure.

Based on trends, the researchers feel that humans will soon reach that point where achieving further benchmarks and milestones will no longer be cost-effective.

"The hardware, environmental, and monetary costs would be prohibitive. And enormous effort is going into improving scaling performance," the researchers said.

[block:block=87]
"We show that computational requirements have escalated rapidly in each of these domains and that these increases in computing power have been central to performance improvements. If progress continues along current lines, these computational requirements will rapidly become technically and economically prohibitive."

While the relationship between performance, model complexity, and computational requirements in deep learning is still not well understood theoretically. Nevertheless, there are important reasons to believe that deep learning is intrinsically more reliant on computing power than other techniques.

This is because of the the role of overparameterization and how this scales as additional training data are used to improve performance.

"It has been proven that there are significant benefits to having a neural network contain more parameters than there are data points available to train it, that is, by overparameterizing it," the researchers said.

The researchers concluded that the likely solutions for these computational limits is to force deep learning towards less computationally-intensive methods of improvement, and push machine learning towards techniques that are more computationally-efficient than deep learning

They also suggest that quantum computing could help address this issue.

"Of these, quantum computing is the approach with perhaps the most long-term upside, since it offers a potential for sustained exponential increases in computing power," the researchers added.