The technology behind generative AI is large-language model (LLM). The technology resides comfortably at the intersection of traditional AI and natural language processing.
Trained on massive datasets, an LLM can understand and produce coherent and contextually relevant responses.
It works by using attention mechanisms, allowing itto consider different parts of the input text when generating output.
While the technology has the word "generative" in it, meaning that it can create new content, such as writing paragraphs, answering questions, or even generating creative pieces, the information is not new.
In other words, the many chatbots created by LLMs that have proven themselves extremely popular, do not generate new knowledge.
Making things worse, AIs like them are prone to confabulation, leading to answers that are fluently plausible but badly flawed.
But this time, in their report, AI scientists claimed to have made an "exciting" discovery when using chatbots to solve mathematics.
They claimed to have made the world’s first scientific discovery using a LLM, saying that the technology behind chatbots like ChatGPT and Bard, can actually generate information that goes beyond human knowledge
The team at Google DeepMind was investigating whether large language models, which underpin modern chatbots such as OpenAI’s ChatGPT and Google’s Bard, can do more than repackaging existing information they learned during training.
This was when they came up with their findings.
"When we started the project there was no indication that it would produce something that’s genuinely new," said Pushmeet Kohli, the head of AI for science at DeepMind.
"As far as we know, this is the first time that a genuine, new scientific discovery has been made by a large language model."
At first, the researchers built what they call the 'FunSearch,' or short for 'searching in the function space.'
Then, the team harnessed an LLM to write solutions to problems in the form of computer programs. To do this, the LLM is paired with an "evaluator" that automatically ranks the programs by how well they perform. The best programs are then combined and fed back to the LLM to be improved.
This method allowed the researchers to steadily evolve poor-performing programs into better ones.
Eventually, newer programs become so much better that they can discover new knowledge on their own.
To prove this theory, the researchers set FunSearch loose on two puzzles.
The first was a longstanding and somewhat arcane challenge in pure mathematics known as the cap set problem. It deals with finding the largest set of points in space where no three points form a straight line. FunSearch churned out programs that generate new large cap sets that go beyond the best that mathematicians have come up with.
"FunSearch generated solutions - in the form of programs - that in some settings discovered the largest cap sets ever found. This represents the largest increase in the size of cap sets in the past 20 years. Moreover, FunSearch outperformed state-of-the-art computational solvers, as this problem scales well beyond their current capabilities," the researchers said.
The second puzzle was the bin packing problem, which looks for the best ways to pack items of different sizes into containers. While it applies to physical objects, such as the most efficient way to arrange boxes in a shipping container, the same maths applies in other areas, such as scheduling computing jobs in datacentres.
The problem is typically solved by either packing items into the first bin that has room, or into the bin with the least available space where the item will still fit.
FunSearch found a better approach that avoided leaving small gaps that were unlikely ever to be filled.
“In the last two or three years there have been some exciting examples of human mathematicians collaborating with AI to obtain advances on unsolved problems,” said Sir Tim Gowers, professor of mathematics at Cambridge University, who was not involved in the research.
“This work potentially gives us another very interesting tool for such collaborations, enabling mathematicians to search efficiently for clever and unexpected constructions. Better still, these constructions are humanly interpretable.”
Large language models, or LLMs, are powerful neural networks that learn the patterns of language, including computer code, from vast amounts of text and other data.
Since the whirlwind arrival of ChatGPT, the technology has debuted both astounding results and a questionable, dystopian future.
And this breakthrough suggests the technology behind ChatGPT and similar programs can generate information that goes beyond human knowledge.
“This is actually going to be transformational in how people approach computer science and algorithmic discovery,” said Kohli. “For the first time, we’re seeing LLMs not taking over, but definitely assisting in pushing the boundaries of what is possible in algorithms.”
Jordan Ellenberg, professor of mathematics at the University of Wisconsin-Madison, and co-author on the paper, said: “What I find really exciting, even more so than the specific results we found, is the prospects it suggests for the future of human-machine interaction in math.
“Instead of generating a solution, FunSearch generates a program that finds the solution. A solution to a specific problem might give me no insight into how to solve other related problems. But a program that finds the solution, that’s something a human being can read and interpret and hopefully thereby generate ideas for the next problem and the next and the next.”
"FunSearch demonstrates that if we safeguard against LLMs’ hallucinations, the power of these models can be harnessed not only to produce new mathematical discoveries, but also to reveal potentially impactful solutions to important real-world problems," the researchers said.
