AI is only as good as the data is has been trained on. But OpenAI shows that its generative AI is indeed smart.
When the company unveiled ChatGPT, its generative AI quickly captivated the world. Sooner than later, it made rivals scramble for their own solutions, people make it help them with various tasks, and researchers tinkering with it to see its innerworkings.
While there is standard to benchmark a generative AI, and it's even said that the world-renown method test from Alan Turing is no longer relevant because of the technology, a study finds that the product from ChatGPT is next-level.
According to a research from the University of Montana (UM) and its partners, it's suggested that the AI can match the top 1% of human thinkers on a standard test for creativity.
To come to this conclusion, the study that was directed by Dr. Erik Guzik, an assistant clinical professor in UM's College of Business, used the Torrance Tests of Creative Thinking (TTCT ), a well-known tool to assess human creativity.
The researchers submitted eight responses generated by GPT-4-powered ChatGPT, and also submitted answers from a control group of 24 UM students taking Guzik's entrepreneurship and personal finance classes.
These scores were then compared with 2,700 college students nationally who took the TTCT in 2016, and scored by Scholastic Testing Service, which didn't know AI was involved.
The results placed ChatGPT high up.
According to the conclusion, the AI was able to achieve remarkable fluency, which is the ability to generate a large volume of ideas, and originality, which is the ability to come up with new ideas.
ChatGPT only scored bad on flexibility, which is the ability to generate different types and categories of ideas.
"For ChatGPT and GPT-4, we showed for the first time that it performs in the top 1% for originality," Guzik said. "That was new."
What this literally means, ChatGTP outperformed the vast majority of college students.
Read: OpenAI Announces 'GPT 4', A More Powerful, More Creative, But Safer GPT
"We were very careful at the conference to not interpret the data very much," Guzik said. "We just presented the results. But we shared strong evidence that AI seems to be developing creative ability on par with or even exceeding human ability."
Guzik said the TTCT test uses prompts that mimic real-life creative tasks.
"Let's say it's a basketball," he said. "Think of as many uses of a basketball as you can. You can shoot it in a hoop and use it in a display. If you force yourself to think of new uses, maybe you cut it up and use it as a planter. Or with a brick you can build things, or it can be used as a paperweight. But maybe you grind it up and reform it into something completely new."
But it's worth noting, that the TTCT is protected proprietary material, meaning that OpenAI has never trained ChatGPT with the data.
Because the material isn't available anywhere, the AI also couldn't "cheat" by accessing information about the test on the internet or in a public database.
Guzik and his team started tinkering with the creativity of ChatGPT after fiddling with it since 2022.
"We had all been exploring with ChatGPT, and we noticed it had been doing some interesting things that we didn't expect," he said. "Some of the responses were novel and surprising. That's when we decided to put it to the test to see how creative it really is."
"I think we know the future is going to include AI in some fashion," Guzik said. "We have to be careful about how it's used and consider needed rules and regulations. But businesses already are using it for many creative tasks. In terms of entrepreneurship and regional innovation, this is a game changer."
In another test, initiated by University of California (UCLA), which tested GPT-3, the 'inferior GPT', it's revealed that with it, ChatGPT can solve problems as well, and also better than college students.
The researchers tested ChatGPT on reasoning problems found in intelligence tests and exams, like the SAT.
This was done because SAT is a standardized test that is used for college admissions in the U.S.. It is a multiple-choice test that measures a student's reading, writing, and math skills, which then score participants on a scale of 400 to 1600, with an average score of 1060.
According to reports, the AI performed impressively well, answering 80% of all questions correctly.
What this means, it effectively outperformed the average score of human participants.
In yet another test, the researchers also challenged the AI model with some SAT analogy questions, where it had to link pairs of words. These questions weren't available on the internet, meaning that the AI couldn't cheat, and still it did better than the average human.
However, there were areas where GPT-3 struggled with social interactions, math reasoning, and problems that required understanding physical space.
While overall, the study found GPT-3's exceptional problem-solving abilities, the researchers concluded that ChatGPT did not reach the level of human-level intelligence or artificial general intelligence.
