A new study suggests that sudden jumps in LLMs’ abilities are neither surprising nor unpredictable, but are actually the consequence of how we measure ability in AI.
Two years ago, in a project called the Beyond the Imitation Game benchmark, or BIG-bench, 450 researchers compiled a list of 204 tasks designed to test the capabilities of large language models, which power chatbots like ChatGPT. On most tasks, performance improved predictably and smoothly as the models scaled up—the larger the model, the better it got. But with other tasks, the jump in ability wasn’t smooth. The performance remained near zero for a while, then performance jumped.
But the Stanford researchers point out that the LLMs were judged only on accuracy: Either they could do it perfectly, or they couldn’t. So even if an LLM predicted most of the digits correctly, it failed. That didn’t seem right. If you’re calculating 100 plus 278, then 376 seems like a much more accurate answer than, say, −9.34. So instead, Koyejo and his collaborators tested the same task using a metric that awards partial credit.
United States Latest News, United States Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
Why large language models aren’t headed toward humanlike understandingUnlike people, today's generative AI isn’t good at learning concepts that it can apply to new situations.
Read more »
Revolutionizing Software Development With Large Language ModelsSon Nguyen is the co-founder & CEO of Neurond AI, a company providing world-class Artificial Intelligence and Data Science services. Read Son Nguyen's full executive profile here.
Read more »
Language structures yield good human and computational language models.Similarities and differences between how humans and computers process language don't lie in the architecture of the processing system but in the language system itself.
Read more »
Are Advanced Language Models Hinting at Machine Consciousness?Advanced large language models (LLMs) are exhibiting behaviors and capabilities that suggest the possibility of genuine machine consciousness similar to human subjectivity.
Read more »
'Emergent gravity' could force us to rewrite the laws of physicsPaul M. Sutter is a research professor in astrophysics at SUNY Stony Brook University and the Flatiron Institute in New York City. He regularly appears on TV and podcasts, including 'Ask a Spaceman.' He is the author of two books, 'Your Place in the Universe' and 'How to Die in Space,' and is a regular contributor to Space.
Read more »
This ‘weapon’ can wipe the AI slate cleanAI booming at current levels raises concerns about large language models (LLMs) being used for harmful purposes like developing weapons.
Read more »