Claude 3 Opus, the next-generation artificial intelligence model from Anthropic has taken the top spot on the Chatbot Arena leaderboard, pushing OpenAI’s GPT-4 to second place for the first time since it launched last year.
Unlike other forms of benchmarking for AI models, the LMSYS Chatbot Arena relies on human votes, with people blind-ranking the output of two different models to the same prompt.
OpenAI’s various GPT-4 versions have held the top spot for so long that any other model coming close to its benchmark scores is known as a GPT-4-class model. Maybe we need to introduce a new Claude-3 class model for future rankings.
It is worth noting that the score between Claude 3 Opus and GPT-4 is very close, and the OpenAI model has been out for a year, with the “markedly different” GPT-5 expected at some point this year — so Anthropic may not hold the position for long.
What is the chatbot arena?
The Chatbot Arena is run by LMSys, the Large Model Systems Organization, and features a wide variety of large language models fighting it out in anonymous randomized battles.
First launched in May last year, it has collected more than 400,000 user votes with models from Anthropic, OpenAI and Google filling most of the top ten throughout that time.
Recently other models from French AI startup Mistral and Chinese companies like Alibaba have started to take more of the top spots and open source models are increasingly present.
| Rank | Model | Elo | Votes |
|---|---|---|---|
| 1 | Claude-3 Opus | 1253 | 33250 |
| 1 | GPT-4-1106-Preview | 1251 | 54141 |
| 1 | GPT-4-0125-preview | 1248 | 34825 |
| 4 | Gemini Pro | 1203 | 12476 |
| 4 | Claude-3 Sonnet | 1198 | 32761 |
| 6 | GPT-4-0314 | 1185 | 33499 |
| 7 | Claude-3 Haiku | 1179 | 18776 |
| 8 | GPT-4-0613 | 1158 | 51860 |
| 8 | Mistral-Large-2402 | 1157 | 26734 |
| 9 | Qwen1.5-72B-Chat | 1148 | 20211 |
| 10 | Claude-1 | 1146 | 21908 |
| 10 | Mistral Medium | 1145 | 26196 |
It uses the Elo rating system which is widely used in games such as chess to calculate the relative skill levels of players. Unlike in chess, this time the ranking is applied to the chatbot and not to the human using the model.
There are limitations to the arena as not all models or versions of models are included, sometimes users find GPT-4 models won’t load, and it can favor models with live internet access such as Google Gemini Pro.
The arena is also missing some high profile models such as Google's Gemini Pro 1.5 with its massive context window and Gemini Ultra.
Claude 3 Haiku might be GPT-4-level
[Arena Update]70K+ new Arena votes🗳️ are in!Claude-3 Haiku has impressed all, even reaching GPT-4 level by our user preference! Its speed, capabilities & context length are unmatched now in the market🔥Congrats @AnthropicAI on the incredible Claude-3 launch!More exciting… pic.twitter.com/p1Guuf0B3KMarch 26, 2024
More than 70,000 new votes made up the latest update that saw Claude 3 Opus take the top spot of the leaderboard, but even the smallest of the Claude 3 models performed well.
LMSYS explained: “Claude-3 Haiku has impressed all, even reaching GPT-4 level by our user preference! Its speed, capabilities & context length are unmatched now in the market.”
What makes this even more impressive is that Claude 3 Haiku is the “local size” model, comparable to Google’s Gemini Nano. It is achieving impressive results without the huge trillion plus parameter scale of Opus or any of the GPT-4-class models.
While not as intelligent as Opus or Sonnet, Anthropic's Haiku is significantly cheaper, much faster and as the arena results suggest — as good as much larger models on blind-tests.
All three Claude 3 models are in the top ten with Opus in the top spot, Sonnet at joint fourth with Gemini Pro and Haiku in join sixth with an earlier version of GPT-4.
A win for closed AI models
Not going to beat centralized AI with more centralized AI.All in on #DecentralizedAI Lots more 🔜 https://t.co/SbEF5zoo05March 23, 2024
All but three of the top 20 large language models in the arena leaderboard are proprietary, suggesting open source has some work to do to reach the big players.
Meta, which is heavily focused on open source AI, is expected to release Llama 3 in the next few months which will likely enter in the top ten as it is expected to be similar in ability to Claude 3 — after all Meta has 300,000 + Nvidia H100 GPUs to train it on.
We’re also seeing other moves in open source and decentralized AI with StabilityAI founder Emad Mostaque stepping back from CEO duties to focus on more distributed and accessible artificial intelligence. He said you can’t beat centralized AI with more centralized AI.
More from Tom's Guide
- Google’s new VLOGGER AI lets you create a lifelike avatar from just a photo
- Apple could bring Google Gemini to the iPhone for AI tasks
- I gave Claude, ChatGPT and Gemini a photo of ingredients to see which came up with the best recipe
