Since Meta released its new model, Llama 3.1 405B, the tech world has been buzzing with excitement. After a leak of Llama 3.1, Meta officially launched the Llama 3.1 405B, an advanced open-source AI model, along with its 70B and 8B versions. They also upgraded the existing 8B and 70B models.
It is the first openly available model that rivals the top AI models in terms of state-of-the-art capabilities in general knowledge, steerability, math, tool use, and multilingual translation.
As Meta CEO Mark Zuckerberg mentioned in a post, Meta’s long-term vision is to build general intelligence, open-source it responsibly, and make it widely available so everyone can benefit.
“I believe we should fear people who think open-source is dangerous when, in fact, open-source is the foundation for all greatness,” said Zuckerberg.
The Llama 3.1 model is reported to outperform GPT-4. Let’s examine the various parameters where Llama 3.1 excels and surpasses GPT-4.
Compare Llama 3.1 Vs GPT 4o
Availability Comparison
Meta Llama 3.1 is an open-source model, making it freely available for download and development. This aligns with Meta’s commitment to open-source innovation, allowing developers to use and modify the model without restrictions.
Additionally, the model can be downloaded from platforms like Hugging Face and Meta’s own distribution channels, making it widely accessible for developers and researchers.
Meanwhile, GPT-4o is a closed-source model. Users can access GPT-4 through APIs provided by OpenAI, but they cannot customise or fine-tune the model in the same way as with open-source models like Llama 3.1.
Benchmark Performance Comparison
The benchmark performance highlighted vital areas such as math reasoning, coding, and common sense reasoning. In the Math Reasoning (GSM8K benchmark), Llama 3.1 scored 96.82%, outperforming GPT-4o, which scored 94.24%, demonstrating superior capability in grade school math reasoning tasks.
In contrast, for the Coding (HumanEval benchmark), GPT-4o excelled with a score of 92.07%, surpassing Llama 3.1’s 85.37%, indicating better performance in coding tasks.
Regarding Common Sense Reasoning (Winograde benchmark), Llama 3.1 again showed its strength with a score of 86.74%, compared to GPT-4o’s 82.16%.
Cost Efficiency Comparison
Meta claims that operating Llama 3.1 in production costs approximately 50% less than using GPT-4. This cost advantage is particularly appealing for organizations looking to implement AI solutions without incurring hefty operational expenses associated with proprietary models.
While closed models are generally more cost-effective, Llama models offer some of the lowest costs per token in the industry, according to testing by Artificial Analysis.
Pricing Comparison
According to a prediction analysis done by artificial analysis, Llama 3.1 405B is expected to be positioned as a more cost-effective alternative to current frontier models like GPT-4o and Claude 3.5 Sonnet, offering similar quality at a lower price.
Providers will likely offer FP16 and FP8 versions at different price points. FP16 requires 2x DGX systems with 8xH100s for operation.
The FP8 versions of Llama 3.1 405B may become the more prominent offering, potentially delivering frontier-class intelligence at prices between $1.50 and $3 (blended 3:1). Projections suggest that FP16 will be priced between $3.5 and $5 (blended 3:1). At the same time, FP8 will range from $1.5 to $3.
Multilingual Capabilities Comparison
Llama 3.1 is designed to handle conversations in multiple languages, including Spanish, Portuguese, Italian, German, Thai, French, and Hindi. This broad multilingual support enhances its utility for various global applications and diverse user bases.
GPT-4o demonstrates superior language comprehension, particularly in complex contexts and nuanced language use. It employs advanced techniques for ambiguity handling and context-switching, which allows it to maintain coherence in more extended conversations.
Final Take
When compared, both models present a win-win scenario. Reddit conversations and discussions are ongoing about which model is better. Users are discussing the limitations of running the 405B model locally and the potential for improved models through continued training.
Additionally, a user mentioned on Reddit that GPT 4o has a significant advantage with its new voice and vision feature, which is very realistic and fast. No other model has come remotely close to the realism and response time showcased in the demos. This feature is essential because it’s the future of how people will interact with chatbots.
