UHG
Search
Close this search box.

Open-Source LLM360 Unveiled by Cerebras Systems, Petuum and MBZUAI

LLM360 releases Amber and CrystalCoder models that are built on Meta’s Llama architecture

Share

AI supercomputer company Cerebras Systems, AI company Petuum, and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) launched LLM360, a framework for creating open-source large language models (LLMs). Developed in partnership with MBZUAI’s Institute of Foundation Models, LLM360 empowers developers by providing detailed insights and methodologies, promising to simplify, expedite, and reduce costs in the development of LLMs. 

Two open-source large language models are released : Amber, a 7 billion parameter English-language model trained on 1.2 trillion tokens, and CrystalCoder, a 7 billion parameter model, trained on 1.4T tokens designed for English language and coding tasks. Both the models are released under the Apache 2.0 license. There is also another model Diamond with 65 billion parameters which is set to release soon. These models are trained on the Condor Galaxy 1 supercomputer, built by G42 and Cerebras systems. 

Both the models are built on Meta’s LLaMA architecture and Amber is said to perform similarly to LLaMA-7B, OpenLLaMA-v2-7B and outperforms Pythia-6.7B. 

Source: LLM360 Blog

CrystalCoder undergoes meticulous training, incorporating a thoughtful blend of text and code data to enhance its effectiveness in both domains. Notably, the introduction of code data occurs early in the pretraining stage, distinguishing it from Code Llama 2, which relies solely on code data during fine-tuning on Llama 2. Furthermore, CrystalCoder is specifically trained on Python and web programming language, strategically designed to elevate its capabilities as a programming assistant.

UAE Heading Towards AI Dominance

With the recent AI developments, UAE is working towards becoming an AI superpower. Following TII’s Falcon and demographic-specific Jais large language model, UAE has been also rallying for open-source models to promote research initiatives. With the recent AI company, A171 that was launched a few weeks ago, UAE looks to even take on AI giant OpenAI. 

📣 Want to advertise in AIM? Book here

Picture of Vandana Nair

Vandana Nair

As a rare blend of engineering, MBA, and journalism degree, Vandana Nair brings a unique combination of technical know-how, business acumen, and storytelling skills to the table. Her insatiable curiosity for all things startups, businesses, and AI technologies ensures that there's always a fresh and insightful perspective to her reporting.
Related Posts
19th - 23rd Aug 2024
Generative AI Crash Course for Non-Techies
Upcoming Large format Conference
Sep 25-27, 2024 | 📍 Bangalore, India
Download the easiest way to
stay informed

Subscribe to The Belamy: Our Weekly Newsletter

Biggest AI stories, delivered to your inbox every week.

Flagship Events

Rising 2024 | DE&I in Tech Summit
April 4 and 5, 2024 | 📍 Hilton Convention Center, Manyata Tech Park, Bangalore
Data Engineering Summit 2024
May 30 and 31, 2024 | 📍 Bangalore, India
MachineCon USA 2024
26 July 2024 | 583 Park Avenue, New York
MachineCon GCC Summit 2024
June 28 2024 | 📍Bangalore, India
Cypher USA 2024
Nov 21-22 2024 | 📍Santa Clara Convention Center, California, USA
Cypher India 2024
September 25-27, 2024 | 📍Bangalore, India
discord-icon
AI Forum for India
Our Discord Community for AI Ecosystem, In collaboration with NVIDIA.