Falcon creators believe that building open source LLMs from scratch is not for everyone. “You need a lot of funding to sustain open source and we believe that not everyone will be able to do it,” said Hakim Hacid, executive director and acting chief researcher at Technology Innovation Institute (TII), in an exclusive interaction with AIM.
Citing Mistral, Hacid said that it is not open source anymore—initially “it started with open source, and now it’s not anymore,” as these things require huge investments and a lot of commitment at policy level.
Interestingly, Mistral AI recently released a large language model Mixtral 8x22B that has 176 billion parameters and a context length of 65,000 tokens.
Falcon For the Kill
The Falcon-makers are super confident that its open-source strategy will remain sustainable due to the commitment of management and leadership who genuinely believe in it.
“It’s not just a matter of building models and having a sort of a final product that is based on the models, but it’s about the impact that this research can have on humanity and then allowing people to access this asset that is the AI,” said Hacid, who had an interesting take on the name of their LLM model.
“For us, the name actually comes from the national bird, the Falcon. It symbolises courage and determination. So, we wanted to associate those characteristics with our Falcon model, and we believe it does so effectively,” said Hacid.
The Falcon 180B model is available on Amazon SageMaker JumpStart to deploy with one-click running inference.
Eyes India
“India is our neighbour, and we have a lot of colleagues coming from India,” said Hacid when talking about the UAE-India relationship. “India is our neighbour, and we have a lot of colleagues who are coming from India.”
Hacid also pointed out how in India, there are a number of fine-tuning of models specifically for Urdu and other languages. “We have internally initiated on multilinguality to integrate Urdu because we have a lot of immigrants in the UAE who are coming from India. The Urdu language is as important as the Arabic language in the UAE.”
“So we are working on that, trying to get a little bit more content to specialise Falcon on the Indian side.”
With a number of emerging Indic language models, such as AI4Bharat, Urdu is offered as one of the languages.
Making UAE an ‘AI’ Nation
UAE is clearly moving away from an oil rich country into a data and knowledge-driven economy. “There is a vision that we want the country to go from an oil producer to a knowledge economy,” said Hacid.
He also said that the idea is to move from a culture where they were used to buying and consuming the technology, to a situation where they actually produce the technology and eventually export that technology to the external world.
In addition to this, the UAE government has been providing a conducive environment for boosting AI developments in the country. Hacid said that the country is also heavily investing and bringing scientists to the UAE to do their research and provide support in all ways possible.
Challenges Galore
It is easier said than done. Hacid believes that one of the biggest problems that all companies working with generative AI will face is the availability of the right talent. “Modern AI is a sort of young discipline. You cannot find people with 20 years of experience in this field,” he added.
Further, he said that finding experts with two to three years of experience is what we will get and he also feels that with the availability of a lot of opportunities in the world, adds to the complexity of the situation. “So, getting the right people will always be a challenge,” he said.
Hacid also emphasised on data localisation and how regulations can pose a challenge for training models. “Data will always be an issue because nowadays we see some regulations coming around limiting the access to data. So that could actually cause some delays in the development of this technology,” he said.
Obviously, compute is another challenge that Hacid also pointed out. While he believes that they have good access to compute, its impact on nature becomes critical. “We are working hard to make sure that the models we build today don’t negatively impact the environment tomorrow,” said Hacid.
Interestingly, the UAE bought thousands of NVIDIA GPUs last year.
AGI, Next?
TII is now looking at building even smaller models and focusing on multimodality – going the Microsoft way.
“We were asking at some point the question, as to how big should we go? I think now the question is how small we could go by keeping a small model,” said Hacid, saying that they are exploring that path.
Further, he said that they are making models smaller because, again, “if we want the pillar of the deployment to succeed, we need to actually have models that can run in devices, and in infrastructure that is not highly demanding.”
From a perspective of wanting to build models that can be serviced to the end users, TII and ATRC (Advanced Technology Research Council) built a startup A171, which will specialise in marketing and supporting Falcon LLMs.
In terms of multimodality, Hacid said that the models will be trained to not only understand text but also images, video, sound, graphs, time series and other types of data.
“We believe if we want to go towards artificial general intelligence, we need to have systems that are able to digest and get a sense out of different data sets. This is how we are learning as humans at the end of the day,” quipped Hacid.
