Indeed, India needs AI that understands the nuances and intricacies of the diverse languages the country speaks; however, Bhavish Aggarwal’s Krutrim may not be the answer – at least for now.
Last year, Aggarwal announced Krutrim as India’s own AI model that can understand 22 Indic languages and generate output in 10 of them. He even claimed that his model outperforms GPT-4 and LLama-2, probably the most advanced and popular LLMs out there, when it comes to Indic languages.
Yet, when the model was actually released last week, it turned out to be a disappointment. The chatbot not only miserably failed to pick the nuances of Indic languages, but in many cases could not even understand the context in languages like Hindi and Marathi.
Aggarwal asserted that Krutrim was trained on 20 times more Indic tokens than any other model. If that were the case, Krutrim should have outperformed other chatbots like Gemini or ChatGPT.
But when we prompted the bot in Assamese, it confused Assamese with Bengali, another Indo-Aryan language, given the languages share the script and have substantial overlaps in vocabulary. In another instance, the bot was found confused between Marathi and Hindi.
High on Hallucinations
We witnessed the same confusion in GPT3.5 as well, which powers OpenAI’s free ChatGPT version, as well as QX Lab’s recently launched Ask Qx. The confusion arises due to the limited datasets of these low-resource languages on which these models, including Krutrim, have been trained.
The company acknowledged this, stating, “Krutrim’s training data is limited, potentially resulting in occasional inaccuracies or biases.”
Aggarwal too, claimed that hallucinations will be there but much lower for Indian contexts than other global platforms. However, that has not been the case. Hallucinations appear to be significantly higher even when prompted in the English language. In a specific case, it asserted it was created by OpenAI.
In our course of testing the bot, we discovered that it has been programmed not to discuss OpenAI, its models or other LLMs like Llama.
Since Krutrim has not disclosed extensive details about the model, including its architecture or the dataset used for training, the errors it made have led many to speculate if it is merely a wrapper of OpenAI’s GPT models.
Previously, AIM had reached out to Ola seeking more information on Krutrim but they declined to comment.
Krutrim’s embarrassment escalated further as an increasing number of users tested the bot and shared its goof-ups on social media. Many raised questions about how Krutrim managed to secure $50 million to become India’s first AI unicorn.
Saving Grace
The only saving grace for Krutrim so far is that it is still in beta version. Even though its limitations are being extensively highlighted, Krurtim has the opportunity to fix the problems, lower the hallucination in the coming days and deliver a better model to developers and enterprises with its API.
For Krutrim to excel in low-resource languages, Ola will have to invest significantly in building datasets for these languages. Currently, there is a lack of sufficient high-quality data available to train a model of ChatGPTs scale on Indic languages.
Moreover, Krutrim will launch a much bigger multimodal model in the second quarter of 2024. We are hopeful that this model, called Krutrim Pro, will perform better and the version released last week.
For Krutrim to have use cases, and enterprise adoption, which might be the target for Aggarwal, he needs to deliver a much better model, which enterprises can trust.
Currently, Krutrim can be a fun chatbot for Indian consumers to play around with and hopefully will get better with time, given it uses reinforcement learning in the human feedback loop, just like ChatGPT.
