Building large language models requires complicated data structures and computations, which conventional databases are not designed to handle. Consequently, the importance of vector databases has surged since the onset of the generative AI race.
This sentiment was reflected in a recent discussion when software and machine learning engineer Santiago Valdarrama said, “You can’t work in AI today without bumping with a vector database. They are everywhere!”
He further added that vector databases, with their ability to store floating-point arrays and be searched using a similarity function, offer a practical and efficient solution for AI applications.
Vector databases provide LLMs with access to real-time proprietary data, enabling the development of RAG applications.
Database companies are pivotal in driving the generative AI revolution and its growth. Redis enhances real-time efficiency for LLM-powered chatbots like ChatGPT, ensuring smooth conversations. At the smae time, enterprises are leveraging MongoDB Atlas and Google Cloud Vertex AI PaLM API to develop advanced chatbots.
Making it Easier
However, major database vendors, regardless if they were originally established as SQL or NoSQL, such as MongoDB, Redis, PlanetScale, and even Oracle have all added vector search features to their existing solutions to capitalise on this growing need.
In an earlier interaction with AIM, Yiftach Shoolman, the co-founder and CTO of Redis, said, “We have been working with vector databases even before generative AI came into action.”
Redis not only fuels the generative AI wave with real-time data but has also partnered with LangChain to launch OpenGPT, an open-source model that allows flexible model selection, data retrieval control, and data storage management.
Another important challenge vector databases claim to solve is hallucinations, which have been a persistent issue for LLMs.
“Pairing vector databases with LLMs allows for the incorporation of proprietary data, effectively reducing the potential range of responses generated by the database,” said Matt Asay, VP, developer relations, in an exclusive interaction with AIM at last year’s Bengaluru chapter of their flagship event MongoDB.local.
During a recent panel discussion, Pinecone founder and CEO Edo Liberty explained that vector databases are made to manage these particular types of information “in the same way that in your brain, the way you remember faces or the way you remember poetry”.
Most of the prominent names in the industry have already implemented vector capabilities. Think Amazon Web Services, Microsoft, IBM, Databricks, MongoDB, Salesforce, and Adobe.
Jonathan Ellis, the co-founder and CTO of DataStax, explained that while OpenAI’s GPT-4 is limited to information up until September 2021, indexing recent data in a vector database and directing GPT-4 to access it can yield more accurate and high-quality answers. This approach eliminates the need for the model to fabricate information, as it is grounded in updated context.
What Next?
However, vector databases are not without challenges. A recent report by Gartner noted that using vector databases for generative AI may raise issues with raw data leakage from embedded vectors. Raw data used to create vector embeddings for GenAI can be re-engineered from vector databases, making data leakage possible.
“Given the compute costs associated with AI, it is crucial for organisations to engage in worker training around vector database capabilities,” Gartner analyst Arun Chandrasekaran emphasised in an interview with Fierce. “This preparation will help them avoid expensive misuse and misalignment in their AI projects.”
Nevertheless, several vector db startups are now gaining prominence. During an otherwise weak year for venture capital, hundreds of dollars are flowing into vector database businesses like Pinecone, which got $100 million in April 2023 from Andreessen Horowitz.
Pinecone is not the only one. Dutch firm Weaviate secured $50 million from Index Ventures. The Weaviate AI-native vector database simplifies vector data management for AI developers.
There are emerging divisions in the vector database arena, particularly between open- and closed-source players, and between dedicated vector databases and those with integrated vector storage and search functionality.
On the dedicated, open-source side, Chroma, Quadrant, and Milvus (in collaboration with IBM) stand out, while Pinecone is a leading dedicated, closed-source player. Meanwhile, Snowflake, although not a dedicated vector database, offers vector search capabilities within its open-source framework.
And there’s a good reason why so many people are jumping into this sector. Chandrasekaran predicts that 30% of organisations will employ vector databases to support their generative AI models by 2026, up from 2% in 2023.
Understanding its importance, Andrew Ng, has also introduced free learning courses on the same with MongoDB, Weavaiate, Neo4j and more.
With the increasing adoption predicted by experts and the introduction of educational resources, vector databases are set to play a pivotal role in shaping the next era of AI technology.
As organisations continue to integrate these powerful tools, the potential for innovation and improved AI capabilities becomes ever more significant, heralding a new age of intelligent applications and solutions.
