At its Data+ AI Summit, Databricks announced an expanded collaboration with NVIDIA to optimise data and AI workloads by integrating NVIDIA CUDA-accelerated computing into the core of Databricks’ Data Intelligence Platform.
The partnership aims to boost the efficiency, accuracy, and performance of AI development pipelines for modern AI factories, as data preparation, curation, and processing are crucial for leveraging enterprise data in generative AI applications.
Through this broadened alliance, Databricks is adding native support for NVIDIA GPU acceleration on its Data Intelligence Platform. The announcement builds upon the companies’ existing collaboration to enrich enterprises’ experiences across various use cases, from training classical machine learning models to building and deploying generative AI applications and optimising digital twins.
“We’re thrilled to continue growing our partnership with NVIDIA to deliver on the promise of data intelligence for our customers from analytics use cases to AI,” said Ali Ghodsi, Co-founder and CEO at Databricks. “Together with NVIDIA, we’re excited to help every organisation build their own AI factories on their own private data.”
Jensen Huang, founder and CEO of NVIDIA, emphasised the importance of accelerated computing in reducing data processing energy demands for sustainable AI platforms. “By bringing NVIDIA CUDA acceleration to Databricks’ core computing stack, we’re laying the foundation for customers everywhere to use their data to power enterprise generative AI,” Huang stated.
A key aspect of the partnership involves Databricks developing native support for NVIDIA-accelerated computing in its next-generation vectorised query engine, Photon. This integration is expected to deliver improved speed and efficiency for customers’ data warehousing and analytics workloads. Photon powers Databricks SQL, the company’s serverless data warehouse known for its industry-leading price-performance and total cost of ownership (TCO). The collaboration is anticipated to lead to the next frontier of price-performance.
Databricks Shares a Unique Partnership with NVIDIA
In the backdrop of Databricks’ Data + AI Summit 2024, Anil Bhasin, the vice president of India and SAARC region at Databricks, told AIM that the company’s partnership with NVIDIA—one of its strategic investors–is significant, alongside helping them improve run times using their SOTA GPUs.
“We’ve always been known as pioneers of the lake house architecture, and now we’ve created a new category called the data intelligence platform. We’ve embedded generative AI in the lake house, which is a unique approach not many companies are taking,” said Bhasin, saying that NVIDIA is aligned with their vision because the future lies in data intelligence platforms.
He said this allows them to serve every use case, ingest data from any source, and maintain unified governance. This strategic differentiation makes their partnership with NVIDIA truly special and aligns perfectly with NVIDIA’s ‘Sovereign AI’ for enterprises. Nobody other than Databricks is enabling this.
“The ability for us to query in natural language, converting it to SQL on the back end, empowers the business user to gain insights. That is true democratisation,” avered Bhasin, saying their vision is powerful, and not just NVIDIA; many companies believe in Databricks’ long-term vision.
NVIDIA x Databricks
Recently, Databricks’ open-source model DBRX became available as an NVIDIA NIM microservice. NVIDIA NIM inference microservices provide fully optimised, pre-built containers for deployment anywhere, significantly increasing enterprise developer productivity by offering a simple, standardised way to add generative AI models to their applications.
Launched in March 2024, DBRX was built entirely on top of Databricks, leveraging the platform’s tools and techniques, and was trained with NVIDIA DGX Cloud, a scalable end-to-end AI platform for developers.
The Databricks Data Intelligence Platform offers a comprehensive solution for building, evaluating, deploying, securing, and monitoring end-to-end generative AI applications. With Databricks Mosaic AI’s data-centric approach, customers benefit from an open, flexible platform to easily scale generative AI applications on their unique data while ensuring safety, accuracy, and governance.
Today’s announcement follows Databricks’ strategic acquisition of Tabular, a data management startup founded by the original creators of Apache Iceberg and Linux Foundation Delta Lake, the two leading open-source lakehouse formats. By bringing together these key players, Databricks aims to lead the way in data compatibility, ensuring organisations are no longer limited by the format of their data.
Driven by the growing demand for data and AI capabilities, Databricks achieved over $1.6 billion in revenue for its fiscal year ending January 31, 2024, representing more than 50% year-over-year growth.
The expanded partnership between Databricks and NVIDIA underscores the critical role of accelerated computing and optimised data processing in enabling enterprises to harness the power of generative AI effectively and efficiently.
