Tens of billions of dollars in platform wrestling, cloud data generative AI battle

Author: Vivek Sabrina, Source: Silicon Rabbit Race

Image source: Generated by Unbounded AI tool

Snowflake and Databricks have always been two companies that have received a lot of attention in the database field. Although they are on the same site, they each have their own characteristics, and the competition has not been put on the table.

In this wave of generative AI, the two companies are very active through acquisitions. Snowflake completed the acquisition of Neeva (enterprise-level AI search engine), and Databricks acquired MosaicML (ML model deployment) for US$1.3 billion, and announced the acquisition in a low-key manner. OmniML (Model Compression) acquisition.

The two companies have changed their previous situation of harmony on the surface and rivalry in secret, and chose to hold the company's most important annual meeting on the same day, highlighting their generative AI layout, and their ambitions cannot be hidden.

Snowflake will go public in 2020, with a current market value of US$57.92 billion (2023.8.1). Databricks has not yet been listed. According to the last round of financing, its valuation has reached US$38 billion. With the blessing of generative AI, can Databricks' valuation/future listing market value catch up with Snowflake? Can Snowflake take it to the next level?

Vivek, a partner of Madrona, and investor Sabrina, who invested in Snowflake, shared their views on the two companies wrestling in the field of generative AI.

Last week was a big one for practitioners in the data and artificial intelligence space, as two of the most important players — Databricks and Snowflake — held their annual conferences in San Francisco and Las Vegas respectively (Databricks’ Data and AI Summit and Snowflake's Summit).

It's no coincidence that these two giants decided to hold their big events in the same week. **Snowflake and Databricks have been friends and rivals for the past decade, but this week made it clear that they are now each other's arch rivals, and that the new battleground is artificial intelligence. ** Not surprisingly, much of the discussion and announcements at both conferences revolved around generative AI. The key message is that in order to build a generative AI strategy, every company must start with a data strategy.

Unsurprisingly, both Databricks and Snowflake made a case for why they can best support customers on this journey.

How did two companies that started out in different parts of the value chain, once even a strategic partnership, evolve into such fierce competitors in this new era of artificial intelligence?

Let's dig deeper.

[Quick disclaimer: Madrona invested in Snowflake’s Series C round and still owns some shares in the company. 】

01Snowflake: From Data Warehouse to Data Cloud

Snowflake was founded in 2012 by Benoît Dageville and Thierry Cruanes. They are two database experts who have worked at Oracle for many years, and they keenly observe that most data warehouses are "hardened, expensive and difficult to use". Dageville and Cruanes partnered with Marcin Zukowski, former CEO of Vectorwise, to build a data warehouse of the future based on three key premises:

1. Completely cloud-based architecture;

2. Separate computing and storage to achieve almost unlimited expansion;

3. Elasticity in computing resource usage, enabling unprecedented speed in query processing and flexibility. Today, Snowflake has grown from "just" a cloud data warehouse to a "data cloud" that provides customers with a single platform to access, build, collaborate and monetize their data. In just over a decade, Snowflake has grown into a $55 billion public company, serving more than 6,000 customers and many Fortune 500 companies. Snowflake has joined forces with the major hyperscale cloud providers (Azure, AWS, and GCP), and now they have their sights clearly set on AI for more attention.

To achieve this goal, they have made a series of acquisitions and product launches in the field of artificial intelligence and machine learning, including:

1. Snowpark allows data scientists to use their preferred programming language for end-to-end machine learning workload development, deployment and orchestration. With Snowpark, customers can ingest, analyze and transform their data to train machine learning models and run more predictive analytics.

2. Streamlit is a data-driven app building tool that Snowflake acquired in March 2022 for $800 million. Streamlit enables customers to develop data-intensive applications with just a few lines of code. Streamlit simplifies the process of contextualizing data analysis tasks and machine learning model output through front-end web applications.

3. Neeva, a company Snowflake acquired earlier this year, aims to accelerate enterprise interaction and search with data, especially in a more conversational way.

02Databricks: Build Lakehouse

Databricks was founded in 2013, just a year after Snowflake. Unlike Benoit and Theirry, who are industry practitioners, Databricks was founded by a group of people with deep ties to academia and the open source community.

The seven founders, including current CEO Ali Ghodsi, were AMPLab researchers at UC Berkeley and conceived Apache Spark, an open-source unified analytics engine for large-scale data processing. Spark has grown to be one of the largest and most used data processing frameworks, playing an important role in large-scale data engineering, data science, and machine learning.

Databricks' original goal was to commercialize Spark, launching an enterprise-grade version of Spark that provides all the features (governance, support, hosting, etc.) needed by large organizations. Databricks then evolved into the innovative "Lakehouse Platform" that unifies data, analytics and artificial intelligence. The unified Lakehouse concept combines "integration, storage, processing, governance, sharing, analytics and artificial intelligence" on a single platform.

Over the past decade, Databricks has become one of the world's most highly valued private companies, with a 2021 valuation of $38 billion and recently achieved a $1 billion revenue milestone. They serve tens of thousands of corporate clients and open source users, and are considered one of the most high-profile IPOs. Amid all this growth, they have increasingly positioned themselves as leaders in AI, with recent notable acquisitions and product launches, including the $1.3 billion acquisition of MosaicML (more on that below), and open-sourcing the Dolly, an instruction-tuned LLM that can be trained for less than $30.

03 Collision in AI

Both Snowflake and Databricks are well-positioned to continue to capitalize on long-term structural trends as enterprises prepare for the shift toward generative AI. Both companies are trying to position themselves as strategic multi-product data platforms as generative AI applications become more widely available. Here are some key announcements from their respective conferences and our take on each company's overall AI strategy.

Snowflake Major Announcement:

DEVELOPER ANNOUNCEMENT

1. Snowflake's native application framework: It can be extended based on Snowflake's data cloud by allowing developers to create, distribute and monetize applications to utilize data in new ways.

2. Snowpark Container Service: Extend data programmability and computing infrastructure to support programming languages, access third-party software, and provide enhanced security and governance for hosting full-stack applications and LLM. Further flexibility is provided by generalizing Snowflake's computing platform, enabling customers to run complete end-to-end applications from the bottom layer (data layer) all the way to the UI layer.

3. Other important announcements: Snowpipe streaming capabilities; Dynamic Tables (also known as Materialized Tables); Document AI (a new service for extracting unstructured data from documents); and Iceberg Tables.

Partner AnnouncementSnowflake announced several key partners including NVIDIA, Microsoft and Weights & Biases.

1. The partnership with NVIDIA plans to embed its NeMo enterprise development framework into Snowflake's data cloud, which will enable Snowflake customers to build and deploy LLMs and AI-based applications utilizing proprietary data stored in Snowflake.

2. The collaboration with Microsoft will expand the partnership with Azure, focusing on new product integrations around Microsoft Azure's OpenAI and Azure AI/ML services. The collaboration has the potential to bring workloads and customers into the data cloud.

3. Cooperating with Weights & Biases, a leading MLOps platform, Snowflake's container service enables Weights & Biases to accelerate the iterative development of ML models, LLMs, and LLM-driven applications in the Snowflake data cloud. Ultimately, this collaboration will help businesses and users more easily build and leverage generative AI.

4. In addition to these two companies, Snowflake has announced many other partnerships with Alteryx, Hex, Dataiku, RelationalAI, Pinecone, and others.

OUR OPINION

Until recently, Snowflake didn't disclose any plans to add generative AI to its existing capabilities, and many investors expressed concern about Snowflake's ability to compete in this space, especially compared to Databricks. However, at the 2023 summit, Snowflake presented a strong vision, positioning itself as a trusted data cloud provider, and with that built a strong story around generative AI.

Snowflake's partnership with Nvidia and the announcement of the Snowpark container service make them a more viable player in the AI data stack. The core point they want to convey is that they can enable customers to securely access, develop and deploy LLMs and AI-based applications in the Snowflake data cloud, while providing accelerated computing with Nvidia GPUs and AI software.

While their story and message is impressive, we think they are still underdogs relative to Databricks in the AI space...

Databricks main announcement:

DEVELOPER ANNOUNCEMENT

1. LakehouseIQ: LLM-based natural language interface for searching and querying data, and powerful understanding of customer's data, internal jargon and usage patterns to understand customer's architecture, documents, queries, systems, etc.

2. **LakehouseAI: **Databricks has announced a number of new features in Databricks ML, including some on LLMOps capabilities such as integrating data, preparing datasets for machine learning, fine-tuning and curating machine learning models, and deploying the models themselves. Databricks also announced a number of features around Vector Search, Feature Services, and MLFlow Gateway.

3. MosaicML: Just before the summit, Databricks announced the acquisition of MosaicML for $1.3 billion, which was positioned as a "machine for building GenAI models" during the summit.

4. **Other noteworthy announcements: **Delta Lake 3.0, MLFlow 2.5 support intelligent monitoring of different backend LLMs, Lakehouse Apps and Databricks Lakehouse Monitoring.

our opinion

Databricks takes a unified approach to AI by bringing together data, AI models, monitoring and governance capabilities into the Lakehouse platform. As a result, Databricks enables customers to develop their GenAI solutions more efficiently, and customers perceive Databricks as a trusted partner who, on average, is faster, more cost-effective, and easier to use in machine learning development.

While already considered a key player in the AI stack, Databricks has solidified its leadership in GenAI through investments in models such as Dolly, an open source instruction-following LLM, and a major acquisition of MosaicML. Databricks went on to highlight their Lakehouse as the best way for GenAI startups to train and deploy their own AI models, leveraging their own proprietary data in a cost-effective manner without being tied down by big tech companies.

**04 Looking ahead, what can we expect? **

Although the generative AI craze has been going on for more than eight months, the past week has made it clear that Snowflake and Databricks are in a race for minds and market share in the space. So, what can we expect from this heightened competition?

1. Acquisitions will continue → Both Snowflake and Databricks are relatively well positioned to continue acquiring smaller companies that complement their overall strategy. Snowflake has about $4 billion in cash on its balance sheet, while Databricks has a high valuation available for trading. Meanwhile, hundreds of AI and data tools startups are eager to find an outlet in a dry IPO market. We don't think Neeva and MosaicML will be the last acquisitions by these giants and there will be consolidation in the market.

2. Customers will benefit → In the escalating competition between Snowflake and Databricks, the clear winner should be their customers. The two giants are rapidly adding novel products and services to their platforms, building "one-stop shops" for customers to build data applications and leverage LLMs. This platform enhancement will help democratize access to AI and allow data scientists, data engineers, and AI practitioners to collaborate more meaningfully.

3. Azure and AWS will earn more profit → As Snowflake and Databricks continue to expand further in the AI market, they will require a lot of computing power, mainly provided by Azure and AWS. Data engineer Anant Packidurali astutely observes this. In the same way that Nvidia benefits in AI, the hyperscale cloud service providers that provide the infrastructure for Snowflake's and Databricks' computing needs stand to gain regardless of who wins the AI race.

As businesses increasingly rely on data to support their generative AI strategies, we believe both Snowflake and Databricks are well-positioned to capitalize on this generational shift. Although they come from different parts of the value chain and their relationship has evolved over the past decade, they are now in a race with huge rewards.