Information Technology
Databricks Donates Declarative Pipelines to Apache Spark™ Open Source Project
Spark Declarative Pipelines is based on Databricks' core declarative ETL framework, which is used by thousands of customers. With the proven ability to handle complex data engineering workloads and low-latency streaming, Spark Declarative Pipelines lays the foundation for the next generation of data processing and governance. With Spark Declarative Pipelines, more community members can begin to cut engineering time and costs and reliably support new AI agent systems and other workloads in production.
"Our commitment to open source is unwavering. With origins in academia and the open source community, Databricks was founded in 2013 by the original creators of the lakehouse architecture and open source projects including Apache Spark, Delta Lake , MLflow and Unity Catalog," said Matei Zaharia , Co-founder and CTO of Databricks. "We worked closely with the community to help remove friction around data formats that kept information siloed. Spark Declarative Pipelines now gives enterprises an open way to build high-quality pipelines."
"Declarative pipelines hide the complexity of modern data engineering under a simple, intuitive programming model. As an engineering manager, I love the fact that my engineers can focus on what matters most to the business. It's exciting to see this level of innovation now being open-sourced, making it accessible to even more teams." —
Jian (Miracle) Zhou , Senior Engineering Manager, Navy Federal Credit Union
"At 84.51˚ we're always looking for ways to make our data pipelines easier to build and maintain, especially as we move toward more open and flexible tools. The declarative approach has been a big help in reducing the amount of code we have to manage, and it's made it easier to support both batch and streaming without stitching together separate systems. Open-sourcing this framework as Spark Declarative Pipelines is a great step for the Spark community." — Brad Turnbaugh, Sr. Data Engineer, 84.51°
Databricks is the Data and AI company. More than 15,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Databricks is headquartered in San Francisco , with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake , MLflow, and Unity Catalog. To learn more, follow Databricks on X , LinkedIn and Facebook .
: Press@databricks.com
Logo - https://mma.prnewswire.com/media/1160675/Databricks_Logo.jpg
View original content:https://www.prnewswire.co.uk/news-releases/databricks-donates-declarative-pipelines-to-apache-spark-open-source-project-302478771.html