Airflow 3 introduces DAG Versioning, remote execution, enhanced security model, and multi-language support to address the evolving needs of AI/ML and data engineering teams worldwide
NEW YORK, April 23, 2025 -- Astronomer, the company behind Astro, the leading unified DataOps platform powered by Apache Airflow®, today celebrated the release of Apache Airflow 3, the most significant evolution of the open source project since its inception. Coming approximately four and a half years after the last major release, Airflow 3 delivers a comprehensive redesign that delivers long-standing community requests while also expanding the platform's capabilities to support complex AI, ML, and near real-time data workloads. Airflow 3 contains a significant number of new features that make Airflow easier to use, more secure, and able to run any task, anywhere.
As organizations push to operationalize AI and deliver on the promise of near real-time, data-driven applications, they need an orchestration framework that combines enterprise-grade security with the flexibility to run workloads anywhere. Airflow 3 delivers precisely that, transforming how teams build, deploy, and manage modern data pipelines. With over 324 million downloads in 2024 - more than all previous years combined - and over 80,000 global organizations relying on Airflow, the technology is more relevant than ever. This watershed release gives data teams everything they need to address the evolving needs of today's data-intensive enterprises and establishes Airflow as the de facto data orchestration tool for years to come.
"Several years ago, Airflow use was limited to supporting internal dashboards, often requiring frequent manual intervention just to keep workflows running reliably," said Vikram Koka, Chief Strategy Officer, Astronomer, and PMC Member of Apache Airflow. "With the maturation of the project, especially since the launch of Airflow 2, data teams have leveraged Airflow to meet the growing demand for data orchestration to power everything from continuous compliance in regulated industries, to production-grade AI deployed at enterprise scale. Airflow 3 is the biggest leap forward in the project's history, and it is purpose-built to support increasingly complex use cases like these. I'm grateful to the community responsible for this release and I can't wait to see what innovations data teams will bring to life with the powerful new foundation that is Airflow 3."
Ease of use
Airflow 3 addresses the perennial challenges of debugging, historical tracking, and workflow management. DAG Versioning allows you to maintain a historical record of your pipeline code in Airflow. This ensures that teams never have to guess which version of the DAG produced a given outcome, dramatically reducing troubleshooting time.
Backfills are historically a point of friction in large-scale data and ML operations, but in Airflow 3 they are first-class citizens. Backfills allow users to rerun missed tasks asynchronously via the UI or API, and to monitor or cancel them with ease. These enhancements collectively empower data practitioners to design and iterate on pipelines with greater confidence and speed.
Airflow 3 introduces a multitude of other UI improvements that make it easier to use than ever: for experienced users, everyday tasks are smoother and faster, and for new users, the UI is less complex and more intuitive. The modernized UI further streamlines day-to-day tasks, offering a more intuitive layout, tabbed left hand navigation, and enhanced visual feedback on task run progress.
Enhanced security
Airflow powers business-critical applications at the world's largest organizations – including ones that must manage sensitive data – so security posture is paramount. Airflow 3 decouples task execution from other Airflow system components, significantly reducing attack surfaces and improving data governance. Instead of having worker processes write directly to the Airflow metadata database, Airflow tasks call an API server to receive and report job statuses.
Each task run is authenticated via strong identities, ensuring robust authorization that meets strict compliance requirements. This decoupled model is particularly valuable for multi-tenant or heavily regulated environments where minimal privilege is non-negotiable. By marrying convenience with comprehensive security controls, Airflow 3 becomes an even more dependable platform for enterprise DataOps.
Run tasks anywhere, at any time
Airflow 3 delivers even more freedom and flexibility by supporting remote execution across a spectrum of environments - from public and private clouds, to on-premises data centers and edge devices.
Event-driven scheduling allows pipelines to respond to external triggers such as updates in messaging queues. By laying the groundwork to run tasks in any language - starting with Golang - Airflow 3 eliminates the need for extensive rewrites, code refactoring or awkward workarounds. This capability brings data, AI, and ML workloads under a single orchestrator and dramatically simplifies the data stack, while giving data teams the freedom to work with data no matter where it is stored or processed.
Airflow 3 introduces several enhancements to support AI inference execution, a critical phase in the ML lifecycle that represents the application of a model to solve real-world problems and generate actionable insights. With ad hoc scheduling, simultaneous DAG execution, event-driven scheduling and more, these enhancements make Airflow 3 more capable of handling diverse inference scenarios. This includes batch processing and on-demand execution, while offering improved flexibility and performance for AI and ML workflows.
Airflow 3 on Astro
In addition to supporting all of the aforementioned Airflow 3 features, Astro - Astronomer's unified DataOps platform powered by Airflow - has unique capabilities that help enterprises gain even more value from this transformative release:
- Remote Execution Agents on Astro delivers the industry's first truly secure, managed orchestration solution that meets rigorous compliance standards without compromising on orchestration capabilities. Remote Execution Agents is an Astro-exclusive feature that completes tasks in the environment while establishing only outbound, encrypted connections to Astro's Orchestration Plane. Sensitive data never leaves local infrastructure, and code remains in secure repositories.
- In order to utilize DAG Versioning to the full extent, open source Airflow users must set up their own Git DAG bundle. Astro comes with DAG Versioning set up out of the box, requiring no user implementation and integrates seamlessly with the user's existing deploy mechanisms.
- Additional Astro-exclusive Airflow 3 features such as rollbacks will continue to be released on a rolling basis.
Apache Airflow 3.0 is available now. To learn more and get started:
Further Resources:
- Download and explore Apache Airflow 3
- Delve into the new features in our detailed blog post
- Register for today's "Introducing Apache Airflow 3" webinar
- RSVP for an upcoming Astronomer Roadshow: Exploring Apache Airflow 3 event in London, New York, Sydney, San Francisco, or Chicago
- Try Astro now with a 14-day free trial, featuring Airflow 3 support
About Astronomer
Astronomer empowers data teams to bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and power data-driven applications. Trusted by more than 700 of the world's leading enterprises, Astronomer lets businesses do more with their data. To learn more, visit www.astronomer.io.
Apache® and Apache Airflow® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.
This News is brought to you by Qube Mark, your trusted source for the latest updates and insights in marketing technology. Stay tuned for more groundbreaking innovations in the world of technology.
