Batch data processing is too slow for real-time AI: How open-source Apache Airflow 3.0 solves the challenge with event-driven data orchestration

The information that leads to the correct location from various sources for the use of AI is a difficult task. This is where the information in detergent technologies Apache furnish to match.

Today, the Apache Air Stream community comes out with the largest update of the debut of 3.0 release. The new edition records its first basic version update for four years. 2, in the X series, continuously increased, continuously active 2.9 and 2.10 update In 2024, he was in a heavy focus on the AI.

In recent years, information engineers accepted Apache Aurflow as their de facto standard. Apache Airflow has established itself as a widespread leading open source source orchestra platform, which is widespread among more than 3,000 contributors and Fortune 500 companies. There are also a large number of trading services based on platform, including platform Astronomer AstroGoogle Cloud composer, Amazon Apache Airflow (MWAA) and Microsoft Azure information factory, managed the flow of air managed among others.

Organizations, disparate systems, clouds and increasing the increasing AI workload, which are struggling to coordinate workflows along the workload, organizations have growing needs. Apache AirFlow 3.0 can make and place a architectural re-designation of an architectural re-application, which appeals to the critical enterprise and place data applications.

“I am a new start, a foundation for great opportunities,” Vikram Koka, Astronomy officer in the Astronomer said in an exclusive interview. “This is an almost complete referee, where the speech of enterprises need the level of critical adoption.”

The complexity of enterprise information has changed the needs of the information orchestra

As soon as enterprises trust in the decision of the information management, the complexity of the information exploded. Now organizations manage a very cloud environment, various sources of information and increasingly developed AI workloads.

Air flow 3.0, it appears as a solution that is specially designed to meet the needs of the developing enterprise. Unlike previous versions, this release is far from a monolithic package that provides a distributed customer model that provides comfort and security. This allows new architectural enterprises:

1. Execute tasks in a very cloud environment.
2. Implement granular security controls.
3. Support various programming languages.
4. Activate real multiple cloud placements.

Extended language support from AirFlow 3.0 is also interesting. Although previous versions are primarily Python-centered, the new release supports many programs in many programs.

Airflow 3.0 is set to support Python and go with a planned support for Java, Type and PAS. This approach says that information engineers will be able to write tasks in friction systems, reducing friction systems in the development and integration of workflow.

Opportunities controlled by event changes the workflows of information

The flow of air is traditionally preferred in the general processing, but the enterprises need real-time data processing opportunities. Air flow 3.0 now supports this need.

“The key change in Airflow 3 is what we call the event based on the incident,” Koka said.

Instead of doing a data processing in each hour, the airflow is now downloaded when a certain data file is loaded or a certain message is visible. This can include information downloaded to the Amazon S3 cloud storage bucket or data message broadcast on the Apache Kief.

An event managed planning ability solves a critical gap between traditional ETL [Extract, Transform and Load] such as tools and flow processing frames Media out or Apache Spark Structured StreamAllows you to use a single orchestrate layer of organizations for both appointments and flow.

Airflow will accelerate the company’s result of AI and complex AI

The information orchestra of the incident will help to support the fastest resulting execution of air flow.

Example, Koka, explain a use in detail where a real-time result is used for professional services such as trailing. In that scenario, airflow can be used to help collect raw data from sources such as calendars, emails and documents. A large language model (LLM) can be used to convert unstructured data into structured data. Then a pre-made model, then structured time can be used to analyze tracking information, determine the calculation of the case, then determine the appropriate calculation codes and ratios.

Coca called this approach as a complex AI system – a workflow that brings together a complex task and a clever issue together. Airflow 3.0’s event managed architecture This type of real-time is a very step-step effect process possible during the use of various enterprises.

Ink AI, a first-defined approach Berkeley Artificial Intelligence Research In 2024, the center and agent are slightly different from AI. Koka, the agent explained that the EU has enabled the Mukhtar EU decision, whereas the compound, the more predictable and valid work flows for work.

Playing the ball with Airflow, how to take advantage of how Texas Rangers

Among many users of the air flow is Texas Rangers Major League Baseball team.

Oliver Dykstra, the full national team in Texas Rangers Baseball Club, told Venturebeat that the team hosted the ‘nerves center’ hosted on the astronomy platform on Astronomy Astro Platform. He noted that all player development, contracts, analysts and of course the game information is orchestated with air flow.

“We expect the flow to increase the flow and increase the event of an event, and increase the incident to the generation of observation and information,” Dykstra. “As we have already trusted air flow to handle the critical AI / ML of pipelines, the supplementature and reliability of AirFlow 3 will help increase the reliability of these information products in our entire organization.”

What does this enterprise mean for the AI ​​adoption

For technical decision makers that evaluate the information strategy, AirFlow provides effective benefits to be implemented in 3.0 stages.

The first step appreciates the current information work that will benefit from the new event management abilities. The organizations can set the data to trigger the planned jobs, but event-based triggers can be more effective. This change can significantly reduce the processing delay while eliminating election operations.

Next, technology leaders should evaluate development environments to determine if the air flow can strengthen the fragmented orchestral tools of new language support. Currently, the teams stored in a separate orchestra for different language environments can start planning a migration strategy to facilitate the technological stack.

For businesses led by the road in the AI, AirFlow 3.0 represents a critical infrastructure component that can solve an important problem in the adoption of the AI: a multi-stage AI workflows of the enterprise. The ability to coordinate the compound AI systems of the platform can help organizations move outside the concept of relevant management, security and reliability.