def push_metadata(**kwargs): ti = kwargs['ti'] ti.xcom_push(key="source_file_path", value="/data/raw/2025-01-01.csv") ti.xcom_push(key="record_count", value=15234) ti.xcom_push(key="processing_status", value="SUCCESS") return "push_complete" # Also auto-pushed with key "return_value"
| Setting | Default | Change in airflow.cfg | |---------|---------|--------------------------| | xcom_backend | airflow.models.xcom.BaseXCom | – | | xcom_backend_kwargs | {} | – | | Max size (SQLite/Postgres) | 1–2 KB | Not recommended to increase → use external storage for >1MB |
However, XCom is often misunderstood and misused. This comprehensive guide explores the best practices for using XCom in your Airflow pipelines, focusing on the philosophy of keeping XCom messages lightweight, manageable, and efficient—what we call . airflow xcom exclusive
This allows you to implement custom logic (like encryption) to ensure only authorized tasks can deserialize the data.
Variables are suitable for global configuration values, not for dynamic task outputs. They are persistent across DAG runs and accessible from anywhere. def push_metadata(**kwargs): ti = kwargs['ti'] ti
In the realm of workflow orchestration, Apache Airflow stands out as a premier tool for managing complex data pipelines. At the heart of its ability to create interdependent, context-aware workflows is , short for "cross-communication." While Airflow's core philosophy emphasizes task isolation, XCom provides the essential bridge for tasks to share small but critical pieces of metadata. The Mechanics of Inter-Task Communication
If you want, I can:
: If a Python task returns a value at the end of its function execution, Airflow automatically saves it. If that data is not needed downstream, return None or set do_xcom_push=False in your operator configuration.
The 48KB limit is not a flaw but a feature: it forces pipeline designers to think carefully about data architecture. Large, heavy data sets belong in external object stores or shared file systems, not in the orchestrator's database. When you need to share more than 48KB, custom XCom backends—such as the object storage backend for S3, GCS, or Azure—provide a clean, scalable solution. Variables are suitable for global configuration values, not
Master Airflow XCom: From Basics to Advanced Custom Backends