Airflow: how to use xcom_push and xcom_pull in non-PythonOperator

Question

I see a lot of examples on how to use xcom_push and xcom_pull with PythonOperators in Airflow.

I need to do xcom_pull from a non-PythonOperator class and couldn't find how to do it.

Any pointer or example will be appreciated!

tobi6 · Accepted Answer · 2018-07-30 05:59:50Z

8

You can access XCom variables from within templated fields. For example, to read from XCom:

myOperator = MyOperator(
    message="Operation result: {{ task_instance.xcom_pull(task_ids=['task1', 'task2'], key='result_status') }}",
    ...

It is also possible to not specify task to get all XCom pushes within one DagRun with the same key name

myOperator = MyOperator(
    message="Warning status: {{ task_instance.xcom_pull(task_ids=None, key='warning_status') }}",
    ...

would return an array.

answered Jul 30, 2018 at 5:59

tobi6

8,2666 gold badges31 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dboshardy Over a year ago

This doesn't address pushing from a non-PythonOperator.

tobi6 Over a year ago

Correct. In my version of the question, only "I need to do xcom_pull from a non-PythonOperator" was asked.

bcb · Accepted Answer · 2020-01-06 00:55:49Z

4

From inside an operator's `execute` method:

Push:

self.xcom_push(context, key, value)

Pull:

self.xcom_pull(context, key=key)

If you have a task instance:

Push:

context["ti"].xcom_push(key, value)

Pull:

context["ti"].xcom_pull(key=key)

edited Jan 6, 2020 at 0:55

answered Apr 7, 2019 at 23:37

bcb

2,0782 gold badges23 silver badges22 bronze badges

2 Comments

Sandeep Veerlapati Over a year ago

Would you please elaborate this answer by mentioning, where these lines can be added?

bcb Over a year ago

Ok, added more info.

Collectives™ on Stack Overflow

Airflow: how to use xcom_push and xcom_pull in non-PythonOperator

2 Answers 2

2 Comments

From inside an operator's `execute` method:

If you have a task instance:

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

From inside an operator's execute method:

If you have a task instance:

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related

From inside an operator's `execute` method: