11,944 questions
0
votes
0
answers
10
views
Reparse external dag from current dag in Airflow 3.0.6
We are planning airflow upgrade from 2.7 to 3.0.6. Here we are triggering external dag it is refreshed and having the expected tasks.
with 2.6 version, there was direct access to database but 3.0.7 ...
0
votes
0
answers
34
views
KubernetesExecutor, Airflow 3,SparkSubmitOperator with pod_overwrite fails with json validation error
I'm trying to figure out how to successfully run dag with SparkSubmitOperator on Airflow 3.1.5,
I have a wrapper which sets pod config:
self.executor_config = {
"pod_override": ...
0
votes
0
answers
54
views
How to call an API when I manually marked a state as success in Airflow UI
I have created custom Airflow operator to trigger some of my api when the job submits. Everything works fine but one problem is in Airflow UI we have an option to mark state as success, when a user ...
- 4,946
1
vote
0
answers
71
views
DBT + Snowflake: how to insert on an autoincrement column?
I am migrating a dag and its queries to a new DB using DBT. The problem is that one of the existing tables has an EVENT_ID column, defined as autoincrement, but when the queries try to insert data, ...
0
votes
1
answer
69
views
Airflow DAG run stuck in queued state after being manually triggered
I triggered a DAG run of a new DAG from the command line and the DAG is stuck at "queued" state. Why won't it run? All my other existing DAGs are running normally.
- 23.4k
0
votes
1
answer
40
views
Airflow 3.0.2 LDAP / Active Directory Authentication Not Working with FabAuthManager
Is there anyone who has configured LDAP or Active Directory in Airflow?
When I try to set things inside webserver_config.py, it doesn’t work.
[core]
auth_manager = airflow.providers.fab.auth_manager....
0
votes
1
answer
53
views
Why does Airflow BashOperator fetch multiple days of data when API URL has single-day date range?
I am learning Apache Airflow and created a simple DAG to fetch daily events from a REST API.When I call the API directly in the browser, it correctly returns one day of data
http://localhost:5000/...
- 23
2
votes
1
answer
71
views
Why are the "Last Run" and "Next Run" the same date?
When we run the simple dag in apache airflow the Last Run and Next Run show same date why is that
dag = DAG(
dag_id = '01_unscheduled',
schedule_interval='@daily',
start_date = dt....
- 23
0
votes
0
answers
20
views
Create Application Pools, Roles etc. on startup
We have an Airflow 3.1.5 instance which is deployed to k8s (AKS) using the official helm chart (1.1.8). The database is a PostgreSQL managed database instance.
This works well, however, I would like ...
3
votes
2
answers
201
views
Retrieve the error cause / exception from failed task instances
Let's say I have a DAG with some source1, source2, ... tasks and a sink task, with a simple [source1, source2, ...] >> sink dependency.
All of them are @task-annotated Python operators, and the ...
- 1,257
Best practices
0
votes
0
replies
37
views
Orchestrating multi-step workflow failure handlers
Let's say we have the following dummy data:
item_input = [
{"value": 1, "fail_step": 1},
{"value": 2, "fail_step": 1},
{"value": 3, "...
- 512
1
vote
1
answer
127
views
XCom limit of 1024 in airflow
I am hitting the 1024 item limit in the size of a returned list in my task in airflow. I am currently on airflow v2.11. A simple bit of code that reproduces the issue I see is below.
import os, ...
1
vote
1
answer
138
views
Depedency Hell airflow+dbt
Hello I'm new to airflow, lately I'm struggling on a project with dbt+airflow+docker. My problem 1) I pip install dbt-core, dbt-duckdb adapter, 2) I try to install airflow with:
pip install "...
- 11
0
votes
1
answer
73
views
Add jinja template value with DateTime value
In an Airflow Dag file, I'm trying to handle following jinja template variables as DateTime values.
Specifically {{ data_interval_start }} and {{ data_interval_end }},
which (according to https://...
- 1,434
0
votes
0
answers
55
views
What does "state attribute is queued" mean for airflow 3?
I try to install the new airflow 3 with docker on a VM with this conf:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE}
restart: unless-stopped
env_file: .env
environment:
# --...
- 2,704