[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-astronomer-airflow-state-store":3,"mdc--2wt3om-key":54,"related-repo-astronomer-airflow-state-store":2867,"related-org-astronomer-airflow-state-store":2959},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":49,"sourceUrl":52,"mdContent":53},"airflow-state-store","persist Airflow task and asset state","Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key\u002Fvalue stores (`task_state_store`, `asset_state_store`) and the crash-safe `ResumableJobMixin`. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or \"what's new in Airflow 3.3\". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state — flag the anti-pattern and recommend task_state_store or asset_state_store instead. Requires Airflow 3.3+.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"astronomer","Astronomer","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fastronomer.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Operations","operations","tag",{"name":17,"slug":18,"type":15},"Airflow","airflow",{"name":20,"slug":21,"type":15},"Data Engineering","data-engineering",{"name":23,"slug":24,"type":15},"Data Pipeline","data-pipeline",412,"https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents","2026-07-07T06:43:11.160671",null,55,[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"agentic-workflow","agents","ai","ai-agents","apache-airflow","claude","cursor","dag","data-pipelines","dbt","llm","mcp","orchestrator","skills","workflow-automation","workflow-management","workflow-orchestration","workflows",{"repoUrl":26,"stars":25,"forks":29,"topics":50,"description":51},[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"AI agent tooling for data engineering workflows.","https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents\u002Ftree\u002FHEAD\u002Fskills\u002Fairflow-state-store","---\nname: airflow-state-store\ndescription: Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key\u002Fvalue stores (`task_state_store`, `asset_state_store`) and the crash-safe `ResumableJobMixin`. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or \"what's new in Airflow 3.3\". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state — flag the anti-pattern and recommend task_state_store or asset_state_store instead. Requires Airflow 3.3+.\n---\n\n# Airflow Task State Store (AIP-103)\n\nAirflow 3.3 ships two key\u002Fvalue stores and a crash-safety mixin for operators that submit external jobs.\n\n> **Requires Airflow 3.3+.** Check first:\n> ```bash\n> af config version\n> ```\n> If the version is below 3.3, tell the user these features are not yet available and link them to the AIP-103 tracking issue instead.\n\n---\n\n## Section 1 — Pick the right primitive\n\n| I need to… | Use |\n|---|---|\n| Persist a cursor, offset, or job ID so a retry can resume instead of restart | `task_state_store` |\n| Pass small coordination state within one task across retries (not between tasks) | `task_state_store` |\n| Store a watermark or last-processed timestamp per asset, surviving across DAG runs | `asset_state_store` |\n| Cache asset-level metadata (manifest hash, row count, schema version) | `asset_state_store` |\n| Make an existing non deferrable operator crash-safe when it submits to an external system | `task_state_store` or `ResumableJobMixin` |\n\n**When NOT to use these:**\n- Passing data *between* tasks -> use XCom\n- Large payloads (model weights, dataframes) -> use XCom with an object storage backend\n- Config or secrets shared across DAGs -> use Variables or Connections\n\n---\n\n## Section 2 — Detect anti-patterns in existing DAGs (on demand)\n\nWhen the user asks to review a DAG or asks \"is there a better way\", scan for these patterns and flag them:\n\n| Pattern seen in DAG | Problem | Recommend |\n|---|---|---|\n| `Variable.get(...)` \u002F `Variable.set(...)` inside a `@task` body for per-run state | Variables are global and shared; no scoping to task instance or retry | `task_state_store` |\n| `context[\"ti\"].xcom_push(key=\"job_id\", ...)` to survive retries | XCom is scoped to a DAG run, not a retry; a new ti_id is issued per retry | `task_state_store` or `ResumableJobMixin` |\n| Manual `if Variable.get(\"job_id\"): reconnect else: submit` retry-resume logic | Reimplements what `ResumableJobMixin` already provides, without the crash-safety guarantee | `ResumableJobMixin` |\n| `Variable.set(\"last_processed_at\", ...)` for watermarks | Global; any DAG or task can overwrite it; no scoping to asset | `asset_state_store` |\n\nShow a before\u002Fafter snippet when flagging. Use the canonical examples in Steps 3–5 as the \"after\".\n\n---\n\n## Section 3 — `task_state_store`: per-task coordination state\n\n`task_state_store` is a key\u002Fvalue store scoped to a single task instance identity (dag_id + run_id + task_id + map_index). It survives retries — a new retry on the same task reads the same store.\n\n```python\nfrom airflow.sdk import dag, task\nfrom pendulum import datetime\n\n@dag(start_date=datetime(2025, 1, 1), schedule=\"@daily\")\ndef etl_with_checkpoint():\n\n    @task(retries=3)\n    def process_records(**context):\n        task_state_store = context[\"task_state_store\"]  # injected by Airflow, no setup needed\n        cursor = task_state_store.get(\"last_cursor\", default=0)\n        records = fetch_records_after(cursor)\n        for record in records:\n            process(record)\n            cursor = record[\"id\"]\n            task_state_store.set(\"last_cursor\", cursor)   # checkpoint after each record\n\n    process_records()\n\netl_with_checkpoint()\n```\n\n**API:**\n```python\nfrom airflow.sdk import NEVER_EXPIRE\n\ntask_state_store.get(key, default=None)                        # returns a JsonValue or default\ntask_state_store.set(key, value)                               # uses default_retention_days\ntask_state_store.set(key, value, retention=timedelta(days=7))  # per-key TTL override\ntask_state_store.set(key, value, retention=NEVER_EXPIRE)       # never expires regardless of config\ntask_state_store.delete(key)                                   # no-op if key does not exist\ntask_state_store.clear()                                       # delete all keys for this task instance\n```\n\n**Key rules:**\n- Values must be JSON-serializable (`str`, `int`, `float`, `bool`, `list`, `dict` — `None` values are rejected).\n- Default expiry is controlled by `[state_store] default_retention_days` (0 = never expire).\n- Use `NEVER_EXPIRE` for keys that must outlive the default retention window (e.g. a job ID for a multi-day Spark job).\n- Max value size defaults to 64 KB; configurable via `[state_store] max_value_storage_bytes` (0 = no limit). For larger payloads, configure a custom `[state_store] backend` or a worker side backend configured via: `[workers] state_store_backend`.\n\n**Mapped tasks — each index has its own namespace:**\n\nWhen a task is dynamically mapped (`task.expand(...)`), each map index gets an isolated `task_state_store` scoped to its own `map_index`. Indices do not share state.\n\n```python\n@task(retries=2)\ndef process_partition(partition_id, **context):\n    task_state_store = context[\"task_state_store\"]\n    # Scoped to THIS index only — other indices have their own copy\n    cursor = task_state_store.get(\"cursor\", default=0)\n    task_state_store.set(\"cursor\", new_cursor)\n\nprocess_partition.expand(partition_id=[0, 1, 2, 3])\n```\n\n`clear()` clears only the current index. To wipe state across all map indices of a task group, use the CLI or core API.\n\n**Before (anti-pattern):**\n```python\n@task\ndef process(**context):\n    cursor = Variable.get(\"etl_cursor\", default_var=0)\n    # ... process ...\n    Variable.set(\"etl_cursor\", new_cursor)  # global, any task can overwrite\n```\n\n**After:**\n```python\n@task(retries=3)\ndef process(**context):\n    task_state_store = context[\"task_state_store\"]\n    cursor = task_state_store.get(\"cursor\", default=0)\n    # ... process ...\n    task_state_store.set(\"cursor\", new_cursor)    # scoped to this task instance\n```\n\n---\n\n## Section 4 — `asset_state_store`: per-asset metadata across DAG runs\n\n`asset_state_store` is scoped to an asset, not a task instance. It persists across DAG runs — the same key on the same asset is readable and writable by any task that produces or consumes it.\n\n```python\nfrom airflow.sdk import DAG, Asset, task\nfrom datetime import datetime, timezone\n\nORDERS = Asset(name=\"orders\u002Fdaily\", uri=\"s3:\u002F\u002Fwarehouse\u002Forders\u002Fdaily\")\n\nwith DAG(dag_id=\"producer\", schedule=None, start_date=datetime(2026, 1, 1), catchup=False):\n\n    @task(inlets=[ORDERS], outlets=[ORDERS])\n    def load(asset_state_store=None):        # asset_state_store injected by Airflow — declare as a kwarg\n        asset_state_store = asset_state_store[ORDERS]\n\n        watermark = asset_state_store.get(\"watermark\", default=\"2026-01-01T00:00:00+00:00\")\n        records = fetch_records_since(watermark)\n\n        now = datetime.now(tz=timezone.utc).isoformat()\n        asset_state_store.set(\"watermark\", now)\n        asset_state_store.set(\"last_run_summary\", {\"rows_loaded\": len(records), \"completed_at\": now})\n\n    load()\n```\n\n**Reading the store from a consumer DAG:**\n```python\nwith DAG(dag_id=\"consumer\", schedule=[ORDERS], start_date=datetime(2026, 1, 1), catchup=False):\n\n    @task(inlets=[ORDERS])\n    def consume(asset_state_store=None):\n        asset_state_store = asset_state_store[ORDERS]\n        summary = asset_state_store.get(\"last_run_summary\") or {}\n        print(f\"Processing {summary.get('rows_loaded')} rows up to {asset_state_store.get('watermark')}\")\n\n    consume()\n```\n\n**Key rules:**\n- `asset_state_store` is injected by Airflow as a named kwarg — declare it as `def my_task(asset_state_store=None)`. Do NOT combine with `**context`; Airflow injects it separately.\n- Use `datetime.now(tz=timezone.utc).isoformat()` for timestamps — never `datetime.utcnow()` (not timezone-aware).\n- Same JSON-serializable value constraint as `task_state_store`.\n- No per-key expiry — asset state store entries have no TTL (the asset outlives any single run).\n- Readable by any DAG that declares the asset as an inlet or outlet.\n\n**Mapped tasks — last writer wins:**\n\n`asset_state_store` is scoped to the asset, not the map index. If multiple mapped indices write the same key concurrently, the last write wins. Use distinct keys per index or ensure only one index writes to a given key.\n\n```python\n@task(outlets=[my_asset])\ndef load_partition(partition_id, asset_state_store=None):\n    asset_state_store = asset_state_store[my_asset]\n    # Distinct key per index — no race condition\n    asset_state_store.set(f\"offset_{partition_id}\", new_offset)\n```\n\n**Before (anti-pattern):**\n```python\nVariable.set(f\"watermark_{asset_name}\", new_offset)   # global, not scoped to asset\n```\n\n**After:**\n```python\n@task(inlets=[my_asset], outlets=[my_asset])\ndef load(asset_state_store=None):\n    asset_state_store = asset_state_store[my_asset]\n    asset_state_store.set(\"watermark\", new_offset)\n```\n\n---\n\n## Section 5 — `ResumableJobMixin`: crash-safe external job submission\n\nUse when an operator submits a job to an external system (Spark, Databricks, dbt Cloud, AWS Batch, etc.) and then polls for completion. Without this mixin, a worker crash during polling means the next retry submits a duplicate job.\n\n**When NOT to use `ResumableJobMixin`:**\n\n| Situation | Use instead | Why |\n|---|---|---|\n| A Triggerer is deployed and a deferrable operator exists (or can be written) | Deferrable operator | Frees the worker slot during polling; more resource-efficient |\n| The task fans out many concurrent I\u002FO operations within a single execution | `async def` task \u002F `BaseAsyncOperator` | Async is for high-throughput I\u002FO, not crash recovery |\n| `retries=0` | — | Crash recovery has nothing to reconnect to |\n| The external system has no trackable job ID (`submit_job` returns `None`) | Plain operator | The mixin's crash-safety guarantee is silently disabled; adds no value |\n\n`ResumableJobMixin` holds the worker slot for the full polling duration — the same as a standard synchronous operator. The benefit is crash safety and job continuity, not resource efficiency.\n\n**Opting out of crash recovery:**\n\nThe mixin ships with `durable=True` by default. Set `durable=False` to skip all `task_state_store` interaction and run a plain submit\u002Fpoll\u002Fresult cycle — useful in test environments or when the external system has its own dedup:\n\n```python\nMyBatchOperator(task_id=\"job\", durable=False)\n\n# Or via default_args to disable for all tasks in a DAG:\nwith DAG(\"my_dag\", default_args={\"durable\": False}):\n    ...\n```\n\n### Implementing the mixin\n\n```python\nfrom airflow.sdk import BaseOperator, ResumableJobMixin\nfrom pydantic import JsonValue\n\n\nclass MyBatchOperator(BaseOperator, ResumableJobMixin):\n\n    external_id_key = \"batch_job_id\"   # key used in task_state_store; set once, never rename\n\n    def execute(self, context):\n        return self.execute_resumable(context)  # never call self.execute() — call this\n\n    def submit_job(self, context) -> JsonValue:\n        # Submit and return the job identifier. This value is persisted to task_state_store\n        # before polling starts. Return None only if the system has no trackable ID\n        # (in that case crash-safety is disabled and the job resubmits on every retry).\n        return self.hook.submit_batch(...)\n\n    def get_job_status(self, external_id: JsonValue, context) -> str:\n        # Query the external system. Return a raw status string.\n        return self.hook.get_status(external_id)\n\n    def is_job_active(self, status: str) -> bool:\n        # Return True if the job is still running and should be reconnected to.\n        return status in (\"RUNNING\", \"PENDING\", \"QUEUED\")\n\n    def is_job_succeeded(self, status: str) -> bool:\n        return status == \"SUCCEEDED\"\n\n    def poll_until_complete(self, external_id: JsonValue, context) -> None:\n        # Block until the job reaches a terminal state. Raise on failure.\n        self.hook.wait(external_id)\n\n    def get_job_result(self, external_id: JsonValue, context):\n        # Return the job result after success. Return None if not applicable.\n        return None\n```\n\n### What happens on retry\n\n| Job state on retry | Mixin behaviour |\n|---|---|\n| Still running | Reconnects — calls `poll_until_complete` without resubmitting |\n| Already succeeded | Returns `get_job_result` immediately |\n| Failed \u002F unknown | Submits a fresh job |\n\n### `external_id_key` warning\n\n> **Never rename `external_id_key` on an operator that is already deployed with in-flight task instances.** The old key is stored in `task_state_store` under the previous name. A rename makes the mixin treat every active retry as a fresh submission, defeating the crash-safety guarantee.\n\n### Before (anti-pattern):\n```python\ndef execute(self, context):\n    job_id = Variable.get(\"spark_job_id\", default_var=None)\n    if job_id and self._is_running(job_id):\n        self._wait(job_id)\n    else:\n        job_id = self.hook.submit(...)\n        Variable.set(\"spark_job_id\", job_id)   # global, race-prone\n        self._wait(job_id)\n```\n\n**After:**\n```python\nclass MySparkOperator(BaseOperator, ResumableJobMixin):\n    external_id_key = \"spark_job_id\"\n    def execute(self, context): return self.execute_resumable(context)\n    def submit_job(self, context): return self.hook.submit(...)\n    # ... implement the 5 other methods ...\n```\n\n---\n\n## Section 6 — Configuration reference\n\n```ini\n[state_store]\n# Full dotted path to the storage backend. Default writes to the Airflow metadata DB.\nbackend = airflow.state.metastore.MetastoreStateStoreBackend\n\n# Days to retain task state store entries after their last update. 0 = disable time-based cleanup.\n# Does NOT affect asset_state_store rows — asset state store has no TTL.\ndefault_retention_days = 30\n\n# Rows deleted per batch during cleanup. 0 = no batching (single unbounded delete).\n# Tune on large deployments to reduce lock contention.\nstate_cleanup_batch_size = 0\n\n# Auto-delete all task state store keys when a task succeeds. Default: False.\n# Does NOT affect asset_state_store — asset state store persists across runs and must be cleared explicitly.\nclear_on_success = False\n```\n\n**Worker-side backend** (optional, `[workers]` section) — routes task state store writes through a local backend before they reach the API server. Useful when large payloads or credentialed storage should stay on the worker:\n\n```ini\n[workers]\nstate_store_backend = mypackage.store.WorkerSideBackend\n```\n\n---\n\n## Section 7 — Safety checklist\n\n- [ ] Airflow version ≥ 3.3 (`af config version`)\n- [ ] Values are JSON-serializable (`str`, `int`, `float`, `bool`, `list`, `dict` — no `datetime`, no custom objects)\n- [ ] `task_state_store` keys are short, descriptive strings (avoid dots and slashes)\n- [ ] Mapped tasks writing to `asset_state_store`: use distinct keys per index or accept last-writer-wins semantics\n- [ ] Mapped tasks: fleet-wide state clear uses CLI\u002Fcore API from a downstream task, not `clear()` inside the task body\n- [ ] `ResumableJobMixin`: `external_id_key` is set and will not be renamed after deployment\n- [ ] `ResumableJobMixin`: `execute()` calls `self.execute_resumable(context)`, not custom logic\n- [ ] `ResumableJobMixin`: `durable=False` is intentional if crash recovery is disabled\n- [ ] Large payloads (> configured `max_value_storage_bytes`) use a custom `[state_store] backend` or a worker side backend configured via: `[workers] state_store_backend`\n\n---\n\n## Related skills\n\n- **authoring-dags** — general DAG writing patterns and conventions.\n- **airflow-hitl** — pausing a DAG for human approval (Airflow 3.1+).\n- **airflow** — `af config`, `af registry`, and general Airflow CLI reference.\n",{"data":55,"body":56},{"name":4,"description":6},{"type":57,"children":58},"root",[59,68,74,129,133,140,258,266,294,297,303,308,474,479,482,495,505,682,690,760,768,883,891,919,989,1000,1008,1055,1063,1114,1117,1130,1140,1293,1301,1377,1384,1454,1462,1472,1519,1526,1540,1547,1585,1588,1601,1606,1621,1751,1761,1769,1797,1843,1850,2143,2149,2226,2238,2265,2270,2340,2347,2394,2397,2403,2529,2547,2570,2573,2579,2806,2809,2815,2861],{"type":60,"tag":61,"props":62,"children":64},"element","h1",{"id":63},"airflow-task-state-store-aip-103",[65],{"type":66,"value":67},"text","Airflow Task State Store (AIP-103)",{"type":60,"tag":69,"props":70,"children":71},"p",{},[72],{"type":66,"value":73},"Airflow 3.3 ships two key\u002Fvalue stores and a crash-safety mixin for operators that submit external jobs.",{"type":60,"tag":75,"props":76,"children":77},"blockquote",{},[78,89,124],{"type":60,"tag":69,"props":79,"children":80},{},[81,87],{"type":60,"tag":82,"props":83,"children":84},"strong",{},[85],{"type":66,"value":86},"Requires Airflow 3.3+.",{"type":66,"value":88}," Check first:",{"type":60,"tag":90,"props":91,"children":96},"pre",{"className":92,"code":93,"language":94,"meta":95,"style":95},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","af config version\n","bash","",[97],{"type":60,"tag":98,"props":99,"children":100},"code",{"__ignoreMap":95},[101],{"type":60,"tag":102,"props":103,"children":106},"span",{"class":104,"line":105},"line",1,[107,113,119],{"type":60,"tag":102,"props":108,"children":110},{"style":109},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[111],{"type":66,"value":112},"af",{"type":60,"tag":102,"props":114,"children":116},{"style":115},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[117],{"type":66,"value":118}," config",{"type":60,"tag":102,"props":120,"children":121},{"style":115},[122],{"type":66,"value":123}," version\n",{"type":60,"tag":69,"props":125,"children":126},{},[127],{"type":66,"value":128},"If the version is below 3.3, tell the user these features are not yet available and link them to the AIP-103 tracking issue instead.",{"type":60,"tag":130,"props":131,"children":132},"hr",{},[],{"type":60,"tag":134,"props":135,"children":137},"h2",{"id":136},"section-1-pick-the-right-primitive",[138],{"type":66,"value":139},"Section 1 — Pick the right primitive",{"type":60,"tag":141,"props":142,"children":143},"table",{},[144,163],{"type":60,"tag":145,"props":146,"children":147},"thead",{},[148],{"type":60,"tag":149,"props":150,"children":151},"tr",{},[152,158],{"type":60,"tag":153,"props":154,"children":155},"th",{},[156],{"type":66,"value":157},"I need to…",{"type":60,"tag":153,"props":159,"children":160},{},[161],{"type":66,"value":162},"Use",{"type":60,"tag":164,"props":165,"children":166},"tbody",{},[167,185,201,218,234],{"type":60,"tag":149,"props":168,"children":169},{},[170,176],{"type":60,"tag":171,"props":172,"children":173},"td",{},[174],{"type":66,"value":175},"Persist a cursor, offset, or job ID so a retry can resume instead of restart",{"type":60,"tag":171,"props":177,"children":178},{},[179],{"type":60,"tag":98,"props":180,"children":182},{"className":181},[],[183],{"type":66,"value":184},"task_state_store",{"type":60,"tag":149,"props":186,"children":187},{},[188,193],{"type":60,"tag":171,"props":189,"children":190},{},[191],{"type":66,"value":192},"Pass small coordination state within one task across retries (not between tasks)",{"type":60,"tag":171,"props":194,"children":195},{},[196],{"type":60,"tag":98,"props":197,"children":199},{"className":198},[],[200],{"type":66,"value":184},{"type":60,"tag":149,"props":202,"children":203},{},[204,209],{"type":60,"tag":171,"props":205,"children":206},{},[207],{"type":66,"value":208},"Store a watermark or last-processed timestamp per asset, surviving across DAG runs",{"type":60,"tag":171,"props":210,"children":211},{},[212],{"type":60,"tag":98,"props":213,"children":215},{"className":214},[],[216],{"type":66,"value":217},"asset_state_store",{"type":60,"tag":149,"props":219,"children":220},{},[221,226],{"type":60,"tag":171,"props":222,"children":223},{},[224],{"type":66,"value":225},"Cache asset-level metadata (manifest hash, row count, schema version)",{"type":60,"tag":171,"props":227,"children":228},{},[229],{"type":60,"tag":98,"props":230,"children":232},{"className":231},[],[233],{"type":66,"value":217},{"type":60,"tag":149,"props":235,"children":236},{},[237,242],{"type":60,"tag":171,"props":238,"children":239},{},[240],{"type":66,"value":241},"Make an existing non deferrable operator crash-safe when it submits to an external system",{"type":60,"tag":171,"props":243,"children":244},{},[245,250,252],{"type":60,"tag":98,"props":246,"children":248},{"className":247},[],[249],{"type":66,"value":184},{"type":66,"value":251}," or ",{"type":60,"tag":98,"props":253,"children":255},{"className":254},[],[256],{"type":66,"value":257},"ResumableJobMixin",{"type":60,"tag":69,"props":259,"children":260},{},[261],{"type":60,"tag":82,"props":262,"children":263},{},[264],{"type":66,"value":265},"When NOT to use these:",{"type":60,"tag":267,"props":268,"children":269},"ul",{},[270,284,289],{"type":60,"tag":271,"props":272,"children":273},"li",{},[274,276,282],{"type":66,"value":275},"Passing data ",{"type":60,"tag":277,"props":278,"children":279},"em",{},[280],{"type":66,"value":281},"between",{"type":66,"value":283}," tasks -> use XCom",{"type":60,"tag":271,"props":285,"children":286},{},[287],{"type":66,"value":288},"Large payloads (model weights, dataframes) -> use XCom with an object storage backend",{"type":60,"tag":271,"props":290,"children":291},{},[292],{"type":66,"value":293},"Config or secrets shared across DAGs -> use Variables or Connections",{"type":60,"tag":130,"props":295,"children":296},{},[],{"type":60,"tag":134,"props":298,"children":300},{"id":299},"section-2-detect-anti-patterns-in-existing-dags-on-demand",[301],{"type":66,"value":302},"Section 2 — Detect anti-patterns in existing DAGs (on demand)",{"type":60,"tag":69,"props":304,"children":305},{},[306],{"type":66,"value":307},"When the user asks to review a DAG or asks \"is there a better way\", scan for these patterns and flag them:",{"type":60,"tag":141,"props":309,"children":310},{},[311,332],{"type":60,"tag":145,"props":312,"children":313},{},[314],{"type":60,"tag":149,"props":315,"children":316},{},[317,322,327],{"type":60,"tag":153,"props":318,"children":319},{},[320],{"type":66,"value":321},"Pattern seen in DAG",{"type":60,"tag":153,"props":323,"children":324},{},[325],{"type":66,"value":326},"Problem",{"type":60,"tag":153,"props":328,"children":329},{},[330],{"type":66,"value":331},"Recommend",{"type":60,"tag":164,"props":333,"children":334},{},[335,378,411,447],{"type":60,"tag":149,"props":336,"children":337},{},[338,365,370],{"type":60,"tag":171,"props":339,"children":340},{},[341,347,349,355,357,363],{"type":60,"tag":98,"props":342,"children":344},{"className":343},[],[345],{"type":66,"value":346},"Variable.get(...)",{"type":66,"value":348}," \u002F ",{"type":60,"tag":98,"props":350,"children":352},{"className":351},[],[353],{"type":66,"value":354},"Variable.set(...)",{"type":66,"value":356}," inside a ",{"type":60,"tag":98,"props":358,"children":360},{"className":359},[],[361],{"type":66,"value":362},"@task",{"type":66,"value":364}," body for per-run state",{"type":60,"tag":171,"props":366,"children":367},{},[368],{"type":66,"value":369},"Variables are global and shared; no scoping to task instance or retry",{"type":60,"tag":171,"props":371,"children":372},{},[373],{"type":60,"tag":98,"props":374,"children":376},{"className":375},[],[377],{"type":66,"value":184},{"type":60,"tag":149,"props":379,"children":380},{},[381,392,397],{"type":60,"tag":171,"props":382,"children":383},{},[384,390],{"type":60,"tag":98,"props":385,"children":387},{"className":386},[],[388],{"type":66,"value":389},"context[\"ti\"].xcom_push(key=\"job_id\", ...)",{"type":66,"value":391}," to survive retries",{"type":60,"tag":171,"props":393,"children":394},{},[395],{"type":66,"value":396},"XCom is scoped to a DAG run, not a retry; a new ti_id is issued per retry",{"type":60,"tag":171,"props":398,"children":399},{},[400,405,406],{"type":60,"tag":98,"props":401,"children":403},{"className":402},[],[404],{"type":66,"value":184},{"type":66,"value":251},{"type":60,"tag":98,"props":407,"children":409},{"className":408},[],[410],{"type":66,"value":257},{"type":60,"tag":149,"props":412,"children":413},{},[414,427,439],{"type":60,"tag":171,"props":415,"children":416},{},[417,419,425],{"type":66,"value":418},"Manual ",{"type":60,"tag":98,"props":420,"children":422},{"className":421},[],[423],{"type":66,"value":424},"if Variable.get(\"job_id\"): reconnect else: submit",{"type":66,"value":426}," retry-resume logic",{"type":60,"tag":171,"props":428,"children":429},{},[430,432,437],{"type":66,"value":431},"Reimplements what ",{"type":60,"tag":98,"props":433,"children":435},{"className":434},[],[436],{"type":66,"value":257},{"type":66,"value":438}," already provides, without the crash-safety guarantee",{"type":60,"tag":171,"props":440,"children":441},{},[442],{"type":60,"tag":98,"props":443,"children":445},{"className":444},[],[446],{"type":66,"value":257},{"type":60,"tag":149,"props":448,"children":449},{},[450,461,466],{"type":60,"tag":171,"props":451,"children":452},{},[453,459],{"type":60,"tag":98,"props":454,"children":456},{"className":455},[],[457],{"type":66,"value":458},"Variable.set(\"last_processed_at\", ...)",{"type":66,"value":460}," for watermarks",{"type":60,"tag":171,"props":462,"children":463},{},[464],{"type":66,"value":465},"Global; any DAG or task can overwrite it; no scoping to asset",{"type":60,"tag":171,"props":467,"children":468},{},[469],{"type":60,"tag":98,"props":470,"children":472},{"className":471},[],[473],{"type":66,"value":217},{"type":60,"tag":69,"props":475,"children":476},{},[477],{"type":66,"value":478},"Show a before\u002Fafter snippet when flagging. Use the canonical examples in Steps 3–5 as the \"after\".",{"type":60,"tag":130,"props":480,"children":481},{},[],{"type":60,"tag":134,"props":483,"children":485},{"id":484},"section-3-task_state_store-per-task-coordination-state",[486,488,493],{"type":66,"value":487},"Section 3 — ",{"type":60,"tag":98,"props":489,"children":491},{"className":490},[],[492],{"type":66,"value":184},{"type":66,"value":494},": per-task coordination state",{"type":60,"tag":69,"props":496,"children":497},{},[498,503],{"type":60,"tag":98,"props":499,"children":501},{"className":500},[],[502],{"type":66,"value":184},{"type":66,"value":504}," is a key\u002Fvalue store scoped to a single task instance identity (dag_id + run_id + task_id + map_index). It survives retries — a new retry on the same task reads the same store.",{"type":60,"tag":90,"props":506,"children":510},{"className":507,"code":508,"language":509,"meta":95,"style":95},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from airflow.sdk import dag, task\nfrom pendulum import datetime\n\n@dag(start_date=datetime(2025, 1, 1), schedule=\"@daily\")\ndef etl_with_checkpoint():\n\n    @task(retries=3)\n    def process_records(**context):\n        task_state_store = context[\"task_state_store\"]  # injected by Airflow, no setup needed\n        cursor = task_state_store.get(\"last_cursor\", default=0)\n        records = fetch_records_after(cursor)\n        for record in records:\n            process(record)\n            cursor = record[\"id\"]\n            task_state_store.set(\"last_cursor\", cursor)   # checkpoint after each record\n\n    process_records()\n\netl_with_checkpoint()\n","python",[511],{"type":60,"tag":98,"props":512,"children":513},{"__ignoreMap":95},[514,522,531,541,550,559,567,576,585,594,603,612,621,630,639,648,656,665,673],{"type":60,"tag":102,"props":515,"children":516},{"class":104,"line":105},[517],{"type":60,"tag":102,"props":518,"children":519},{},[520],{"type":66,"value":521},"from airflow.sdk import dag, task\n",{"type":60,"tag":102,"props":523,"children":525},{"class":104,"line":524},2,[526],{"type":60,"tag":102,"props":527,"children":528},{},[529],{"type":66,"value":530},"from pendulum import datetime\n",{"type":60,"tag":102,"props":532,"children":534},{"class":104,"line":533},3,[535],{"type":60,"tag":102,"props":536,"children":538},{"emptyLinePlaceholder":537},true,[539],{"type":66,"value":540},"\n",{"type":60,"tag":102,"props":542,"children":544},{"class":104,"line":543},4,[545],{"type":60,"tag":102,"props":546,"children":547},{},[548],{"type":66,"value":549},"@dag(start_date=datetime(2025, 1, 1), schedule=\"@daily\")\n",{"type":60,"tag":102,"props":551,"children":553},{"class":104,"line":552},5,[554],{"type":60,"tag":102,"props":555,"children":556},{},[557],{"type":66,"value":558},"def etl_with_checkpoint():\n",{"type":60,"tag":102,"props":560,"children":562},{"class":104,"line":561},6,[563],{"type":60,"tag":102,"props":564,"children":565},{"emptyLinePlaceholder":537},[566],{"type":66,"value":540},{"type":60,"tag":102,"props":568,"children":570},{"class":104,"line":569},7,[571],{"type":60,"tag":102,"props":572,"children":573},{},[574],{"type":66,"value":575},"    @task(retries=3)\n",{"type":60,"tag":102,"props":577,"children":579},{"class":104,"line":578},8,[580],{"type":60,"tag":102,"props":581,"children":582},{},[583],{"type":66,"value":584},"    def process_records(**context):\n",{"type":60,"tag":102,"props":586,"children":588},{"class":104,"line":587},9,[589],{"type":60,"tag":102,"props":590,"children":591},{},[592],{"type":66,"value":593},"        task_state_store = context[\"task_state_store\"]  # injected by Airflow, no setup needed\n",{"type":60,"tag":102,"props":595,"children":597},{"class":104,"line":596},10,[598],{"type":60,"tag":102,"props":599,"children":600},{},[601],{"type":66,"value":602},"        cursor = task_state_store.get(\"last_cursor\", default=0)\n",{"type":60,"tag":102,"props":604,"children":606},{"class":104,"line":605},11,[607],{"type":60,"tag":102,"props":608,"children":609},{},[610],{"type":66,"value":611},"        records = fetch_records_after(cursor)\n",{"type":60,"tag":102,"props":613,"children":615},{"class":104,"line":614},12,[616],{"type":60,"tag":102,"props":617,"children":618},{},[619],{"type":66,"value":620},"        for record in records:\n",{"type":60,"tag":102,"props":622,"children":624},{"class":104,"line":623},13,[625],{"type":60,"tag":102,"props":626,"children":627},{},[628],{"type":66,"value":629},"            process(record)\n",{"type":60,"tag":102,"props":631,"children":633},{"class":104,"line":632},14,[634],{"type":60,"tag":102,"props":635,"children":636},{},[637],{"type":66,"value":638},"            cursor = record[\"id\"]\n",{"type":60,"tag":102,"props":640,"children":642},{"class":104,"line":641},15,[643],{"type":60,"tag":102,"props":644,"children":645},{},[646],{"type":66,"value":647},"            task_state_store.set(\"last_cursor\", cursor)   # checkpoint after each record\n",{"type":60,"tag":102,"props":649,"children":651},{"class":104,"line":650},16,[652],{"type":60,"tag":102,"props":653,"children":654},{"emptyLinePlaceholder":537},[655],{"type":66,"value":540},{"type":60,"tag":102,"props":657,"children":659},{"class":104,"line":658},17,[660],{"type":60,"tag":102,"props":661,"children":662},{},[663],{"type":66,"value":664},"    process_records()\n",{"type":60,"tag":102,"props":666,"children":668},{"class":104,"line":667},18,[669],{"type":60,"tag":102,"props":670,"children":671},{"emptyLinePlaceholder":537},[672],{"type":66,"value":540},{"type":60,"tag":102,"props":674,"children":676},{"class":104,"line":675},19,[677],{"type":60,"tag":102,"props":678,"children":679},{},[680],{"type":66,"value":681},"etl_with_checkpoint()\n",{"type":60,"tag":69,"props":683,"children":684},{},[685],{"type":60,"tag":82,"props":686,"children":687},{},[688],{"type":66,"value":689},"API:",{"type":60,"tag":90,"props":691,"children":693},{"className":507,"code":692,"language":509,"meta":95,"style":95},"from airflow.sdk import NEVER_EXPIRE\n\ntask_state_store.get(key, default=None)                        # returns a JsonValue or default\ntask_state_store.set(key, value)                               # uses default_retention_days\ntask_state_store.set(key, value, retention=timedelta(days=7))  # per-key TTL override\ntask_state_store.set(key, value, retention=NEVER_EXPIRE)       # never expires regardless of config\ntask_state_store.delete(key)                                   # no-op if key does not exist\ntask_state_store.clear()                                       # delete all keys for this task instance\n",[694],{"type":60,"tag":98,"props":695,"children":696},{"__ignoreMap":95},[697,705,712,720,728,736,744,752],{"type":60,"tag":102,"props":698,"children":699},{"class":104,"line":105},[700],{"type":60,"tag":102,"props":701,"children":702},{},[703],{"type":66,"value":704},"from airflow.sdk import NEVER_EXPIRE\n",{"type":60,"tag":102,"props":706,"children":707},{"class":104,"line":524},[708],{"type":60,"tag":102,"props":709,"children":710},{"emptyLinePlaceholder":537},[711],{"type":66,"value":540},{"type":60,"tag":102,"props":713,"children":714},{"class":104,"line":533},[715],{"type":60,"tag":102,"props":716,"children":717},{},[718],{"type":66,"value":719},"task_state_store.get(key, default=None)                        # returns a JsonValue or default\n",{"type":60,"tag":102,"props":721,"children":722},{"class":104,"line":543},[723],{"type":60,"tag":102,"props":724,"children":725},{},[726],{"type":66,"value":727},"task_state_store.set(key, value)                               # uses default_retention_days\n",{"type":60,"tag":102,"props":729,"children":730},{"class":104,"line":552},[731],{"type":60,"tag":102,"props":732,"children":733},{},[734],{"type":66,"value":735},"task_state_store.set(key, value, retention=timedelta(days=7))  # per-key TTL override\n",{"type":60,"tag":102,"props":737,"children":738},{"class":104,"line":561},[739],{"type":60,"tag":102,"props":740,"children":741},{},[742],{"type":66,"value":743},"task_state_store.set(key, value, retention=NEVER_EXPIRE)       # never expires regardless of config\n",{"type":60,"tag":102,"props":745,"children":746},{"class":104,"line":569},[747],{"type":60,"tag":102,"props":748,"children":749},{},[750],{"type":66,"value":751},"task_state_store.delete(key)                                   # no-op if key does not exist\n",{"type":60,"tag":102,"props":753,"children":754},{"class":104,"line":578},[755],{"type":60,"tag":102,"props":756,"children":757},{},[758],{"type":66,"value":759},"task_state_store.clear()                                       # delete all keys for this task instance\n",{"type":60,"tag":69,"props":761,"children":762},{},[763],{"type":60,"tag":82,"props":764,"children":765},{},[766],{"type":66,"value":767},"Key rules:",{"type":60,"tag":267,"props":769,"children":770},{},[771,828,841,854],{"type":60,"tag":271,"props":772,"children":773},{},[774,776,782,784,790,791,797,798,804,805,811,812,818,820,826],{"type":66,"value":775},"Values must be JSON-serializable (",{"type":60,"tag":98,"props":777,"children":779},{"className":778},[],[780],{"type":66,"value":781},"str",{"type":66,"value":783},", ",{"type":60,"tag":98,"props":785,"children":787},{"className":786},[],[788],{"type":66,"value":789},"int",{"type":66,"value":783},{"type":60,"tag":98,"props":792,"children":794},{"className":793},[],[795],{"type":66,"value":796},"float",{"type":66,"value":783},{"type":60,"tag":98,"props":799,"children":801},{"className":800},[],[802],{"type":66,"value":803},"bool",{"type":66,"value":783},{"type":60,"tag":98,"props":806,"children":808},{"className":807},[],[809],{"type":66,"value":810},"list",{"type":66,"value":783},{"type":60,"tag":98,"props":813,"children":815},{"className":814},[],[816],{"type":66,"value":817},"dict",{"type":66,"value":819}," — ",{"type":60,"tag":98,"props":821,"children":823},{"className":822},[],[824],{"type":66,"value":825},"None",{"type":66,"value":827}," values are rejected).",{"type":60,"tag":271,"props":829,"children":830},{},[831,833,839],{"type":66,"value":832},"Default expiry is controlled by ",{"type":60,"tag":98,"props":834,"children":836},{"className":835},[],[837],{"type":66,"value":838},"[state_store] default_retention_days",{"type":66,"value":840}," (0 = never expire).",{"type":60,"tag":271,"props":842,"children":843},{},[844,846,852],{"type":66,"value":845},"Use ",{"type":60,"tag":98,"props":847,"children":849},{"className":848},[],[850],{"type":66,"value":851},"NEVER_EXPIRE",{"type":66,"value":853}," for keys that must outlive the default retention window (e.g. a job ID for a multi-day Spark job).",{"type":60,"tag":271,"props":855,"children":856},{},[857,859,865,867,873,875,881],{"type":66,"value":858},"Max value size defaults to 64 KB; configurable via ",{"type":60,"tag":98,"props":860,"children":862},{"className":861},[],[863],{"type":66,"value":864},"[state_store] max_value_storage_bytes",{"type":66,"value":866}," (0 = no limit). For larger payloads, configure a custom ",{"type":60,"tag":98,"props":868,"children":870},{"className":869},[],[871],{"type":66,"value":872},"[state_store] backend",{"type":66,"value":874}," or a worker side backend configured via: ",{"type":60,"tag":98,"props":876,"children":878},{"className":877},[],[879],{"type":66,"value":880},"[workers] state_store_backend",{"type":66,"value":882},".",{"type":60,"tag":69,"props":884,"children":885},{},[886],{"type":60,"tag":82,"props":887,"children":888},{},[889],{"type":66,"value":890},"Mapped tasks — each index has its own namespace:",{"type":60,"tag":69,"props":892,"children":893},{},[894,896,902,904,909,911,917],{"type":66,"value":895},"When a task is dynamically mapped (",{"type":60,"tag":98,"props":897,"children":899},{"className":898},[],[900],{"type":66,"value":901},"task.expand(...)",{"type":66,"value":903},"), each map index gets an isolated ",{"type":60,"tag":98,"props":905,"children":907},{"className":906},[],[908],{"type":66,"value":184},{"type":66,"value":910}," scoped to its own ",{"type":60,"tag":98,"props":912,"children":914},{"className":913},[],[915],{"type":66,"value":916},"map_index",{"type":66,"value":918},". Indices do not share state.",{"type":60,"tag":90,"props":920,"children":922},{"className":507,"code":921,"language":509,"meta":95,"style":95},"@task(retries=2)\ndef process_partition(partition_id, **context):\n    task_state_store = context[\"task_state_store\"]\n    # Scoped to THIS index only — other indices have their own copy\n    cursor = task_state_store.get(\"cursor\", default=0)\n    task_state_store.set(\"cursor\", new_cursor)\n\nprocess_partition.expand(partition_id=[0, 1, 2, 3])\n",[923],{"type":60,"tag":98,"props":924,"children":925},{"__ignoreMap":95},[926,934,942,950,958,966,974,981],{"type":60,"tag":102,"props":927,"children":928},{"class":104,"line":105},[929],{"type":60,"tag":102,"props":930,"children":931},{},[932],{"type":66,"value":933},"@task(retries=2)\n",{"type":60,"tag":102,"props":935,"children":936},{"class":104,"line":524},[937],{"type":60,"tag":102,"props":938,"children":939},{},[940],{"type":66,"value":941},"def process_partition(partition_id, **context):\n",{"type":60,"tag":102,"props":943,"children":944},{"class":104,"line":533},[945],{"type":60,"tag":102,"props":946,"children":947},{},[948],{"type":66,"value":949},"    task_state_store = context[\"task_state_store\"]\n",{"type":60,"tag":102,"props":951,"children":952},{"class":104,"line":543},[953],{"type":60,"tag":102,"props":954,"children":955},{},[956],{"type":66,"value":957},"    # Scoped to THIS index only — other indices have their own copy\n",{"type":60,"tag":102,"props":959,"children":960},{"class":104,"line":552},[961],{"type":60,"tag":102,"props":962,"children":963},{},[964],{"type":66,"value":965},"    cursor = task_state_store.get(\"cursor\", default=0)\n",{"type":60,"tag":102,"props":967,"children":968},{"class":104,"line":561},[969],{"type":60,"tag":102,"props":970,"children":971},{},[972],{"type":66,"value":973},"    task_state_store.set(\"cursor\", new_cursor)\n",{"type":60,"tag":102,"props":975,"children":976},{"class":104,"line":569},[977],{"type":60,"tag":102,"props":978,"children":979},{"emptyLinePlaceholder":537},[980],{"type":66,"value":540},{"type":60,"tag":102,"props":982,"children":983},{"class":104,"line":578},[984],{"type":60,"tag":102,"props":985,"children":986},{},[987],{"type":66,"value":988},"process_partition.expand(partition_id=[0, 1, 2, 3])\n",{"type":60,"tag":69,"props":990,"children":991},{},[992,998],{"type":60,"tag":98,"props":993,"children":995},{"className":994},[],[996],{"type":66,"value":997},"clear()",{"type":66,"value":999}," clears only the current index. To wipe state across all map indices of a task group, use the CLI or core API.",{"type":60,"tag":69,"props":1001,"children":1002},{},[1003],{"type":60,"tag":82,"props":1004,"children":1005},{},[1006],{"type":66,"value":1007},"Before (anti-pattern):",{"type":60,"tag":90,"props":1009,"children":1011},{"className":507,"code":1010,"language":509,"meta":95,"style":95},"@task\ndef process(**context):\n    cursor = Variable.get(\"etl_cursor\", default_var=0)\n    # ... process ...\n    Variable.set(\"etl_cursor\", new_cursor)  # global, any task can overwrite\n",[1012],{"type":60,"tag":98,"props":1013,"children":1014},{"__ignoreMap":95},[1015,1023,1031,1039,1047],{"type":60,"tag":102,"props":1016,"children":1017},{"class":104,"line":105},[1018],{"type":60,"tag":102,"props":1019,"children":1020},{},[1021],{"type":66,"value":1022},"@task\n",{"type":60,"tag":102,"props":1024,"children":1025},{"class":104,"line":524},[1026],{"type":60,"tag":102,"props":1027,"children":1028},{},[1029],{"type":66,"value":1030},"def process(**context):\n",{"type":60,"tag":102,"props":1032,"children":1033},{"class":104,"line":533},[1034],{"type":60,"tag":102,"props":1035,"children":1036},{},[1037],{"type":66,"value":1038},"    cursor = Variable.get(\"etl_cursor\", default_var=0)\n",{"type":60,"tag":102,"props":1040,"children":1041},{"class":104,"line":543},[1042],{"type":60,"tag":102,"props":1043,"children":1044},{},[1045],{"type":66,"value":1046},"    # ... process ...\n",{"type":60,"tag":102,"props":1048,"children":1049},{"class":104,"line":552},[1050],{"type":60,"tag":102,"props":1051,"children":1052},{},[1053],{"type":66,"value":1054},"    Variable.set(\"etl_cursor\", new_cursor)  # global, any task can overwrite\n",{"type":60,"tag":69,"props":1056,"children":1057},{},[1058],{"type":60,"tag":82,"props":1059,"children":1060},{},[1061],{"type":66,"value":1062},"After:",{"type":60,"tag":90,"props":1064,"children":1066},{"className":507,"code":1065,"language":509,"meta":95,"style":95},"@task(retries=3)\ndef process(**context):\n    task_state_store = context[\"task_state_store\"]\n    cursor = task_state_store.get(\"cursor\", default=0)\n    # ... process ...\n    task_state_store.set(\"cursor\", new_cursor)    # scoped to this task instance\n",[1067],{"type":60,"tag":98,"props":1068,"children":1069},{"__ignoreMap":95},[1070,1078,1085,1092,1099,1106],{"type":60,"tag":102,"props":1071,"children":1072},{"class":104,"line":105},[1073],{"type":60,"tag":102,"props":1074,"children":1075},{},[1076],{"type":66,"value":1077},"@task(retries=3)\n",{"type":60,"tag":102,"props":1079,"children":1080},{"class":104,"line":524},[1081],{"type":60,"tag":102,"props":1082,"children":1083},{},[1084],{"type":66,"value":1030},{"type":60,"tag":102,"props":1086,"children":1087},{"class":104,"line":533},[1088],{"type":60,"tag":102,"props":1089,"children":1090},{},[1091],{"type":66,"value":949},{"type":60,"tag":102,"props":1093,"children":1094},{"class":104,"line":543},[1095],{"type":60,"tag":102,"props":1096,"children":1097},{},[1098],{"type":66,"value":965},{"type":60,"tag":102,"props":1100,"children":1101},{"class":104,"line":552},[1102],{"type":60,"tag":102,"props":1103,"children":1104},{},[1105],{"type":66,"value":1046},{"type":60,"tag":102,"props":1107,"children":1108},{"class":104,"line":561},[1109],{"type":60,"tag":102,"props":1110,"children":1111},{},[1112],{"type":66,"value":1113},"    task_state_store.set(\"cursor\", new_cursor)    # scoped to this task instance\n",{"type":60,"tag":130,"props":1115,"children":1116},{},[],{"type":60,"tag":134,"props":1118,"children":1120},{"id":1119},"section-4-asset_state_store-per-asset-metadata-across-dag-runs",[1121,1123,1128],{"type":66,"value":1122},"Section 4 — ",{"type":60,"tag":98,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":66,"value":217},{"type":66,"value":1129},": per-asset metadata across DAG runs",{"type":60,"tag":69,"props":1131,"children":1132},{},[1133,1138],{"type":60,"tag":98,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":66,"value":217},{"type":66,"value":1139}," is scoped to an asset, not a task instance. It persists across DAG runs — the same key on the same asset is readable and writable by any task that produces or consumes it.",{"type":60,"tag":90,"props":1141,"children":1143},{"className":507,"code":1142,"language":509,"meta":95,"style":95},"from airflow.sdk import DAG, Asset, task\nfrom datetime import datetime, timezone\n\nORDERS = Asset(name=\"orders\u002Fdaily\", uri=\"s3:\u002F\u002Fwarehouse\u002Forders\u002Fdaily\")\n\nwith DAG(dag_id=\"producer\", schedule=None, start_date=datetime(2026, 1, 1), catchup=False):\n\n    @task(inlets=[ORDERS], outlets=[ORDERS])\n    def load(asset_state_store=None):        # asset_state_store injected by Airflow — declare as a kwarg\n        asset_state_store = asset_state_store[ORDERS]\n\n        watermark = asset_state_store.get(\"watermark\", default=\"2026-01-01T00:00:00+00:00\")\n        records = fetch_records_since(watermark)\n\n        now = datetime.now(tz=timezone.utc).isoformat()\n        asset_state_store.set(\"watermark\", now)\n        asset_state_store.set(\"last_run_summary\", {\"rows_loaded\": len(records), \"completed_at\": now})\n\n    load()\n",[1144],{"type":60,"tag":98,"props":1145,"children":1146},{"__ignoreMap":95},[1147,1155,1163,1170,1178,1185,1193,1200,1208,1216,1224,1231,1239,1247,1254,1262,1270,1278,1285],{"type":60,"tag":102,"props":1148,"children":1149},{"class":104,"line":105},[1150],{"type":60,"tag":102,"props":1151,"children":1152},{},[1153],{"type":66,"value":1154},"from airflow.sdk import DAG, Asset, task\n",{"type":60,"tag":102,"props":1156,"children":1157},{"class":104,"line":524},[1158],{"type":60,"tag":102,"props":1159,"children":1160},{},[1161],{"type":66,"value":1162},"from datetime import datetime, timezone\n",{"type":60,"tag":102,"props":1164,"children":1165},{"class":104,"line":533},[1166],{"type":60,"tag":102,"props":1167,"children":1168},{"emptyLinePlaceholder":537},[1169],{"type":66,"value":540},{"type":60,"tag":102,"props":1171,"children":1172},{"class":104,"line":543},[1173],{"type":60,"tag":102,"props":1174,"children":1175},{},[1176],{"type":66,"value":1177},"ORDERS = Asset(name=\"orders\u002Fdaily\", uri=\"s3:\u002F\u002Fwarehouse\u002Forders\u002Fdaily\")\n",{"type":60,"tag":102,"props":1179,"children":1180},{"class":104,"line":552},[1181],{"type":60,"tag":102,"props":1182,"children":1183},{"emptyLinePlaceholder":537},[1184],{"type":66,"value":540},{"type":60,"tag":102,"props":1186,"children":1187},{"class":104,"line":561},[1188],{"type":60,"tag":102,"props":1189,"children":1190},{},[1191],{"type":66,"value":1192},"with DAG(dag_id=\"producer\", schedule=None, start_date=datetime(2026, 1, 1), catchup=False):\n",{"type":60,"tag":102,"props":1194,"children":1195},{"class":104,"line":569},[1196],{"type":60,"tag":102,"props":1197,"children":1198},{"emptyLinePlaceholder":537},[1199],{"type":66,"value":540},{"type":60,"tag":102,"props":1201,"children":1202},{"class":104,"line":578},[1203],{"type":60,"tag":102,"props":1204,"children":1205},{},[1206],{"type":66,"value":1207},"    @task(inlets=[ORDERS], outlets=[ORDERS])\n",{"type":60,"tag":102,"props":1209,"children":1210},{"class":104,"line":587},[1211],{"type":60,"tag":102,"props":1212,"children":1213},{},[1214],{"type":66,"value":1215},"    def load(asset_state_store=None):        # asset_state_store injected by Airflow — declare as a kwarg\n",{"type":60,"tag":102,"props":1217,"children":1218},{"class":104,"line":596},[1219],{"type":60,"tag":102,"props":1220,"children":1221},{},[1222],{"type":66,"value":1223},"        asset_state_store = asset_state_store[ORDERS]\n",{"type":60,"tag":102,"props":1225,"children":1226},{"class":104,"line":605},[1227],{"type":60,"tag":102,"props":1228,"children":1229},{"emptyLinePlaceholder":537},[1230],{"type":66,"value":540},{"type":60,"tag":102,"props":1232,"children":1233},{"class":104,"line":614},[1234],{"type":60,"tag":102,"props":1235,"children":1236},{},[1237],{"type":66,"value":1238},"        watermark = asset_state_store.get(\"watermark\", default=\"2026-01-01T00:00:00+00:00\")\n",{"type":60,"tag":102,"props":1240,"children":1241},{"class":104,"line":623},[1242],{"type":60,"tag":102,"props":1243,"children":1244},{},[1245],{"type":66,"value":1246},"        records = fetch_records_since(watermark)\n",{"type":60,"tag":102,"props":1248,"children":1249},{"class":104,"line":632},[1250],{"type":60,"tag":102,"props":1251,"children":1252},{"emptyLinePlaceholder":537},[1253],{"type":66,"value":540},{"type":60,"tag":102,"props":1255,"children":1256},{"class":104,"line":641},[1257],{"type":60,"tag":102,"props":1258,"children":1259},{},[1260],{"type":66,"value":1261},"        now = datetime.now(tz=timezone.utc).isoformat()\n",{"type":60,"tag":102,"props":1263,"children":1264},{"class":104,"line":650},[1265],{"type":60,"tag":102,"props":1266,"children":1267},{},[1268],{"type":66,"value":1269},"        asset_state_store.set(\"watermark\", now)\n",{"type":60,"tag":102,"props":1271,"children":1272},{"class":104,"line":658},[1273],{"type":60,"tag":102,"props":1274,"children":1275},{},[1276],{"type":66,"value":1277},"        asset_state_store.set(\"last_run_summary\", {\"rows_loaded\": len(records), \"completed_at\": now})\n",{"type":60,"tag":102,"props":1279,"children":1280},{"class":104,"line":667},[1281],{"type":60,"tag":102,"props":1282,"children":1283},{"emptyLinePlaceholder":537},[1284],{"type":66,"value":540},{"type":60,"tag":102,"props":1286,"children":1287},{"class":104,"line":675},[1288],{"type":60,"tag":102,"props":1289,"children":1290},{},[1291],{"type":66,"value":1292},"    load()\n",{"type":60,"tag":69,"props":1294,"children":1295},{},[1296],{"type":60,"tag":82,"props":1297,"children":1298},{},[1299],{"type":66,"value":1300},"Reading the store from a consumer DAG:",{"type":60,"tag":90,"props":1302,"children":1304},{"className":507,"code":1303,"language":509,"meta":95,"style":95},"with DAG(dag_id=\"consumer\", schedule=[ORDERS], start_date=datetime(2026, 1, 1), catchup=False):\n\n    @task(inlets=[ORDERS])\n    def consume(asset_state_store=None):\n        asset_state_store = asset_state_store[ORDERS]\n        summary = asset_state_store.get(\"last_run_summary\") or {}\n        print(f\"Processing {summary.get('rows_loaded')} rows up to {asset_state_store.get('watermark')}\")\n\n    consume()\n",[1305],{"type":60,"tag":98,"props":1306,"children":1307},{"__ignoreMap":95},[1308,1316,1323,1331,1339,1346,1354,1362,1369],{"type":60,"tag":102,"props":1309,"children":1310},{"class":104,"line":105},[1311],{"type":60,"tag":102,"props":1312,"children":1313},{},[1314],{"type":66,"value":1315},"with DAG(dag_id=\"consumer\", schedule=[ORDERS], start_date=datetime(2026, 1, 1), catchup=False):\n",{"type":60,"tag":102,"props":1317,"children":1318},{"class":104,"line":524},[1319],{"type":60,"tag":102,"props":1320,"children":1321},{"emptyLinePlaceholder":537},[1322],{"type":66,"value":540},{"type":60,"tag":102,"props":1324,"children":1325},{"class":104,"line":533},[1326],{"type":60,"tag":102,"props":1327,"children":1328},{},[1329],{"type":66,"value":1330},"    @task(inlets=[ORDERS])\n",{"type":60,"tag":102,"props":1332,"children":1333},{"class":104,"line":543},[1334],{"type":60,"tag":102,"props":1335,"children":1336},{},[1337],{"type":66,"value":1338},"    def consume(asset_state_store=None):\n",{"type":60,"tag":102,"props":1340,"children":1341},{"class":104,"line":552},[1342],{"type":60,"tag":102,"props":1343,"children":1344},{},[1345],{"type":66,"value":1223},{"type":60,"tag":102,"props":1347,"children":1348},{"class":104,"line":561},[1349],{"type":60,"tag":102,"props":1350,"children":1351},{},[1352],{"type":66,"value":1353},"        summary = asset_state_store.get(\"last_run_summary\") or {}\n",{"type":60,"tag":102,"props":1355,"children":1356},{"class":104,"line":569},[1357],{"type":60,"tag":102,"props":1358,"children":1359},{},[1360],{"type":66,"value":1361},"        print(f\"Processing {summary.get('rows_loaded')} rows up to {asset_state_store.get('watermark')}\")\n",{"type":60,"tag":102,"props":1363,"children":1364},{"class":104,"line":578},[1365],{"type":60,"tag":102,"props":1366,"children":1367},{"emptyLinePlaceholder":537},[1368],{"type":66,"value":540},{"type":60,"tag":102,"props":1370,"children":1371},{"class":104,"line":587},[1372],{"type":60,"tag":102,"props":1373,"children":1374},{},[1375],{"type":66,"value":1376},"    consume()\n",{"type":60,"tag":69,"props":1378,"children":1379},{},[1380],{"type":60,"tag":82,"props":1381,"children":1382},{},[1383],{"type":66,"value":767},{"type":60,"tag":267,"props":1385,"children":1386},{},[1387,1413,1433,1444,1449],{"type":60,"tag":271,"props":1388,"children":1389},{},[1390,1395,1397,1403,1405,1411],{"type":60,"tag":98,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":66,"value":217},{"type":66,"value":1396}," is injected by Airflow as a named kwarg — declare it as ",{"type":60,"tag":98,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":66,"value":1402},"def my_task(asset_state_store=None)",{"type":66,"value":1404},". Do NOT combine with ",{"type":60,"tag":98,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":66,"value":1410},"**context",{"type":66,"value":1412},"; Airflow injects it separately.",{"type":60,"tag":271,"props":1414,"children":1415},{},[1416,1417,1423,1425,1431],{"type":66,"value":845},{"type":60,"tag":98,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":66,"value":1422},"datetime.now(tz=timezone.utc).isoformat()",{"type":66,"value":1424}," for timestamps — never ",{"type":60,"tag":98,"props":1426,"children":1428},{"className":1427},[],[1429],{"type":66,"value":1430},"datetime.utcnow()",{"type":66,"value":1432}," (not timezone-aware).",{"type":60,"tag":271,"props":1434,"children":1435},{},[1436,1438,1443],{"type":66,"value":1437},"Same JSON-serializable value constraint as ",{"type":60,"tag":98,"props":1439,"children":1441},{"className":1440},[],[1442],{"type":66,"value":184},{"type":66,"value":882},{"type":60,"tag":271,"props":1445,"children":1446},{},[1447],{"type":66,"value":1448},"No per-key expiry — asset state store entries have no TTL (the asset outlives any single run).",{"type":60,"tag":271,"props":1450,"children":1451},{},[1452],{"type":66,"value":1453},"Readable by any DAG that declares the asset as an inlet or outlet.",{"type":60,"tag":69,"props":1455,"children":1456},{},[1457],{"type":60,"tag":82,"props":1458,"children":1459},{},[1460],{"type":66,"value":1461},"Mapped tasks — last writer wins:",{"type":60,"tag":69,"props":1463,"children":1464},{},[1465,1470],{"type":60,"tag":98,"props":1466,"children":1468},{"className":1467},[],[1469],{"type":66,"value":217},{"type":66,"value":1471}," is scoped to the asset, not the map index. If multiple mapped indices write the same key concurrently, the last write wins. Use distinct keys per index or ensure only one index writes to a given key.",{"type":60,"tag":90,"props":1473,"children":1475},{"className":507,"code":1474,"language":509,"meta":95,"style":95},"@task(outlets=[my_asset])\ndef load_partition(partition_id, asset_state_store=None):\n    asset_state_store = asset_state_store[my_asset]\n    # Distinct key per index — no race condition\n    asset_state_store.set(f\"offset_{partition_id}\", new_offset)\n",[1476],{"type":60,"tag":98,"props":1477,"children":1478},{"__ignoreMap":95},[1479,1487,1495,1503,1511],{"type":60,"tag":102,"props":1480,"children":1481},{"class":104,"line":105},[1482],{"type":60,"tag":102,"props":1483,"children":1484},{},[1485],{"type":66,"value":1486},"@task(outlets=[my_asset])\n",{"type":60,"tag":102,"props":1488,"children":1489},{"class":104,"line":524},[1490],{"type":60,"tag":102,"props":1491,"children":1492},{},[1493],{"type":66,"value":1494},"def load_partition(partition_id, asset_state_store=None):\n",{"type":60,"tag":102,"props":1496,"children":1497},{"class":104,"line":533},[1498],{"type":60,"tag":102,"props":1499,"children":1500},{},[1501],{"type":66,"value":1502},"    asset_state_store = asset_state_store[my_asset]\n",{"type":60,"tag":102,"props":1504,"children":1505},{"class":104,"line":543},[1506],{"type":60,"tag":102,"props":1507,"children":1508},{},[1509],{"type":66,"value":1510},"    # Distinct key per index — no race condition\n",{"type":60,"tag":102,"props":1512,"children":1513},{"class":104,"line":552},[1514],{"type":60,"tag":102,"props":1515,"children":1516},{},[1517],{"type":66,"value":1518},"    asset_state_store.set(f\"offset_{partition_id}\", new_offset)\n",{"type":60,"tag":69,"props":1520,"children":1521},{},[1522],{"type":60,"tag":82,"props":1523,"children":1524},{},[1525],{"type":66,"value":1007},{"type":60,"tag":90,"props":1527,"children":1529},{"className":507,"code":1528,"language":509,"meta":95,"style":95},"Variable.set(f\"watermark_{asset_name}\", new_offset)   # global, not scoped to asset\n",[1530],{"type":60,"tag":98,"props":1531,"children":1532},{"__ignoreMap":95},[1533],{"type":60,"tag":102,"props":1534,"children":1535},{"class":104,"line":105},[1536],{"type":60,"tag":102,"props":1537,"children":1538},{},[1539],{"type":66,"value":1528},{"type":60,"tag":69,"props":1541,"children":1542},{},[1543],{"type":60,"tag":82,"props":1544,"children":1545},{},[1546],{"type":66,"value":1062},{"type":60,"tag":90,"props":1548,"children":1550},{"className":507,"code":1549,"language":509,"meta":95,"style":95},"@task(inlets=[my_asset], outlets=[my_asset])\ndef load(asset_state_store=None):\n    asset_state_store = asset_state_store[my_asset]\n    asset_state_store.set(\"watermark\", new_offset)\n",[1551],{"type":60,"tag":98,"props":1552,"children":1553},{"__ignoreMap":95},[1554,1562,1570,1577],{"type":60,"tag":102,"props":1555,"children":1556},{"class":104,"line":105},[1557],{"type":60,"tag":102,"props":1558,"children":1559},{},[1560],{"type":66,"value":1561},"@task(inlets=[my_asset], outlets=[my_asset])\n",{"type":60,"tag":102,"props":1563,"children":1564},{"class":104,"line":524},[1565],{"type":60,"tag":102,"props":1566,"children":1567},{},[1568],{"type":66,"value":1569},"def load(asset_state_store=None):\n",{"type":60,"tag":102,"props":1571,"children":1572},{"class":104,"line":533},[1573],{"type":60,"tag":102,"props":1574,"children":1575},{},[1576],{"type":66,"value":1502},{"type":60,"tag":102,"props":1578,"children":1579},{"class":104,"line":543},[1580],{"type":60,"tag":102,"props":1581,"children":1582},{},[1583],{"type":66,"value":1584},"    asset_state_store.set(\"watermark\", new_offset)\n",{"type":60,"tag":130,"props":1586,"children":1587},{},[],{"type":60,"tag":134,"props":1589,"children":1591},{"id":1590},"section-5-resumablejobmixin-crash-safe-external-job-submission",[1592,1594,1599],{"type":66,"value":1593},"Section 5 — ",{"type":60,"tag":98,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":66,"value":257},{"type":66,"value":1600},": crash-safe external job submission",{"type":60,"tag":69,"props":1602,"children":1603},{},[1604],{"type":66,"value":1605},"Use when an operator submits a job to an external system (Spark, Databricks, dbt Cloud, AWS Batch, etc.) and then polls for completion. Without this mixin, a worker crash during polling means the next retry submits a duplicate job.",{"type":60,"tag":69,"props":1607,"children":1608},{},[1609],{"type":60,"tag":82,"props":1610,"children":1611},{},[1612,1614,1619],{"type":66,"value":1613},"When NOT to use ",{"type":60,"tag":98,"props":1615,"children":1617},{"className":1616},[],[1618],{"type":66,"value":257},{"type":66,"value":1620},":",{"type":60,"tag":141,"props":1622,"children":1623},{},[1624,1645],{"type":60,"tag":145,"props":1625,"children":1626},{},[1627],{"type":60,"tag":149,"props":1628,"children":1629},{},[1630,1635,1640],{"type":60,"tag":153,"props":1631,"children":1632},{},[1633],{"type":66,"value":1634},"Situation",{"type":60,"tag":153,"props":1636,"children":1637},{},[1638],{"type":66,"value":1639},"Use instead",{"type":60,"tag":153,"props":1641,"children":1642},{},[1643],{"type":66,"value":1644},"Why",{"type":60,"tag":164,"props":1646,"children":1647},{},[1648,1666,1696,1718],{"type":60,"tag":149,"props":1649,"children":1650},{},[1651,1656,1661],{"type":60,"tag":171,"props":1652,"children":1653},{},[1654],{"type":66,"value":1655},"A Triggerer is deployed and a deferrable operator exists (or can be written)",{"type":60,"tag":171,"props":1657,"children":1658},{},[1659],{"type":66,"value":1660},"Deferrable operator",{"type":60,"tag":171,"props":1662,"children":1663},{},[1664],{"type":66,"value":1665},"Frees the worker slot during polling; more resource-efficient",{"type":60,"tag":149,"props":1667,"children":1668},{},[1669,1674,1691],{"type":60,"tag":171,"props":1670,"children":1671},{},[1672],{"type":66,"value":1673},"The task fans out many concurrent I\u002FO operations within a single execution",{"type":60,"tag":171,"props":1675,"children":1676},{},[1677,1683,1685],{"type":60,"tag":98,"props":1678,"children":1680},{"className":1679},[],[1681],{"type":66,"value":1682},"async def",{"type":66,"value":1684}," task \u002F ",{"type":60,"tag":98,"props":1686,"children":1688},{"className":1687},[],[1689],{"type":66,"value":1690},"BaseAsyncOperator",{"type":60,"tag":171,"props":1692,"children":1693},{},[1694],{"type":66,"value":1695},"Async is for high-throughput I\u002FO, not crash recovery",{"type":60,"tag":149,"props":1697,"children":1698},{},[1699,1708,1713],{"type":60,"tag":171,"props":1700,"children":1701},{},[1702],{"type":60,"tag":98,"props":1703,"children":1705},{"className":1704},[],[1706],{"type":66,"value":1707},"retries=0",{"type":60,"tag":171,"props":1709,"children":1710},{},[1711],{"type":66,"value":1712},"—",{"type":60,"tag":171,"props":1714,"children":1715},{},[1716],{"type":66,"value":1717},"Crash recovery has nothing to reconnect to",{"type":60,"tag":149,"props":1719,"children":1720},{},[1721,1741,1746],{"type":60,"tag":171,"props":1722,"children":1723},{},[1724,1726,1732,1734,1739],{"type":66,"value":1725},"The external system has no trackable job ID (",{"type":60,"tag":98,"props":1727,"children":1729},{"className":1728},[],[1730],{"type":66,"value":1731},"submit_job",{"type":66,"value":1733}," returns ",{"type":60,"tag":98,"props":1735,"children":1737},{"className":1736},[],[1738],{"type":66,"value":825},{"type":66,"value":1740},")",{"type":60,"tag":171,"props":1742,"children":1743},{},[1744],{"type":66,"value":1745},"Plain operator",{"type":60,"tag":171,"props":1747,"children":1748},{},[1749],{"type":66,"value":1750},"The mixin's crash-safety guarantee is silently disabled; adds no value",{"type":60,"tag":69,"props":1752,"children":1753},{},[1754,1759],{"type":60,"tag":98,"props":1755,"children":1757},{"className":1756},[],[1758],{"type":66,"value":257},{"type":66,"value":1760}," holds the worker slot for the full polling duration — the same as a standard synchronous operator. The benefit is crash safety and job continuity, not resource efficiency.",{"type":60,"tag":69,"props":1762,"children":1763},{},[1764],{"type":60,"tag":82,"props":1765,"children":1766},{},[1767],{"type":66,"value":1768},"Opting out of crash recovery:",{"type":60,"tag":69,"props":1770,"children":1771},{},[1772,1774,1780,1782,1788,1790,1795],{"type":66,"value":1773},"The mixin ships with ",{"type":60,"tag":98,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":66,"value":1779},"durable=True",{"type":66,"value":1781}," by default. Set ",{"type":60,"tag":98,"props":1783,"children":1785},{"className":1784},[],[1786],{"type":66,"value":1787},"durable=False",{"type":66,"value":1789}," to skip all ",{"type":60,"tag":98,"props":1791,"children":1793},{"className":1792},[],[1794],{"type":66,"value":184},{"type":66,"value":1796}," interaction and run a plain submit\u002Fpoll\u002Fresult cycle — useful in test environments or when the external system has its own dedup:",{"type":60,"tag":90,"props":1798,"children":1800},{"className":507,"code":1799,"language":509,"meta":95,"style":95},"MyBatchOperator(task_id=\"job\", durable=False)\n\n# Or via default_args to disable for all tasks in a DAG:\nwith DAG(\"my_dag\", default_args={\"durable\": False}):\n    ...\n",[1801],{"type":60,"tag":98,"props":1802,"children":1803},{"__ignoreMap":95},[1804,1812,1819,1827,1835],{"type":60,"tag":102,"props":1805,"children":1806},{"class":104,"line":105},[1807],{"type":60,"tag":102,"props":1808,"children":1809},{},[1810],{"type":66,"value":1811},"MyBatchOperator(task_id=\"job\", durable=False)\n",{"type":60,"tag":102,"props":1813,"children":1814},{"class":104,"line":524},[1815],{"type":60,"tag":102,"props":1816,"children":1817},{"emptyLinePlaceholder":537},[1818],{"type":66,"value":540},{"type":60,"tag":102,"props":1820,"children":1821},{"class":104,"line":533},[1822],{"type":60,"tag":102,"props":1823,"children":1824},{},[1825],{"type":66,"value":1826},"# Or via default_args to disable for all tasks in a DAG:\n",{"type":60,"tag":102,"props":1828,"children":1829},{"class":104,"line":543},[1830],{"type":60,"tag":102,"props":1831,"children":1832},{},[1833],{"type":66,"value":1834},"with DAG(\"my_dag\", default_args={\"durable\": False}):\n",{"type":60,"tag":102,"props":1836,"children":1837},{"class":104,"line":552},[1838],{"type":60,"tag":102,"props":1839,"children":1840},{},[1841],{"type":66,"value":1842},"    ...\n",{"type":60,"tag":1844,"props":1845,"children":1847},"h3",{"id":1846},"implementing-the-mixin",[1848],{"type":66,"value":1849},"Implementing the mixin",{"type":60,"tag":90,"props":1851,"children":1853},{"className":507,"code":1852,"language":509,"meta":95,"style":95},"from airflow.sdk import BaseOperator, ResumableJobMixin\nfrom pydantic import JsonValue\n\n\nclass MyBatchOperator(BaseOperator, ResumableJobMixin):\n\n    external_id_key = \"batch_job_id\"   # key used in task_state_store; set once, never rename\n\n    def execute(self, context):\n        return self.execute_resumable(context)  # never call self.execute() — call this\n\n    def submit_job(self, context) -> JsonValue:\n        # Submit and return the job identifier. This value is persisted to task_state_store\n        # before polling starts. Return None only if the system has no trackable ID\n        # (in that case crash-safety is disabled and the job resubmits on every retry).\n        return self.hook.submit_batch(...)\n\n    def get_job_status(self, external_id: JsonValue, context) -> str:\n        # Query the external system. Return a raw status string.\n        return self.hook.get_status(external_id)\n\n    def is_job_active(self, status: str) -> bool:\n        # Return True if the job is still running and should be reconnected to.\n        return status in (\"RUNNING\", \"PENDING\", \"QUEUED\")\n\n    def is_job_succeeded(self, status: str) -> bool:\n        return status == \"SUCCEEDED\"\n\n    def poll_until_complete(self, external_id: JsonValue, context) -> None:\n        # Block until the job reaches a terminal state. Raise on failure.\n        self.hook.wait(external_id)\n\n    def get_job_result(self, external_id: JsonValue, context):\n        # Return the job result after success. Return None if not applicable.\n        return None\n",[1854],{"type":60,"tag":98,"props":1855,"children":1856},{"__ignoreMap":95},[1857,1865,1873,1880,1887,1895,1902,1910,1917,1925,1933,1940,1948,1956,1964,1972,1980,1987,1995,2003,2012,2020,2029,2038,2047,2055,2064,2073,2081,2090,2099,2108,2116,2125,2134],{"type":60,"tag":102,"props":1858,"children":1859},{"class":104,"line":105},[1860],{"type":60,"tag":102,"props":1861,"children":1862},{},[1863],{"type":66,"value":1864},"from airflow.sdk import BaseOperator, ResumableJobMixin\n",{"type":60,"tag":102,"props":1866,"children":1867},{"class":104,"line":524},[1868],{"type":60,"tag":102,"props":1869,"children":1870},{},[1871],{"type":66,"value":1872},"from pydantic import JsonValue\n",{"type":60,"tag":102,"props":1874,"children":1875},{"class":104,"line":533},[1876],{"type":60,"tag":102,"props":1877,"children":1878},{"emptyLinePlaceholder":537},[1879],{"type":66,"value":540},{"type":60,"tag":102,"props":1881,"children":1882},{"class":104,"line":543},[1883],{"type":60,"tag":102,"props":1884,"children":1885},{"emptyLinePlaceholder":537},[1886],{"type":66,"value":540},{"type":60,"tag":102,"props":1888,"children":1889},{"class":104,"line":552},[1890],{"type":60,"tag":102,"props":1891,"children":1892},{},[1893],{"type":66,"value":1894},"class MyBatchOperator(BaseOperator, ResumableJobMixin):\n",{"type":60,"tag":102,"props":1896,"children":1897},{"class":104,"line":561},[1898],{"type":60,"tag":102,"props":1899,"children":1900},{"emptyLinePlaceholder":537},[1901],{"type":66,"value":540},{"type":60,"tag":102,"props":1903,"children":1904},{"class":104,"line":569},[1905],{"type":60,"tag":102,"props":1906,"children":1907},{},[1908],{"type":66,"value":1909},"    external_id_key = \"batch_job_id\"   # key used in task_state_store; set once, never rename\n",{"type":60,"tag":102,"props":1911,"children":1912},{"class":104,"line":578},[1913],{"type":60,"tag":102,"props":1914,"children":1915},{"emptyLinePlaceholder":537},[1916],{"type":66,"value":540},{"type":60,"tag":102,"props":1918,"children":1919},{"class":104,"line":587},[1920],{"type":60,"tag":102,"props":1921,"children":1922},{},[1923],{"type":66,"value":1924},"    def execute(self, context):\n",{"type":60,"tag":102,"props":1926,"children":1927},{"class":104,"line":596},[1928],{"type":60,"tag":102,"props":1929,"children":1930},{},[1931],{"type":66,"value":1932},"        return self.execute_resumable(context)  # never call self.execute() — call this\n",{"type":60,"tag":102,"props":1934,"children":1935},{"class":104,"line":605},[1936],{"type":60,"tag":102,"props":1937,"children":1938},{"emptyLinePlaceholder":537},[1939],{"type":66,"value":540},{"type":60,"tag":102,"props":1941,"children":1942},{"class":104,"line":614},[1943],{"type":60,"tag":102,"props":1944,"children":1945},{},[1946],{"type":66,"value":1947},"    def submit_job(self, context) -> JsonValue:\n",{"type":60,"tag":102,"props":1949,"children":1950},{"class":104,"line":623},[1951],{"type":60,"tag":102,"props":1952,"children":1953},{},[1954],{"type":66,"value":1955},"        # Submit and return the job identifier. This value is persisted to task_state_store\n",{"type":60,"tag":102,"props":1957,"children":1958},{"class":104,"line":632},[1959],{"type":60,"tag":102,"props":1960,"children":1961},{},[1962],{"type":66,"value":1963},"        # before polling starts. Return None only if the system has no trackable ID\n",{"type":60,"tag":102,"props":1965,"children":1966},{"class":104,"line":641},[1967],{"type":60,"tag":102,"props":1968,"children":1969},{},[1970],{"type":66,"value":1971},"        # (in that case crash-safety is disabled and the job resubmits on every retry).\n",{"type":60,"tag":102,"props":1973,"children":1974},{"class":104,"line":650},[1975],{"type":60,"tag":102,"props":1976,"children":1977},{},[1978],{"type":66,"value":1979},"        return self.hook.submit_batch(...)\n",{"type":60,"tag":102,"props":1981,"children":1982},{"class":104,"line":658},[1983],{"type":60,"tag":102,"props":1984,"children":1985},{"emptyLinePlaceholder":537},[1986],{"type":66,"value":540},{"type":60,"tag":102,"props":1988,"children":1989},{"class":104,"line":667},[1990],{"type":60,"tag":102,"props":1991,"children":1992},{},[1993],{"type":66,"value":1994},"    def get_job_status(self, external_id: JsonValue, context) -> str:\n",{"type":60,"tag":102,"props":1996,"children":1997},{"class":104,"line":675},[1998],{"type":60,"tag":102,"props":1999,"children":2000},{},[2001],{"type":66,"value":2002},"        # Query the external system. Return a raw status string.\n",{"type":60,"tag":102,"props":2004,"children":2006},{"class":104,"line":2005},20,[2007],{"type":60,"tag":102,"props":2008,"children":2009},{},[2010],{"type":66,"value":2011},"        return self.hook.get_status(external_id)\n",{"type":60,"tag":102,"props":2013,"children":2015},{"class":104,"line":2014},21,[2016],{"type":60,"tag":102,"props":2017,"children":2018},{"emptyLinePlaceholder":537},[2019],{"type":66,"value":540},{"type":60,"tag":102,"props":2021,"children":2023},{"class":104,"line":2022},22,[2024],{"type":60,"tag":102,"props":2025,"children":2026},{},[2027],{"type":66,"value":2028},"    def is_job_active(self, status: str) -> bool:\n",{"type":60,"tag":102,"props":2030,"children":2032},{"class":104,"line":2031},23,[2033],{"type":60,"tag":102,"props":2034,"children":2035},{},[2036],{"type":66,"value":2037},"        # Return True if the job is still running and should be reconnected to.\n",{"type":60,"tag":102,"props":2039,"children":2041},{"class":104,"line":2040},24,[2042],{"type":60,"tag":102,"props":2043,"children":2044},{},[2045],{"type":66,"value":2046},"        return status in (\"RUNNING\", \"PENDING\", \"QUEUED\")\n",{"type":60,"tag":102,"props":2048,"children":2050},{"class":104,"line":2049},25,[2051],{"type":60,"tag":102,"props":2052,"children":2053},{"emptyLinePlaceholder":537},[2054],{"type":66,"value":540},{"type":60,"tag":102,"props":2056,"children":2058},{"class":104,"line":2057},26,[2059],{"type":60,"tag":102,"props":2060,"children":2061},{},[2062],{"type":66,"value":2063},"    def is_job_succeeded(self, status: str) -> bool:\n",{"type":60,"tag":102,"props":2065,"children":2067},{"class":104,"line":2066},27,[2068],{"type":60,"tag":102,"props":2069,"children":2070},{},[2071],{"type":66,"value":2072},"        return status == \"SUCCEEDED\"\n",{"type":60,"tag":102,"props":2074,"children":2076},{"class":104,"line":2075},28,[2077],{"type":60,"tag":102,"props":2078,"children":2079},{"emptyLinePlaceholder":537},[2080],{"type":66,"value":540},{"type":60,"tag":102,"props":2082,"children":2084},{"class":104,"line":2083},29,[2085],{"type":60,"tag":102,"props":2086,"children":2087},{},[2088],{"type":66,"value":2089},"    def poll_until_complete(self, external_id: JsonValue, context) -> None:\n",{"type":60,"tag":102,"props":2091,"children":2093},{"class":104,"line":2092},30,[2094],{"type":60,"tag":102,"props":2095,"children":2096},{},[2097],{"type":66,"value":2098},"        # Block until the job reaches a terminal state. Raise on failure.\n",{"type":60,"tag":102,"props":2100,"children":2102},{"class":104,"line":2101},31,[2103],{"type":60,"tag":102,"props":2104,"children":2105},{},[2106],{"type":66,"value":2107},"        self.hook.wait(external_id)\n",{"type":60,"tag":102,"props":2109,"children":2111},{"class":104,"line":2110},32,[2112],{"type":60,"tag":102,"props":2113,"children":2114},{"emptyLinePlaceholder":537},[2115],{"type":66,"value":540},{"type":60,"tag":102,"props":2117,"children":2119},{"class":104,"line":2118},33,[2120],{"type":60,"tag":102,"props":2121,"children":2122},{},[2123],{"type":66,"value":2124},"    def get_job_result(self, external_id: JsonValue, context):\n",{"type":60,"tag":102,"props":2126,"children":2128},{"class":104,"line":2127},34,[2129],{"type":60,"tag":102,"props":2130,"children":2131},{},[2132],{"type":66,"value":2133},"        # Return the job result after success. Return None if not applicable.\n",{"type":60,"tag":102,"props":2135,"children":2137},{"class":104,"line":2136},35,[2138],{"type":60,"tag":102,"props":2139,"children":2140},{},[2141],{"type":66,"value":2142},"        return None\n",{"type":60,"tag":1844,"props":2144,"children":2146},{"id":2145},"what-happens-on-retry",[2147],{"type":66,"value":2148},"What happens on retry",{"type":60,"tag":141,"props":2150,"children":2151},{},[2152,2168],{"type":60,"tag":145,"props":2153,"children":2154},{},[2155],{"type":60,"tag":149,"props":2156,"children":2157},{},[2158,2163],{"type":60,"tag":153,"props":2159,"children":2160},{},[2161],{"type":66,"value":2162},"Job state on retry",{"type":60,"tag":153,"props":2164,"children":2165},{},[2166],{"type":66,"value":2167},"Mixin behaviour",{"type":60,"tag":164,"props":2169,"children":2170},{},[2171,2192,2213],{"type":60,"tag":149,"props":2172,"children":2173},{},[2174,2179],{"type":60,"tag":171,"props":2175,"children":2176},{},[2177],{"type":66,"value":2178},"Still running",{"type":60,"tag":171,"props":2180,"children":2181},{},[2182,2184,2190],{"type":66,"value":2183},"Reconnects — calls ",{"type":60,"tag":98,"props":2185,"children":2187},{"className":2186},[],[2188],{"type":66,"value":2189},"poll_until_complete",{"type":66,"value":2191}," without resubmitting",{"type":60,"tag":149,"props":2193,"children":2194},{},[2195,2200],{"type":60,"tag":171,"props":2196,"children":2197},{},[2198],{"type":66,"value":2199},"Already succeeded",{"type":60,"tag":171,"props":2201,"children":2202},{},[2203,2205,2211],{"type":66,"value":2204},"Returns ",{"type":60,"tag":98,"props":2206,"children":2208},{"className":2207},[],[2209],{"type":66,"value":2210},"get_job_result",{"type":66,"value":2212}," immediately",{"type":60,"tag":149,"props":2214,"children":2215},{},[2216,2221],{"type":60,"tag":171,"props":2217,"children":2218},{},[2219],{"type":66,"value":2220},"Failed \u002F unknown",{"type":60,"tag":171,"props":2222,"children":2223},{},[2224],{"type":66,"value":2225},"Submits a fresh job",{"type":60,"tag":1844,"props":2227,"children":2229},{"id":2228},"external_id_key-warning",[2230,2236],{"type":60,"tag":98,"props":2231,"children":2233},{"className":2232},[],[2234],{"type":66,"value":2235},"external_id_key",{"type":66,"value":2237}," warning",{"type":60,"tag":75,"props":2239,"children":2240},{},[2241],{"type":60,"tag":69,"props":2242,"children":2243},{},[2244,2256,2258,2263],{"type":60,"tag":82,"props":2245,"children":2246},{},[2247,2249,2254],{"type":66,"value":2248},"Never rename ",{"type":60,"tag":98,"props":2250,"children":2252},{"className":2251},[],[2253],{"type":66,"value":2235},{"type":66,"value":2255}," on an operator that is already deployed with in-flight task instances.",{"type":66,"value":2257}," The old key is stored in ",{"type":60,"tag":98,"props":2259,"children":2261},{"className":2260},[],[2262],{"type":66,"value":184},{"type":66,"value":2264}," under the previous name. A rename makes the mixin treat every active retry as a fresh submission, defeating the crash-safety guarantee.",{"type":60,"tag":1844,"props":2266,"children":2268},{"id":2267},"before-anti-pattern",[2269],{"type":66,"value":1007},{"type":60,"tag":90,"props":2271,"children":2273},{"className":507,"code":2272,"language":509,"meta":95,"style":95},"def execute(self, context):\n    job_id = Variable.get(\"spark_job_id\", default_var=None)\n    if job_id and self._is_running(job_id):\n        self._wait(job_id)\n    else:\n        job_id = self.hook.submit(...)\n        Variable.set(\"spark_job_id\", job_id)   # global, race-prone\n        self._wait(job_id)\n",[2274],{"type":60,"tag":98,"props":2275,"children":2276},{"__ignoreMap":95},[2277,2285,2293,2301,2309,2317,2325,2333],{"type":60,"tag":102,"props":2278,"children":2279},{"class":104,"line":105},[2280],{"type":60,"tag":102,"props":2281,"children":2282},{},[2283],{"type":66,"value":2284},"def execute(self, context):\n",{"type":60,"tag":102,"props":2286,"children":2287},{"class":104,"line":524},[2288],{"type":60,"tag":102,"props":2289,"children":2290},{},[2291],{"type":66,"value":2292},"    job_id = Variable.get(\"spark_job_id\", default_var=None)\n",{"type":60,"tag":102,"props":2294,"children":2295},{"class":104,"line":533},[2296],{"type":60,"tag":102,"props":2297,"children":2298},{},[2299],{"type":66,"value":2300},"    if job_id and self._is_running(job_id):\n",{"type":60,"tag":102,"props":2302,"children":2303},{"class":104,"line":543},[2304],{"type":60,"tag":102,"props":2305,"children":2306},{},[2307],{"type":66,"value":2308},"        self._wait(job_id)\n",{"type":60,"tag":102,"props":2310,"children":2311},{"class":104,"line":552},[2312],{"type":60,"tag":102,"props":2313,"children":2314},{},[2315],{"type":66,"value":2316},"    else:\n",{"type":60,"tag":102,"props":2318,"children":2319},{"class":104,"line":561},[2320],{"type":60,"tag":102,"props":2321,"children":2322},{},[2323],{"type":66,"value":2324},"        job_id = self.hook.submit(...)\n",{"type":60,"tag":102,"props":2326,"children":2327},{"class":104,"line":569},[2328],{"type":60,"tag":102,"props":2329,"children":2330},{},[2331],{"type":66,"value":2332},"        Variable.set(\"spark_job_id\", job_id)   # global, race-prone\n",{"type":60,"tag":102,"props":2334,"children":2335},{"class":104,"line":578},[2336],{"type":60,"tag":102,"props":2337,"children":2338},{},[2339],{"type":66,"value":2308},{"type":60,"tag":69,"props":2341,"children":2342},{},[2343],{"type":60,"tag":82,"props":2344,"children":2345},{},[2346],{"type":66,"value":1062},{"type":60,"tag":90,"props":2348,"children":2350},{"className":507,"code":2349,"language":509,"meta":95,"style":95},"class MySparkOperator(BaseOperator, ResumableJobMixin):\n    external_id_key = \"spark_job_id\"\n    def execute(self, context): return self.execute_resumable(context)\n    def submit_job(self, context): return self.hook.submit(...)\n    # ... implement the 5 other methods ...\n",[2351],{"type":60,"tag":98,"props":2352,"children":2353},{"__ignoreMap":95},[2354,2362,2370,2378,2386],{"type":60,"tag":102,"props":2355,"children":2356},{"class":104,"line":105},[2357],{"type":60,"tag":102,"props":2358,"children":2359},{},[2360],{"type":66,"value":2361},"class MySparkOperator(BaseOperator, ResumableJobMixin):\n",{"type":60,"tag":102,"props":2363,"children":2364},{"class":104,"line":524},[2365],{"type":60,"tag":102,"props":2366,"children":2367},{},[2368],{"type":66,"value":2369},"    external_id_key = \"spark_job_id\"\n",{"type":60,"tag":102,"props":2371,"children":2372},{"class":104,"line":533},[2373],{"type":60,"tag":102,"props":2374,"children":2375},{},[2376],{"type":66,"value":2377},"    def execute(self, context): return self.execute_resumable(context)\n",{"type":60,"tag":102,"props":2379,"children":2380},{"class":104,"line":543},[2381],{"type":60,"tag":102,"props":2382,"children":2383},{},[2384],{"type":66,"value":2385},"    def submit_job(self, context): return self.hook.submit(...)\n",{"type":60,"tag":102,"props":2387,"children":2388},{"class":104,"line":552},[2389],{"type":60,"tag":102,"props":2390,"children":2391},{},[2392],{"type":66,"value":2393},"    # ... implement the 5 other methods ...\n",{"type":60,"tag":130,"props":2395,"children":2396},{},[],{"type":60,"tag":134,"props":2398,"children":2400},{"id":2399},"section-6-configuration-reference",[2401],{"type":66,"value":2402},"Section 6 — Configuration reference",{"type":60,"tag":90,"props":2404,"children":2408},{"className":2405,"code":2406,"language":2407,"meta":95,"style":95},"language-ini shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[state_store]\n# Full dotted path to the storage backend. Default writes to the Airflow metadata DB.\nbackend = airflow.state.metastore.MetastoreStateStoreBackend\n\n# Days to retain task state store entries after their last update. 0 = disable time-based cleanup.\n# Does NOT affect asset_state_store rows — asset state store has no TTL.\ndefault_retention_days = 30\n\n# Rows deleted per batch during cleanup. 0 = no batching (single unbounded delete).\n# Tune on large deployments to reduce lock contention.\nstate_cleanup_batch_size = 0\n\n# Auto-delete all task state store keys when a task succeeds. Default: False.\n# Does NOT affect asset_state_store — asset state store persists across runs and must be cleared explicitly.\nclear_on_success = False\n","ini",[2409],{"type":60,"tag":98,"props":2410,"children":2411},{"__ignoreMap":95},[2412,2420,2428,2436,2443,2451,2459,2467,2474,2482,2490,2498,2505,2513,2521],{"type":60,"tag":102,"props":2413,"children":2414},{"class":104,"line":105},[2415],{"type":60,"tag":102,"props":2416,"children":2417},{},[2418],{"type":66,"value":2419},"[state_store]\n",{"type":60,"tag":102,"props":2421,"children":2422},{"class":104,"line":524},[2423],{"type":60,"tag":102,"props":2424,"children":2425},{},[2426],{"type":66,"value":2427},"# Full dotted path to the storage backend. Default writes to the Airflow metadata DB.\n",{"type":60,"tag":102,"props":2429,"children":2430},{"class":104,"line":533},[2431],{"type":60,"tag":102,"props":2432,"children":2433},{},[2434],{"type":66,"value":2435},"backend = airflow.state.metastore.MetastoreStateStoreBackend\n",{"type":60,"tag":102,"props":2437,"children":2438},{"class":104,"line":543},[2439],{"type":60,"tag":102,"props":2440,"children":2441},{"emptyLinePlaceholder":537},[2442],{"type":66,"value":540},{"type":60,"tag":102,"props":2444,"children":2445},{"class":104,"line":552},[2446],{"type":60,"tag":102,"props":2447,"children":2448},{},[2449],{"type":66,"value":2450},"# Days to retain task state store entries after their last update. 0 = disable time-based cleanup.\n",{"type":60,"tag":102,"props":2452,"children":2453},{"class":104,"line":561},[2454],{"type":60,"tag":102,"props":2455,"children":2456},{},[2457],{"type":66,"value":2458},"# Does NOT affect asset_state_store rows — asset state store has no TTL.\n",{"type":60,"tag":102,"props":2460,"children":2461},{"class":104,"line":569},[2462],{"type":60,"tag":102,"props":2463,"children":2464},{},[2465],{"type":66,"value":2466},"default_retention_days = 30\n",{"type":60,"tag":102,"props":2468,"children":2469},{"class":104,"line":578},[2470],{"type":60,"tag":102,"props":2471,"children":2472},{"emptyLinePlaceholder":537},[2473],{"type":66,"value":540},{"type":60,"tag":102,"props":2475,"children":2476},{"class":104,"line":587},[2477],{"type":60,"tag":102,"props":2478,"children":2479},{},[2480],{"type":66,"value":2481},"# Rows deleted per batch during cleanup. 0 = no batching (single unbounded delete).\n",{"type":60,"tag":102,"props":2483,"children":2484},{"class":104,"line":596},[2485],{"type":60,"tag":102,"props":2486,"children":2487},{},[2488],{"type":66,"value":2489},"# Tune on large deployments to reduce lock contention.\n",{"type":60,"tag":102,"props":2491,"children":2492},{"class":104,"line":605},[2493],{"type":60,"tag":102,"props":2494,"children":2495},{},[2496],{"type":66,"value":2497},"state_cleanup_batch_size = 0\n",{"type":60,"tag":102,"props":2499,"children":2500},{"class":104,"line":614},[2501],{"type":60,"tag":102,"props":2502,"children":2503},{"emptyLinePlaceholder":537},[2504],{"type":66,"value":540},{"type":60,"tag":102,"props":2506,"children":2507},{"class":104,"line":623},[2508],{"type":60,"tag":102,"props":2509,"children":2510},{},[2511],{"type":66,"value":2512},"# Auto-delete all task state store keys when a task succeeds. Default: False.\n",{"type":60,"tag":102,"props":2514,"children":2515},{"class":104,"line":632},[2516],{"type":60,"tag":102,"props":2517,"children":2518},{},[2519],{"type":66,"value":2520},"# Does NOT affect asset_state_store — asset state store persists across runs and must be cleared explicitly.\n",{"type":60,"tag":102,"props":2522,"children":2523},{"class":104,"line":641},[2524],{"type":60,"tag":102,"props":2525,"children":2526},{},[2527],{"type":66,"value":2528},"clear_on_success = False\n",{"type":60,"tag":69,"props":2530,"children":2531},{},[2532,2537,2539,2545],{"type":60,"tag":82,"props":2533,"children":2534},{},[2535],{"type":66,"value":2536},"Worker-side backend",{"type":66,"value":2538}," (optional, ",{"type":60,"tag":98,"props":2540,"children":2542},{"className":2541},[],[2543],{"type":66,"value":2544},"[workers]",{"type":66,"value":2546}," section) — routes task state store writes through a local backend before they reach the API server. Useful when large payloads or credentialed storage should stay on the worker:",{"type":60,"tag":90,"props":2548,"children":2550},{"className":2405,"code":2549,"language":2407,"meta":95,"style":95},"[workers]\nstate_store_backend = mypackage.store.WorkerSideBackend\n",[2551],{"type":60,"tag":98,"props":2552,"children":2553},{"__ignoreMap":95},[2554,2562],{"type":60,"tag":102,"props":2555,"children":2556},{"class":104,"line":105},[2557],{"type":60,"tag":102,"props":2558,"children":2559},{},[2560],{"type":66,"value":2561},"[workers]\n",{"type":60,"tag":102,"props":2563,"children":2564},{"class":104,"line":524},[2565],{"type":60,"tag":102,"props":2566,"children":2567},{},[2568],{"type":66,"value":2569},"state_store_backend = mypackage.store.WorkerSideBackend\n",{"type":60,"tag":130,"props":2571,"children":2572},{},[],{"type":60,"tag":134,"props":2574,"children":2576},{"id":2575},"section-7-safety-checklist",[2577],{"type":66,"value":2578},"Section 7 — Safety checklist",{"type":60,"tag":267,"props":2580,"children":2583},{"className":2581},[2582],"contains-task-list",[2584,2603,2657,2673,2689,2705,2727,2757,2778],{"type":60,"tag":271,"props":2585,"children":2588},{"className":2586},[2587],"task-list-item",[2589,2594,2596,2602],{"type":60,"tag":2590,"props":2591,"children":2593},"input",{"disabled":537,"type":2592},"checkbox",[],{"type":66,"value":2595}," Airflow version ≥ 3.3 (",{"type":60,"tag":98,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":66,"value":2601},"af config version",{"type":66,"value":1740},{"type":60,"tag":271,"props":2604,"children":2606},{"className":2605},[2587],[2607,2610,2612,2617,2618,2623,2624,2629,2630,2635,2636,2641,2642,2647,2649,2655],{"type":60,"tag":2590,"props":2608,"children":2609},{"disabled":537,"type":2592},[],{"type":66,"value":2611}," Values are JSON-serializable (",{"type":60,"tag":98,"props":2613,"children":2615},{"className":2614},[],[2616],{"type":66,"value":781},{"type":66,"value":783},{"type":60,"tag":98,"props":2619,"children":2621},{"className":2620},[],[2622],{"type":66,"value":789},{"type":66,"value":783},{"type":60,"tag":98,"props":2625,"children":2627},{"className":2626},[],[2628],{"type":66,"value":796},{"type":66,"value":783},{"type":60,"tag":98,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":66,"value":803},{"type":66,"value":783},{"type":60,"tag":98,"props":2637,"children":2639},{"className":2638},[],[2640],{"type":66,"value":810},{"type":66,"value":783},{"type":60,"tag":98,"props":2643,"children":2645},{"className":2644},[],[2646],{"type":66,"value":817},{"type":66,"value":2648}," — no ",{"type":60,"tag":98,"props":2650,"children":2652},{"className":2651},[],[2653],{"type":66,"value":2654},"datetime",{"type":66,"value":2656},", no custom objects)",{"type":60,"tag":271,"props":2658,"children":2660},{"className":2659},[2587],[2661,2664,2666,2671],{"type":60,"tag":2590,"props":2662,"children":2663},{"disabled":537,"type":2592},[],{"type":66,"value":2665}," ",{"type":60,"tag":98,"props":2667,"children":2669},{"className":2668},[],[2670],{"type":66,"value":184},{"type":66,"value":2672}," keys are short, descriptive strings (avoid dots and slashes)",{"type":60,"tag":271,"props":2674,"children":2676},{"className":2675},[2587],[2677,2680,2682,2687],{"type":60,"tag":2590,"props":2678,"children":2679},{"disabled":537,"type":2592},[],{"type":66,"value":2681}," Mapped tasks writing to ",{"type":60,"tag":98,"props":2683,"children":2685},{"className":2684},[],[2686],{"type":66,"value":217},{"type":66,"value":2688},": use distinct keys per index or accept last-writer-wins semantics",{"type":60,"tag":271,"props":2690,"children":2692},{"className":2691},[2587],[2693,2696,2698,2703],{"type":60,"tag":2590,"props":2694,"children":2695},{"disabled":537,"type":2592},[],{"type":66,"value":2697}," Mapped tasks: fleet-wide state clear uses CLI\u002Fcore API from a downstream task, not ",{"type":60,"tag":98,"props":2699,"children":2701},{"className":2700},[],[2702],{"type":66,"value":997},{"type":66,"value":2704}," inside the task body",{"type":60,"tag":271,"props":2706,"children":2708},{"className":2707},[2587],[2709,2712,2713,2718,2720,2725],{"type":60,"tag":2590,"props":2710,"children":2711},{"disabled":537,"type":2592},[],{"type":66,"value":2665},{"type":60,"tag":98,"props":2714,"children":2716},{"className":2715},[],[2717],{"type":66,"value":257},{"type":66,"value":2719},": ",{"type":60,"tag":98,"props":2721,"children":2723},{"className":2722},[],[2724],{"type":66,"value":2235},{"type":66,"value":2726}," is set and will not be renamed after deployment",{"type":60,"tag":271,"props":2728,"children":2730},{"className":2729},[2587],[2731,2734,2735,2740,2741,2747,2749,2755],{"type":60,"tag":2590,"props":2732,"children":2733},{"disabled":537,"type":2592},[],{"type":66,"value":2665},{"type":60,"tag":98,"props":2736,"children":2738},{"className":2737},[],[2739],{"type":66,"value":257},{"type":66,"value":2719},{"type":60,"tag":98,"props":2742,"children":2744},{"className":2743},[],[2745],{"type":66,"value":2746},"execute()",{"type":66,"value":2748}," calls ",{"type":60,"tag":98,"props":2750,"children":2752},{"className":2751},[],[2753],{"type":66,"value":2754},"self.execute_resumable(context)",{"type":66,"value":2756},", not custom logic",{"type":60,"tag":271,"props":2758,"children":2760},{"className":2759},[2587],[2761,2764,2765,2770,2771,2776],{"type":60,"tag":2590,"props":2762,"children":2763},{"disabled":537,"type":2592},[],{"type":66,"value":2665},{"type":60,"tag":98,"props":2766,"children":2768},{"className":2767},[],[2769],{"type":66,"value":257},{"type":66,"value":2719},{"type":60,"tag":98,"props":2772,"children":2774},{"className":2773},[],[2775],{"type":66,"value":1787},{"type":66,"value":2777}," is intentional if crash recovery is disabled",{"type":60,"tag":271,"props":2779,"children":2781},{"className":2780},[2587],[2782,2785,2787,2793,2795,2800,2801],{"type":60,"tag":2590,"props":2783,"children":2784},{"disabled":537,"type":2592},[],{"type":66,"value":2786}," Large payloads (> configured ",{"type":60,"tag":98,"props":2788,"children":2790},{"className":2789},[],[2791],{"type":66,"value":2792},"max_value_storage_bytes",{"type":66,"value":2794},") use a custom ",{"type":60,"tag":98,"props":2796,"children":2798},{"className":2797},[],[2799],{"type":66,"value":872},{"type":66,"value":874},{"type":60,"tag":98,"props":2802,"children":2804},{"className":2803},[],[2805],{"type":66,"value":880},{"type":60,"tag":130,"props":2807,"children":2808},{},[],{"type":60,"tag":134,"props":2810,"children":2812},{"id":2811},"related-skills",[2813],{"type":66,"value":2814},"Related skills",{"type":60,"tag":267,"props":2816,"children":2817},{},[2818,2828,2838],{"type":60,"tag":271,"props":2819,"children":2820},{},[2821,2826],{"type":60,"tag":82,"props":2822,"children":2823},{},[2824],{"type":66,"value":2825},"authoring-dags",{"type":66,"value":2827}," — general DAG writing patterns and conventions.",{"type":60,"tag":271,"props":2829,"children":2830},{},[2831,2836],{"type":60,"tag":82,"props":2832,"children":2833},{},[2834],{"type":66,"value":2835},"airflow-hitl",{"type":66,"value":2837}," — pausing a DAG for human approval (Airflow 3.1+).",{"type":60,"tag":271,"props":2839,"children":2840},{},[2841,2845,2846,2852,2853,2859],{"type":60,"tag":82,"props":2842,"children":2843},{},[2844],{"type":66,"value":18},{"type":66,"value":819},{"type":60,"tag":98,"props":2847,"children":2849},{"className":2848},[],[2850],{"type":66,"value":2851},"af config",{"type":66,"value":783},{"type":60,"tag":98,"props":2854,"children":2856},{"className":2855},[],[2857],{"type":66,"value":2858},"af registry",{"type":66,"value":2860},", and general Airflow CLI reference.",{"type":60,"tag":2862,"props":2863,"children":2864},"style",{},[2865],{"type":66,"value":2866},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2868,"total":2127},[2869,2883,2894,2910,2917,2934,2947],{"slug":18,"name":18,"fn":2870,"description":2871,"org":2872,"tags":2873,"stars":25,"repoUrl":26,"updatedAt":2882},"manage and troubleshoot Airflow via CLI","Queries, manages, and troubleshoots Apache Airflow using the `af` CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading task logs, diagnosing failures, debugging import and parse errors, checking connections, variables and pools, exploring the REST API, and monitoring health (for example \"trigger a pipeline\", \"retry a run\", \"list connections\", \"check Airflow health\", \"why did my DAG fail\"). This is the entrypoint that routes to sibling skills for authoring, testing, deploying, and migrating Airflow 2 to 3. Not for warehouse\u002FSQL analytics on Airflow metadata tables (use analyzing-data); for deep root-cause reports use debugging-dags or airflow-investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2874,2875,2878,2879],{"name":17,"slug":18,"type":15},{"name":2876,"slug":2877,"type":15},"CLI","cli",{"name":23,"slug":24,"type":15},{"name":2880,"slug":2881,"type":15},"Debugging","debugging","2026-04-06T18:01:43.992997",{"slug":2835,"name":2835,"fn":2884,"description":2885,"org":2886,"tags":2887,"stars":25,"repoUrl":26,"updatedAt":2893},"add human-in-the-loop steps to Airflow DAGs","Builds human-in-the-loop (HITL) Airflow workflows - approval gates, form input, and human-driven branching. Use when a DAG needs a human in the loop - an approval or reject step, sign-off before a task runs, a decision or approval UI, branching on a human choice, or collecting form input mid-run; also on mentions of ApprovalOperator, HITLOperator, HITLBranchOperator, HITLEntryOperator, or HITLTrigger. Requires Airflow 3.1+. Not for AI\u002FLLM task calls (see migrating-ai-sdk-to-common-ai).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2888,2889,2892],{"name":17,"slug":18,"type":15},{"name":2890,"slug":2891,"type":15},"Approvals","approvals",{"name":23,"slug":24,"type":15},"2026-04-06T18:01:46.758548",{"slug":2895,"name":2895,"fn":2896,"description":2897,"org":2898,"tags":2899,"stars":25,"repoUrl":26,"updatedAt":2909},"airflow-plugins","build Airflow UI plugins","Builds Airflow 3.1+ plugins that embed FastAPI apps, custom UI pages, React components, middleware, macros, and operator links directly into the Airflow UI. Use when building anything custom inside Airflow 3.1+ that involves Python and a browser-facing interface - creating an Airflow plugin, adding a custom UI page or nav entry, building FastAPI-backed endpoints inside Airflow, serving static assets from a plugin, embedding a React app, adding middleware to the API server, creating custom operator extra links, or calling the Airflow REST API from inside a plugin; also when AirflowPlugin, fastapi_apps, external_views, react_apps, or plugin registration come up.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2900,2901,2904,2906],{"name":17,"slug":18,"type":15},{"name":2902,"slug":2903,"type":15},"Plugin Development","plugin-development",{"name":2905,"slug":509,"type":15},"Python",{"name":2907,"slug":2908,"type":15},"UI Components","ui-components","2026-04-06T18:01:56.827891",{"slug":4,"name":4,"fn":5,"description":6,"org":2911,"tags":2912,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2913,2914,2915,2916],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":2918,"name":2918,"fn":2919,"description":2920,"org":2921,"tags":2922,"stars":25,"repoUrl":26,"updatedAt":2933},"analyzing-data","query data warehouses for business questions","Queries the data warehouse with SQL and answers business questions about data. Use when answering anything that needs warehouse data - counts, metrics, trends, aggregations, joins across tables, data lookups, or ad-hoc SQL analysis (for example \"who uses X\", \"how many Y\", \"show me Z\", \"find customers\", \"what is the count\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2923,2926,2929,2930],{"name":2924,"slug":2925,"type":15},"Analytics","analytics",{"name":2927,"slug":2928,"type":15},"Data Analysis","data-analysis",{"name":20,"slug":21,"type":15},{"name":2931,"slug":2932,"type":15},"SQL","sql","2026-04-06T18:01:49.599775",{"slug":2935,"name":2935,"fn":2936,"description":2937,"org":2938,"tags":2939,"stars":25,"repoUrl":26,"updatedAt":2946},"annotating-task-lineage","annotate Airflow tasks with data lineage","Annotate Airflow tasks with data lineage using inlets and outlets. Use when the user wants to add lineage metadata to tasks, specify input\u002Foutput datasets, or enable lineage tracking for operators without built-in OpenLineage extraction.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2940,2941,2942,2943],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":2944,"slug":2945,"type":15},"Observability","observability","2026-04-06T18:02:03.487365",{"slug":2825,"name":2825,"fn":2948,"description":2949,"org":2950,"tags":2951,"stars":25,"repoUrl":26,"updatedAt":2958},"author Airflow DAGs","Workflow and best practices for writing Apache Airflow DAGs. Use when creating a new DAG, write pipeline code, handling questions about DAG patterns and conventions or extending an existing DAG with a follow-up\u002Fdownstream task. ANY request shaped like 'add a DAG named X', 'write a pipeline', 'add a task that runs after Y', or 'extend the DAG'. For testing and debugging DAGs, see the testing-dags skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2952,2953,2954,2957],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2955,"slug":2956,"type":15},"ETL","etl",{"name":2905,"slug":509,"type":15},"2026-04-06T18:01:52.679888",{"items":2960,"total":2127},[2961,2968,2974,2981,2988,2995,3002,3009,3024,3038,3048,3061],{"slug":18,"name":18,"fn":2870,"description":2871,"org":2962,"tags":2963,"stars":25,"repoUrl":26,"updatedAt":2882},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2964,2965,2966,2967],{"name":17,"slug":18,"type":15},{"name":2876,"slug":2877,"type":15},{"name":23,"slug":24,"type":15},{"name":2880,"slug":2881,"type":15},{"slug":2835,"name":2835,"fn":2884,"description":2885,"org":2969,"tags":2970,"stars":25,"repoUrl":26,"updatedAt":2893},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2971,2972,2973],{"name":17,"slug":18,"type":15},{"name":2890,"slug":2891,"type":15},{"name":23,"slug":24,"type":15},{"slug":2895,"name":2895,"fn":2896,"description":2897,"org":2975,"tags":2976,"stars":25,"repoUrl":26,"updatedAt":2909},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2977,2978,2979,2980],{"name":17,"slug":18,"type":15},{"name":2902,"slug":2903,"type":15},{"name":2905,"slug":509,"type":15},{"name":2907,"slug":2908,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2982,"tags":2983,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2984,2985,2986,2987],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":2918,"name":2918,"fn":2919,"description":2920,"org":2989,"tags":2990,"stars":25,"repoUrl":26,"updatedAt":2933},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2991,2992,2993,2994],{"name":2924,"slug":2925,"type":15},{"name":2927,"slug":2928,"type":15},{"name":20,"slug":21,"type":15},{"name":2931,"slug":2932,"type":15},{"slug":2935,"name":2935,"fn":2936,"description":2937,"org":2996,"tags":2997,"stars":25,"repoUrl":26,"updatedAt":2946},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2998,2999,3000,3001],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":2944,"slug":2945,"type":15},{"slug":2825,"name":2825,"fn":2948,"description":2949,"org":3003,"tags":3004,"stars":25,"repoUrl":26,"updatedAt":2958},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3005,3006,3007,3008],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2955,"slug":2956,"type":15},{"name":2905,"slug":509,"type":15},{"slug":3010,"name":3010,"fn":3011,"description":3012,"org":3013,"tags":3014,"stars":25,"repoUrl":26,"updatedAt":3023},"authoring-go-sdk-tasks","implement Airflow tasks in Go","Writes Airflow task logic in Go using the Airflow Go SDK. Use when the user wants to implement Airflow tasks in Go, asks about `BundleProvider`\u002F`RegisterDags`, the `bundlev1` Registry\u002FDag interfaces, registering Go tasks (`AddTask`\u002F`AddTaskWithName`), dependency injection by parameter type (`context.Context`, `sdk.TIRunContext`, `*slog.Logger`, `sdk.Client`), or reading connections\u002Fvariables\u002FXComs from Go. This skill covers the Go-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fpacking\u002Fshipping the bundle see deploying-go-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3015,3016,3019,3020],{"name":17,"slug":18,"type":15},{"name":3017,"slug":3018,"type":15},"API Development","api-development",{"name":23,"slug":24,"type":15},{"name":3021,"slug":3022,"type":15},"Go","go","2026-07-11T05:39:13.552213",{"slug":3025,"name":3025,"fn":3026,"description":3027,"org":3028,"tags":3029,"stars":25,"repoUrl":26,"updatedAt":3037},"authoring-java-sdk-tasks","implement Airflow tasks in Java","Writes Airflow task logic in Java, Kotlin, or any JVM language using the Airflow Java SDK. Use when the user wants to implement Airflow tasks in Java\u002FJVM, asks about `@Builder.Dag`\u002F`@Builder.Task`\u002F`@Builder.XCom`, the `Task`\u002F`BundleBuilder` interfaces, reading connections\u002Fvariables\u002FXComs from Java, the JSON-to-Java type mapping, or logging from Java tasks. This skill covers the Java-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fshipping the bundle see deploying-java-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3030,3031,3034],{"name":23,"slug":24,"type":15},{"name":3032,"slug":3033,"type":15},"Engineering","engineering",{"name":3035,"slug":3036,"type":15},"Java","java","2026-07-18T05:48:13.374003",{"slug":3039,"name":3039,"fn":3040,"description":3041,"org":3042,"tags":3043,"stars":25,"repoUrl":26,"updatedAt":3047},"authoring-language-sdk-tasks","implement non-Python Airflow tasks","The language-neutral foundation for Airflow language SDKs — implement task logic in a non-Python language while the DAG stays in Python. Use when the user wants to run an Airflow task in another language (Java, Kotlin, Go, or other JVM\u002Fnative languages), asks how the Python `@task.stub` pairs with native task code, how task\u002FDAG IDs must match across the two sides, how data passes via XCom as JSON, or which language SDKs exist. This skill owns the shared Python-stub pattern and conceptual model; for a specific language's native API, build, and runtime, use that language's skill (e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3044,3045,3046],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":3032,"slug":3033,"type":15},"2026-07-18T05:11:54.496539",{"slug":3049,"name":3049,"fn":3050,"description":3051,"org":3052,"tags":3053,"stars":25,"repoUrl":26,"updatedAt":3060},"blueprint","build reusable Airflow task group templates","Define reusable Airflow task group templates with Pydantic validation and compose DAGs from YAML. Use when creating blueprint templates, composing DAGs from YAML, validating configurations, or enabling no-code DAG authoring for non-engineers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3054,3055,3056,3057],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2955,"slug":2956,"type":15},{"name":3058,"slug":3059,"type":15},"Templates","templates","2026-04-06T18:01:45.361425",{"slug":3062,"name":3062,"fn":3063,"description":3064,"org":3065,"tags":3066,"stars":25,"repoUrl":26,"updatedAt":3073},"checking-freshness","check data freshness in warehouses","Quick data freshness check. Use when the user asks if data is up to date, when a table was last updated, if data is stale, or needs to verify data currency before using it.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3067,3068,3069,3072],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":3070,"slug":3071,"type":15},"Data Quality","data-quality",{"name":2955,"slug":2956,"type":15},"2026-04-06T18:02:02.138565"]