[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-astronomer-annotating-task-lineage":3,"mdc-eee64p-key":54,"related-org-astronomer-annotating-task-lineage":2366,"related-repo-astronomer-annotating-task-lineage":2525},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":49,"sourceUrl":52,"mdContent":53},"annotating-task-lineage","annotate Airflow tasks with data lineage","Annotate Airflow tasks with data lineage using inlets and outlets. Use when the user wants to add lineage metadata to tasks, specify input\u002Foutput datasets, or enable lineage tracking for operators without built-in OpenLineage extraction.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"astronomer","Astronomer","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fastronomer.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Airflow","airflow",{"name":20,"slug":21,"type":15},"Data Engineering","data-engineering",{"name":23,"slug":24,"type":15},"Data Pipeline","data-pipeline",412,"https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents","2026-04-06T18:02:03.487365",null,55,[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"agentic-workflow","agents","ai","ai-agents","apache-airflow","claude","cursor","dag","data-pipelines","dbt","llm","mcp","orchestrator","skills","workflow-automation","workflow-management","workflow-orchestration","workflows",{"repoUrl":26,"stars":25,"forks":29,"topics":50,"description":51},[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"AI agent tooling for data engineering workflows.","https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents\u002Ftree\u002FHEAD\u002Fskills\u002Fannotating-task-lineage","---\nname: annotating-task-lineage\ndescription: Annotate Airflow tasks with data lineage using inlets and outlets. Use when the user wants to add lineage metadata to tasks, specify input\u002Foutput datasets, or enable lineage tracking for operators without built-in OpenLineage extraction.\n---\n\n# Annotating Task Lineage with Inlets & Outlets\n\nThis skill guides you through adding manual lineage annotations to Airflow tasks using `inlets` and `outlets`.\n\n> **Reference:** See the [OpenLineage provider developer guide](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow-providers-openlineage\u002Fstable\u002Fguides\u002Fdeveloper.html) for the latest supported operators and patterns.\n\n### On Astro\n\nLineage annotations defined with inlets and outlets are visualized in Astro's enhanced **Lineage tab**, which provides cross-DAG and cross-deployment lineage views. This means your annotations are immediately visible in the Astro UI, giving you a unified view of data flow across your entire Astro organization.\n\n## When to Use This Approach\n\n| Scenario | Use Inlets\u002FOutlets? |\n|----------|---------------------|\n| Operator has OpenLineage methods (`get_openlineage_facets_on_*`) | ❌ Modify the OL method directly |\n| Operator has no built-in OpenLineage extractor | ✅ Yes |\n| Simple table-level lineage is sufficient | ✅ Yes |\n| Quick lineage setup without custom code | ✅ Yes |\n| Need column-level lineage | ❌ Use OpenLineage methods or custom extractor |\n| Complex extraction logic needed | ❌ Use OpenLineage methods or custom extractor |\n\n> **Note:** Inlets\u002Foutlets are the lowest-priority fallback. If an OpenLineage extractor or method exists for the operator, it takes precedence. Use this approach for operators without extractors.\n\n---\n\n## Supported Types for Inlets\u002FOutlets\n\nYou can use **OpenLineage Dataset** objects or **Airflow Assets** for inlets and outlets:\n\n### OpenLineage Datasets (Recommended)\n\n```python\nfrom openlineage.client.event_v2 import Dataset\n\n# Database tables\nsource_table = Dataset(\n    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n    name=\"public.orders\",\n)\ntarget_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"staging.orders_clean\",\n)\n\n# Files\ninput_file = Dataset(\n    namespace=\"s3:\u002F\u002Fmy-bucket\",\n    name=\"raw\u002Fevents\u002F2024-01-01.json\",\n)\n```\n\n### Airflow Assets (Airflow 3+)\n\n```python\nfrom airflow.sdk import Asset\n\n# Using Airflow's native Asset type\norders_asset = Asset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n```\n\n### Airflow Datasets (Airflow 2.4+)\n\n```python\nfrom airflow.datasets import Dataset\n\n# Using Airflow's Dataset type (Airflow 2.4-2.x)\norders_dataset = Dataset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n```\n\n---\n\n## Basic Usage\n\n### Setting Inlets and Outlets on Operators\n\n```python\nfrom airflow import DAG\nfrom airflow.operators.bash import BashOperator\nfrom openlineage.client.event_v2 import Dataset\nimport pendulum\n\n# Define your lineage datasets\nsource_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"raw.orders\",\n)\ntarget_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"staging.orders_clean\",\n)\noutput_file = Dataset(\n    namespace=\"s3:\u002F\u002Fmy-bucket\",\n    name=\"exports\u002Forders.parquet\",\n)\n\nwith DAG(\n    dag_id=\"etl_with_lineage\",\n    start_date=pendulum.datetime(2024, 1, 1, tz=\"UTC\"),\n    schedule=\"@daily\",\n) as dag:\n\n    transform = BashOperator(\n        task_id=\"transform_orders\",\n        bash_command=\"echo 'transforming...'\",\n        inlets=[source_table],           # What this task reads\n        outlets=[target_table],          # What this task writes\n    )\n\n    export = BashOperator(\n        task_id=\"export_to_s3\",\n        bash_command=\"echo 'exporting...'\",\n        inlets=[target_table],           # Reads from previous output\n        outlets=[output_file],           # Writes to S3\n    )\n\n    transform >> export\n```\n\n### Multiple Inputs and Outputs\n\nTasks often read from multiple sources and write to multiple destinations:\n\n```python\nfrom openlineage.client.event_v2 import Dataset\n\n# Multiple source tables\ncustomers = Dataset(namespace=\"postgres:\u002F\u002Fcrm:5432\", name=\"public.customers\")\norders = Dataset(namespace=\"postgres:\u002F\u002Fsales:5432\", name=\"public.orders\")\nproducts = Dataset(namespace=\"postgres:\u002F\u002Finventory:5432\", name=\"public.products\")\n\n# Multiple output tables\ndaily_summary = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.daily_summary\")\ncustomer_metrics = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.customer_metrics\")\n\naggregate_task = PythonOperator(\n    task_id=\"build_daily_aggregates\",\n    python_callable=build_aggregates,\n    inlets=[customers, orders, products],      # All inputs\n    outlets=[daily_summary, customer_metrics], # All outputs\n)\n```\n\n---\n\n## Setting Lineage in Custom Operators\n\nWhen building custom operators, you have two options:\n\n### Option 1: Implement OpenLineage Methods (Recommended)\n\nThis is the preferred approach as it gives you full control over lineage extraction:\n\n```python\nfrom airflow.models import BaseOperator\n\n\nclass MyCustomOperator(BaseOperator):\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n\n    def execute(self, context):\n        # ... perform the actual work ...\n        self.log.info(f\"Processing {self.source_table} -> {self.target_table}\")\n\n    def get_openlineage_facets_on_complete(self, task_instance):\n        \"\"\"Return lineage after successful execution.\"\"\"\n        from openlineage.client.event_v2 import Dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)],\n        )\n```\n\n### Option 2: Set Inlets\u002FOutlets Dynamically\n\nFor simpler cases, set lineage within the `execute` method (non-deferrable operators only):\n\n```python\nfrom airflow.models import BaseOperator\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass MyCustomOperator(BaseOperator):\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n\n    def execute(self, context):\n        # Set lineage dynamically based on operator parameters\n        self.inlets = [\n            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)\n        ]\n        self.outlets = [\n            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)\n        ]\n\n        # ... perform the actual work ...\n        self.log.info(f\"Processing {self.source_table} -> {self.target_table}\")\n```\n\n---\n\n## Dataset Naming Helpers\n\nUse the [OpenLineage dataset naming helpers](https:\u002F\u002Fopenlineage.io\u002Fdocs\u002Fclient\u002Fpython\u002Fbest-practices#dataset-naming-helpers) to ensure consistent naming across platforms:\n\n```python\nfrom openlineage.client.event_v2 import Dataset\n\n# Snowflake\nfrom openlineage.client.naming.snowflake import SnowflakeDatasetNaming\n\nnaming = SnowflakeDatasetNaming(\n    account_identifier=\"myorg-myaccount\",\n    database=\"mydb\",\n    schema=\"myschema\",\n    table=\"mytable\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"snowflake:\u002F\u002Fmyorg-myaccount\", name: \"mydb.myschema.mytable\"\n\n# BigQuery\nfrom openlineage.client.naming.bigquery import BigQueryDatasetNaming\n\nnaming = BigQueryDatasetNaming(\n    project=\"my-project\",\n    dataset=\"my_dataset\",\n    table=\"my_table\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"bigquery\", name: \"my-project.my_dataset.my_table\"\n\n# S3\nfrom openlineage.client.naming.s3 import S3DatasetNaming\n\nnaming = S3DatasetNaming(bucket=\"my-bucket\", key=\"path\u002Fto\u002Ffile.parquet\")\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"s3:\u002F\u002Fmy-bucket\", name: \"path\u002Fto\u002Ffile.parquet\"\n\n# PostgreSQL\nfrom openlineage.client.naming.postgres import PostgresDatasetNaming\n\nnaming = PostgresDatasetNaming(\n    host=\"localhost\",\n    port=5432,\n    database=\"mydb\",\n    schema=\"public\",\n    table=\"users\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"postgres:\u002F\u002Flocalhost:5432\", name: \"mydb.public.users\"\n```\n\n> **Note:** Always use the naming helpers instead of constructing namespaces manually. If a helper is missing for your platform, check the [OpenLineage repo](https:\u002F\u002Fgithub.com\u002FOpenLineage\u002FOpenLineage) or request it.\n\n---\n\n## Precedence Rules\n\nOpenLineage uses this precedence for lineage extraction:\n\n1. **Custom Extractors** (highest) - User-registered extractors\n2. **OpenLineage Methods** - `get_openlineage_facets_on_*` in operator\n3. **Hook-Level Lineage** - Lineage collected from hooks via `HookLineageCollector`\n4. **Inlets\u002FOutlets** (lowest) - Falls back to these if nothing else extracts lineage\n\n> **Note:** If an extractor or method exists but returns no datasets, OpenLineage will check hook-level lineage, then fall back to inlets\u002Foutlets.\n\n---\n\n## Best Practices\n\n### Use the Naming Helpers\n\nAlways use OpenLineage naming helpers for consistent dataset creation:\n\n```python\nfrom openlineage.client.event_v2 import Dataset\nfrom openlineage.client.naming.snowflake import SnowflakeDatasetNaming\n\n\ndef snowflake_dataset(schema: str, table: str) -> Dataset:\n    \"\"\"Create a Snowflake Dataset using the naming helper.\"\"\"\n    naming = SnowflakeDatasetNaming(\n        account_identifier=\"mycompany\",\n        database=\"analytics\",\n        schema=schema,\n        table=table,\n    )\n    return Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n\n\n# Usage\nsource = snowflake_dataset(\"raw\", \"orders\")\ntarget = snowflake_dataset(\"staging\", \"orders_clean\")\n```\n\n### Document Your Lineage\n\nAdd comments explaining the data flow:\n\n```python\ntransform = SqlOperator(\n    task_id=\"transform_orders\",\n    sql=\"...\",\n    # Lineage: Reads raw orders, joins with customers, writes to staging\n    inlets=[\n        snowflake_dataset(\"raw\", \"orders\"),\n        snowflake_dataset(\"raw\", \"customers\"),\n    ],\n    outlets=[\n        snowflake_dataset(\"staging\", \"order_details\"),\n    ],\n)\n```\n\n### Keep Lineage Accurate\n\n- Update inlets\u002Foutlets when SQL queries change\n- Include all tables referenced in JOINs as inlets\n- Include all tables written to (including temp tables if relevant)\n- **Outlet-only and inlet-only annotations are valid.** One-sided annotations are encouraged for lineage visibility even without a corresponding inlet or outlet in another DAG.\n\n---\n\n## Limitations\n\n| Limitation | Workaround |\n|------------|------------|\n| Table-level only (no column lineage) | Use OpenLineage methods or custom extractor |\n| Overridden by extractors\u002Fmethods | Only use for operators without extractors |\n| Static at DAG parse time | Set dynamically in `execute()` or use OL methods |\n| Deferrable operators lose dynamic lineage | Use OL methods instead; attributes set in `execute()` are lost when deferring |\n\n---\n\n## Related Skills\n\n- **creating-openlineage-extractors**: For column-level lineage or complex extraction\n- **tracing-upstream-lineage**: Investigate where data comes from\n- **tracing-downstream-lineage**: Investigate what depends on data\n",{"data":55,"body":56},{"name":4,"description":6},{"type":57,"children":58},"root",[59,68,91,117,124,136,143,254,267,271,277,296,302,466,472,510,516,554,557,563,569,903,909,914,1052,1055,1061,1066,1072,1077,1255,1261,1274,1434,1437,1443,1457,1804,1825,1828,1834,1839,1897,1909,1912,1918,1924,1929,2073,2079,2084,2185,2191,2220,2223,2229,2318,2321,2327,2360],{"type":60,"tag":61,"props":62,"children":64},"element","h1",{"id":63},"annotating-task-lineage-with-inlets-outlets",[65],{"type":66,"value":67},"text","Annotating Task Lineage with Inlets & Outlets",{"type":60,"tag":69,"props":70,"children":71},"p",{},[72,74,81,83,89],{"type":66,"value":73},"This skill guides you through adding manual lineage annotations to Airflow tasks using ",{"type":60,"tag":75,"props":76,"children":78},"code",{"className":77},[],[79],{"type":66,"value":80},"inlets",{"type":66,"value":82}," and ",{"type":60,"tag":75,"props":84,"children":86},{"className":85},[],[87],{"type":66,"value":88},"outlets",{"type":66,"value":90},".",{"type":60,"tag":92,"props":93,"children":94},"blockquote",{},[95],{"type":60,"tag":69,"props":96,"children":97},{},[98,104,106,115],{"type":60,"tag":99,"props":100,"children":101},"strong",{},[102],{"type":66,"value":103},"Reference:",{"type":66,"value":105}," See the ",{"type":60,"tag":107,"props":108,"children":112},"a",{"href":109,"rel":110},"https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow-providers-openlineage\u002Fstable\u002Fguides\u002Fdeveloper.html",[111],"nofollow",[113],{"type":66,"value":114},"OpenLineage provider developer guide",{"type":66,"value":116}," for the latest supported operators and patterns.",{"type":60,"tag":118,"props":119,"children":121},"h3",{"id":120},"on-astro",[122],{"type":66,"value":123},"On Astro",{"type":60,"tag":69,"props":125,"children":126},{},[127,129,134],{"type":66,"value":128},"Lineage annotations defined with inlets and outlets are visualized in Astro's enhanced ",{"type":60,"tag":99,"props":130,"children":131},{},[132],{"type":66,"value":133},"Lineage tab",{"type":66,"value":135},", which provides cross-DAG and cross-deployment lineage views. This means your annotations are immediately visible in the Astro UI, giving you a unified view of data flow across your entire Astro organization.",{"type":60,"tag":137,"props":138,"children":140},"h2",{"id":139},"when-to-use-this-approach",[141],{"type":66,"value":142},"When to Use This Approach",{"type":60,"tag":144,"props":145,"children":146},"table",{},[147,166],{"type":60,"tag":148,"props":149,"children":150},"thead",{},[151],{"type":60,"tag":152,"props":153,"children":154},"tr",{},[155,161],{"type":60,"tag":156,"props":157,"children":158},"th",{},[159],{"type":66,"value":160},"Scenario",{"type":60,"tag":156,"props":162,"children":163},{},[164],{"type":66,"value":165},"Use Inlets\u002FOutlets?",{"type":60,"tag":167,"props":168,"children":169},"tbody",{},[170,192,205,217,229,242],{"type":60,"tag":152,"props":171,"children":172},{},[173,187],{"type":60,"tag":174,"props":175,"children":176},"td",{},[177,179,185],{"type":66,"value":178},"Operator has OpenLineage methods (",{"type":60,"tag":75,"props":180,"children":182},{"className":181},[],[183],{"type":66,"value":184},"get_openlineage_facets_on_*",{"type":66,"value":186},")",{"type":60,"tag":174,"props":188,"children":189},{},[190],{"type":66,"value":191},"❌ Modify the OL method directly",{"type":60,"tag":152,"props":193,"children":194},{},[195,200],{"type":60,"tag":174,"props":196,"children":197},{},[198],{"type":66,"value":199},"Operator has no built-in OpenLineage extractor",{"type":60,"tag":174,"props":201,"children":202},{},[203],{"type":66,"value":204},"✅ Yes",{"type":60,"tag":152,"props":206,"children":207},{},[208,213],{"type":60,"tag":174,"props":209,"children":210},{},[211],{"type":66,"value":212},"Simple table-level lineage is sufficient",{"type":60,"tag":174,"props":214,"children":215},{},[216],{"type":66,"value":204},{"type":60,"tag":152,"props":218,"children":219},{},[220,225],{"type":60,"tag":174,"props":221,"children":222},{},[223],{"type":66,"value":224},"Quick lineage setup without custom code",{"type":60,"tag":174,"props":226,"children":227},{},[228],{"type":66,"value":204},{"type":60,"tag":152,"props":230,"children":231},{},[232,237],{"type":60,"tag":174,"props":233,"children":234},{},[235],{"type":66,"value":236},"Need column-level lineage",{"type":60,"tag":174,"props":238,"children":239},{},[240],{"type":66,"value":241},"❌ Use OpenLineage methods or custom extractor",{"type":60,"tag":152,"props":243,"children":244},{},[245,250],{"type":60,"tag":174,"props":246,"children":247},{},[248],{"type":66,"value":249},"Complex extraction logic needed",{"type":60,"tag":174,"props":251,"children":252},{},[253],{"type":66,"value":241},{"type":60,"tag":92,"props":255,"children":256},{},[257],{"type":60,"tag":69,"props":258,"children":259},{},[260,265],{"type":60,"tag":99,"props":261,"children":262},{},[263],{"type":66,"value":264},"Note:",{"type":66,"value":266}," Inlets\u002Foutlets are the lowest-priority fallback. If an OpenLineage extractor or method exists for the operator, it takes precedence. Use this approach for operators without extractors.",{"type":60,"tag":268,"props":269,"children":270},"hr",{},[],{"type":60,"tag":137,"props":272,"children":274},{"id":273},"supported-types-for-inletsoutlets",[275],{"type":66,"value":276},"Supported Types for Inlets\u002FOutlets",{"type":60,"tag":69,"props":278,"children":279},{},[280,282,287,289,294],{"type":66,"value":281},"You can use ",{"type":60,"tag":99,"props":283,"children":284},{},[285],{"type":66,"value":286},"OpenLineage Dataset",{"type":66,"value":288}," objects or ",{"type":60,"tag":99,"props":290,"children":291},{},[292],{"type":66,"value":293},"Airflow Assets",{"type":66,"value":295}," for inlets and outlets:",{"type":60,"tag":118,"props":297,"children":299},{"id":298},"openlineage-datasets-recommended",[300],{"type":66,"value":301},"OpenLineage Datasets (Recommended)",{"type":60,"tag":303,"props":304,"children":309},"pre",{"className":305,"code":306,"language":307,"meta":308,"style":308},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from openlineage.client.event_v2 import Dataset\n\n# Database tables\nsource_table = Dataset(\n    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n    name=\"public.orders\",\n)\ntarget_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"staging.orders_clean\",\n)\n\n# Files\ninput_file = Dataset(\n    namespace=\"s3:\u002F\u002Fmy-bucket\",\n    name=\"raw\u002Fevents\u002F2024-01-01.json\",\n)\n","python","",[310],{"type":60,"tag":75,"props":311,"children":312},{"__ignoreMap":308},[313,324,334,343,352,361,370,379,388,397,406,414,422,431,440,449,458],{"type":60,"tag":314,"props":315,"children":318},"span",{"class":316,"line":317},"line",1,[319],{"type":60,"tag":314,"props":320,"children":321},{},[322],{"type":66,"value":323},"from openlineage.client.event_v2 import Dataset\n",{"type":60,"tag":314,"props":325,"children":327},{"class":316,"line":326},2,[328],{"type":60,"tag":314,"props":329,"children":331},{"emptyLinePlaceholder":330},true,[332],{"type":66,"value":333},"\n",{"type":60,"tag":314,"props":335,"children":337},{"class":316,"line":336},3,[338],{"type":60,"tag":314,"props":339,"children":340},{},[341],{"type":66,"value":342},"# Database tables\n",{"type":60,"tag":314,"props":344,"children":346},{"class":316,"line":345},4,[347],{"type":60,"tag":314,"props":348,"children":349},{},[350],{"type":66,"value":351},"source_table = Dataset(\n",{"type":60,"tag":314,"props":353,"children":355},{"class":316,"line":354},5,[356],{"type":60,"tag":314,"props":357,"children":358},{},[359],{"type":66,"value":360},"    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n",{"type":60,"tag":314,"props":362,"children":364},{"class":316,"line":363},6,[365],{"type":60,"tag":314,"props":366,"children":367},{},[368],{"type":66,"value":369},"    name=\"public.orders\",\n",{"type":60,"tag":314,"props":371,"children":373},{"class":316,"line":372},7,[374],{"type":60,"tag":314,"props":375,"children":376},{},[377],{"type":66,"value":378},")\n",{"type":60,"tag":314,"props":380,"children":382},{"class":316,"line":381},8,[383],{"type":60,"tag":314,"props":384,"children":385},{},[386],{"type":66,"value":387},"target_table = Dataset(\n",{"type":60,"tag":314,"props":389,"children":391},{"class":316,"line":390},9,[392],{"type":60,"tag":314,"props":393,"children":394},{},[395],{"type":66,"value":396},"    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n",{"type":60,"tag":314,"props":398,"children":400},{"class":316,"line":399},10,[401],{"type":60,"tag":314,"props":402,"children":403},{},[404],{"type":66,"value":405},"    name=\"staging.orders_clean\",\n",{"type":60,"tag":314,"props":407,"children":409},{"class":316,"line":408},11,[410],{"type":60,"tag":314,"props":411,"children":412},{},[413],{"type":66,"value":378},{"type":60,"tag":314,"props":415,"children":417},{"class":316,"line":416},12,[418],{"type":60,"tag":314,"props":419,"children":420},{"emptyLinePlaceholder":330},[421],{"type":66,"value":333},{"type":60,"tag":314,"props":423,"children":425},{"class":316,"line":424},13,[426],{"type":60,"tag":314,"props":427,"children":428},{},[429],{"type":66,"value":430},"# Files\n",{"type":60,"tag":314,"props":432,"children":434},{"class":316,"line":433},14,[435],{"type":60,"tag":314,"props":436,"children":437},{},[438],{"type":66,"value":439},"input_file = Dataset(\n",{"type":60,"tag":314,"props":441,"children":443},{"class":316,"line":442},15,[444],{"type":60,"tag":314,"props":445,"children":446},{},[447],{"type":66,"value":448},"    namespace=\"s3:\u002F\u002Fmy-bucket\",\n",{"type":60,"tag":314,"props":450,"children":452},{"class":316,"line":451},16,[453],{"type":60,"tag":314,"props":454,"children":455},{},[456],{"type":66,"value":457},"    name=\"raw\u002Fevents\u002F2024-01-01.json\",\n",{"type":60,"tag":314,"props":459,"children":461},{"class":316,"line":460},17,[462],{"type":60,"tag":314,"props":463,"children":464},{},[465],{"type":66,"value":378},{"type":60,"tag":118,"props":467,"children":469},{"id":468},"airflow-assets-airflow-3",[470],{"type":66,"value":471},"Airflow Assets (Airflow 3+)",{"type":60,"tag":303,"props":473,"children":475},{"className":305,"code":474,"language":307,"meta":308,"style":308},"from airflow.sdk import Asset\n\n# Using Airflow's native Asset type\norders_asset = Asset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n",[476],{"type":60,"tag":75,"props":477,"children":478},{"__ignoreMap":308},[479,487,494,502],{"type":60,"tag":314,"props":480,"children":481},{"class":316,"line":317},[482],{"type":60,"tag":314,"props":483,"children":484},{},[485],{"type":66,"value":486},"from airflow.sdk import Asset\n",{"type":60,"tag":314,"props":488,"children":489},{"class":316,"line":326},[490],{"type":60,"tag":314,"props":491,"children":492},{"emptyLinePlaceholder":330},[493],{"type":66,"value":333},{"type":60,"tag":314,"props":495,"children":496},{"class":316,"line":336},[497],{"type":60,"tag":314,"props":498,"children":499},{},[500],{"type":66,"value":501},"# Using Airflow's native Asset type\n",{"type":60,"tag":314,"props":503,"children":504},{"class":316,"line":345},[505],{"type":60,"tag":314,"props":506,"children":507},{},[508],{"type":66,"value":509},"orders_asset = Asset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n",{"type":60,"tag":118,"props":511,"children":513},{"id":512},"airflow-datasets-airflow-24",[514],{"type":66,"value":515},"Airflow Datasets (Airflow 2.4+)",{"type":60,"tag":303,"props":517,"children":519},{"className":305,"code":518,"language":307,"meta":308,"style":308},"from airflow.datasets import Dataset\n\n# Using Airflow's Dataset type (Airflow 2.4-2.x)\norders_dataset = Dataset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n",[520],{"type":60,"tag":75,"props":521,"children":522},{"__ignoreMap":308},[523,531,538,546],{"type":60,"tag":314,"props":524,"children":525},{"class":316,"line":317},[526],{"type":60,"tag":314,"props":527,"children":528},{},[529],{"type":66,"value":530},"from airflow.datasets import Dataset\n",{"type":60,"tag":314,"props":532,"children":533},{"class":316,"line":326},[534],{"type":60,"tag":314,"props":535,"children":536},{"emptyLinePlaceholder":330},[537],{"type":66,"value":333},{"type":60,"tag":314,"props":539,"children":540},{"class":316,"line":336},[541],{"type":60,"tag":314,"props":542,"children":543},{},[544],{"type":66,"value":545},"# Using Airflow's Dataset type (Airflow 2.4-2.x)\n",{"type":60,"tag":314,"props":547,"children":548},{"class":316,"line":345},[549],{"type":60,"tag":314,"props":550,"children":551},{},[552],{"type":66,"value":553},"orders_dataset = Dataset(uri=\"s3:\u002F\u002Fmy-bucket\u002Fdata\u002Forders\")\n",{"type":60,"tag":268,"props":555,"children":556},{},[],{"type":60,"tag":137,"props":558,"children":560},{"id":559},"basic-usage",[561],{"type":66,"value":562},"Basic Usage",{"type":60,"tag":118,"props":564,"children":566},{"id":565},"setting-inlets-and-outlets-on-operators",[567],{"type":66,"value":568},"Setting Inlets and Outlets on Operators",{"type":60,"tag":303,"props":570,"children":572},{"className":305,"code":571,"language":307,"meta":308,"style":308},"from airflow import DAG\nfrom airflow.operators.bash import BashOperator\nfrom openlineage.client.event_v2 import Dataset\nimport pendulum\n\n# Define your lineage datasets\nsource_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"raw.orders\",\n)\ntarget_table = Dataset(\n    namespace=\"snowflake:\u002F\u002Faccount.snowflakecomputing.com\",\n    name=\"staging.orders_clean\",\n)\noutput_file = Dataset(\n    namespace=\"s3:\u002F\u002Fmy-bucket\",\n    name=\"exports\u002Forders.parquet\",\n)\n\nwith DAG(\n    dag_id=\"etl_with_lineage\",\n    start_date=pendulum.datetime(2024, 1, 1, tz=\"UTC\"),\n    schedule=\"@daily\",\n) as dag:\n\n    transform = BashOperator(\n        task_id=\"transform_orders\",\n        bash_command=\"echo 'transforming...'\",\n        inlets=[source_table],           # What this task reads\n        outlets=[target_table],          # What this task writes\n    )\n\n    export = BashOperator(\n        task_id=\"export_to_s3\",\n        bash_command=\"echo 'exporting...'\",\n        inlets=[target_table],           # Reads from previous output\n        outlets=[output_file],           # Writes to S3\n    )\n\n    transform >> export\n",[573],{"type":60,"tag":75,"props":574,"children":575},{"__ignoreMap":308},[576,584,592,599,607,614,622,629,636,644,651,658,665,672,679,687,694,702,710,718,727,736,745,754,763,771,780,789,798,807,816,825,833,842,851,860,869,878,886,894],{"type":60,"tag":314,"props":577,"children":578},{"class":316,"line":317},[579],{"type":60,"tag":314,"props":580,"children":581},{},[582],{"type":66,"value":583},"from airflow import DAG\n",{"type":60,"tag":314,"props":585,"children":586},{"class":316,"line":326},[587],{"type":60,"tag":314,"props":588,"children":589},{},[590],{"type":66,"value":591},"from airflow.operators.bash import BashOperator\n",{"type":60,"tag":314,"props":593,"children":594},{"class":316,"line":336},[595],{"type":60,"tag":314,"props":596,"children":597},{},[598],{"type":66,"value":323},{"type":60,"tag":314,"props":600,"children":601},{"class":316,"line":345},[602],{"type":60,"tag":314,"props":603,"children":604},{},[605],{"type":66,"value":606},"import pendulum\n",{"type":60,"tag":314,"props":608,"children":609},{"class":316,"line":354},[610],{"type":60,"tag":314,"props":611,"children":612},{"emptyLinePlaceholder":330},[613],{"type":66,"value":333},{"type":60,"tag":314,"props":615,"children":616},{"class":316,"line":363},[617],{"type":60,"tag":314,"props":618,"children":619},{},[620],{"type":66,"value":621},"# Define your lineage datasets\n",{"type":60,"tag":314,"props":623,"children":624},{"class":316,"line":372},[625],{"type":60,"tag":314,"props":626,"children":627},{},[628],{"type":66,"value":351},{"type":60,"tag":314,"props":630,"children":631},{"class":316,"line":381},[632],{"type":60,"tag":314,"props":633,"children":634},{},[635],{"type":66,"value":396},{"type":60,"tag":314,"props":637,"children":638},{"class":316,"line":390},[639],{"type":60,"tag":314,"props":640,"children":641},{},[642],{"type":66,"value":643},"    name=\"raw.orders\",\n",{"type":60,"tag":314,"props":645,"children":646},{"class":316,"line":399},[647],{"type":60,"tag":314,"props":648,"children":649},{},[650],{"type":66,"value":378},{"type":60,"tag":314,"props":652,"children":653},{"class":316,"line":408},[654],{"type":60,"tag":314,"props":655,"children":656},{},[657],{"type":66,"value":387},{"type":60,"tag":314,"props":659,"children":660},{"class":316,"line":416},[661],{"type":60,"tag":314,"props":662,"children":663},{},[664],{"type":66,"value":396},{"type":60,"tag":314,"props":666,"children":667},{"class":316,"line":424},[668],{"type":60,"tag":314,"props":669,"children":670},{},[671],{"type":66,"value":405},{"type":60,"tag":314,"props":673,"children":674},{"class":316,"line":433},[675],{"type":60,"tag":314,"props":676,"children":677},{},[678],{"type":66,"value":378},{"type":60,"tag":314,"props":680,"children":681},{"class":316,"line":442},[682],{"type":60,"tag":314,"props":683,"children":684},{},[685],{"type":66,"value":686},"output_file = Dataset(\n",{"type":60,"tag":314,"props":688,"children":689},{"class":316,"line":451},[690],{"type":60,"tag":314,"props":691,"children":692},{},[693],{"type":66,"value":448},{"type":60,"tag":314,"props":695,"children":696},{"class":316,"line":460},[697],{"type":60,"tag":314,"props":698,"children":699},{},[700],{"type":66,"value":701},"    name=\"exports\u002Forders.parquet\",\n",{"type":60,"tag":314,"props":703,"children":705},{"class":316,"line":704},18,[706],{"type":60,"tag":314,"props":707,"children":708},{},[709],{"type":66,"value":378},{"type":60,"tag":314,"props":711,"children":713},{"class":316,"line":712},19,[714],{"type":60,"tag":314,"props":715,"children":716},{"emptyLinePlaceholder":330},[717],{"type":66,"value":333},{"type":60,"tag":314,"props":719,"children":721},{"class":316,"line":720},20,[722],{"type":60,"tag":314,"props":723,"children":724},{},[725],{"type":66,"value":726},"with DAG(\n",{"type":60,"tag":314,"props":728,"children":730},{"class":316,"line":729},21,[731],{"type":60,"tag":314,"props":732,"children":733},{},[734],{"type":66,"value":735},"    dag_id=\"etl_with_lineage\",\n",{"type":60,"tag":314,"props":737,"children":739},{"class":316,"line":738},22,[740],{"type":60,"tag":314,"props":741,"children":742},{},[743],{"type":66,"value":744},"    start_date=pendulum.datetime(2024, 1, 1, tz=\"UTC\"),\n",{"type":60,"tag":314,"props":746,"children":748},{"class":316,"line":747},23,[749],{"type":60,"tag":314,"props":750,"children":751},{},[752],{"type":66,"value":753},"    schedule=\"@daily\",\n",{"type":60,"tag":314,"props":755,"children":757},{"class":316,"line":756},24,[758],{"type":60,"tag":314,"props":759,"children":760},{},[761],{"type":66,"value":762},") as dag:\n",{"type":60,"tag":314,"props":764,"children":766},{"class":316,"line":765},25,[767],{"type":60,"tag":314,"props":768,"children":769},{"emptyLinePlaceholder":330},[770],{"type":66,"value":333},{"type":60,"tag":314,"props":772,"children":774},{"class":316,"line":773},26,[775],{"type":60,"tag":314,"props":776,"children":777},{},[778],{"type":66,"value":779},"    transform = BashOperator(\n",{"type":60,"tag":314,"props":781,"children":783},{"class":316,"line":782},27,[784],{"type":60,"tag":314,"props":785,"children":786},{},[787],{"type":66,"value":788},"        task_id=\"transform_orders\",\n",{"type":60,"tag":314,"props":790,"children":792},{"class":316,"line":791},28,[793],{"type":60,"tag":314,"props":794,"children":795},{},[796],{"type":66,"value":797},"        bash_command=\"echo 'transforming...'\",\n",{"type":60,"tag":314,"props":799,"children":801},{"class":316,"line":800},29,[802],{"type":60,"tag":314,"props":803,"children":804},{},[805],{"type":66,"value":806},"        inlets=[source_table],           # What this task reads\n",{"type":60,"tag":314,"props":808,"children":810},{"class":316,"line":809},30,[811],{"type":60,"tag":314,"props":812,"children":813},{},[814],{"type":66,"value":815},"        outlets=[target_table],          # What this task writes\n",{"type":60,"tag":314,"props":817,"children":819},{"class":316,"line":818},31,[820],{"type":60,"tag":314,"props":821,"children":822},{},[823],{"type":66,"value":824},"    )\n",{"type":60,"tag":314,"props":826,"children":828},{"class":316,"line":827},32,[829],{"type":60,"tag":314,"props":830,"children":831},{"emptyLinePlaceholder":330},[832],{"type":66,"value":333},{"type":60,"tag":314,"props":834,"children":836},{"class":316,"line":835},33,[837],{"type":60,"tag":314,"props":838,"children":839},{},[840],{"type":66,"value":841},"    export = BashOperator(\n",{"type":60,"tag":314,"props":843,"children":845},{"class":316,"line":844},34,[846],{"type":60,"tag":314,"props":847,"children":848},{},[849],{"type":66,"value":850},"        task_id=\"export_to_s3\",\n",{"type":60,"tag":314,"props":852,"children":854},{"class":316,"line":853},35,[855],{"type":60,"tag":314,"props":856,"children":857},{},[858],{"type":66,"value":859},"        bash_command=\"echo 'exporting...'\",\n",{"type":60,"tag":314,"props":861,"children":863},{"class":316,"line":862},36,[864],{"type":60,"tag":314,"props":865,"children":866},{},[867],{"type":66,"value":868},"        inlets=[target_table],           # Reads from previous output\n",{"type":60,"tag":314,"props":870,"children":872},{"class":316,"line":871},37,[873],{"type":60,"tag":314,"props":874,"children":875},{},[876],{"type":66,"value":877},"        outlets=[output_file],           # Writes to S3\n",{"type":60,"tag":314,"props":879,"children":881},{"class":316,"line":880},38,[882],{"type":60,"tag":314,"props":883,"children":884},{},[885],{"type":66,"value":824},{"type":60,"tag":314,"props":887,"children":889},{"class":316,"line":888},39,[890],{"type":60,"tag":314,"props":891,"children":892},{"emptyLinePlaceholder":330},[893],{"type":66,"value":333},{"type":60,"tag":314,"props":895,"children":897},{"class":316,"line":896},40,[898],{"type":60,"tag":314,"props":899,"children":900},{},[901],{"type":66,"value":902},"    transform >> export\n",{"type":60,"tag":118,"props":904,"children":906},{"id":905},"multiple-inputs-and-outputs",[907],{"type":66,"value":908},"Multiple Inputs and Outputs",{"type":60,"tag":69,"props":910,"children":911},{},[912],{"type":66,"value":913},"Tasks often read from multiple sources and write to multiple destinations:",{"type":60,"tag":303,"props":915,"children":917},{"className":305,"code":916,"language":307,"meta":308,"style":308},"from openlineage.client.event_v2 import Dataset\n\n# Multiple source tables\ncustomers = Dataset(namespace=\"postgres:\u002F\u002Fcrm:5432\", name=\"public.customers\")\norders = Dataset(namespace=\"postgres:\u002F\u002Fsales:5432\", name=\"public.orders\")\nproducts = Dataset(namespace=\"postgres:\u002F\u002Finventory:5432\", name=\"public.products\")\n\n# Multiple output tables\ndaily_summary = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.daily_summary\")\ncustomer_metrics = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.customer_metrics\")\n\naggregate_task = PythonOperator(\n    task_id=\"build_daily_aggregates\",\n    python_callable=build_aggregates,\n    inlets=[customers, orders, products],      # All inputs\n    outlets=[daily_summary, customer_metrics], # All outputs\n)\n",[918],{"type":60,"tag":75,"props":919,"children":920},{"__ignoreMap":308},[921,928,935,943,951,959,967,974,982,990,998,1005,1013,1021,1029,1037,1045],{"type":60,"tag":314,"props":922,"children":923},{"class":316,"line":317},[924],{"type":60,"tag":314,"props":925,"children":926},{},[927],{"type":66,"value":323},{"type":60,"tag":314,"props":929,"children":930},{"class":316,"line":326},[931],{"type":60,"tag":314,"props":932,"children":933},{"emptyLinePlaceholder":330},[934],{"type":66,"value":333},{"type":60,"tag":314,"props":936,"children":937},{"class":316,"line":336},[938],{"type":60,"tag":314,"props":939,"children":940},{},[941],{"type":66,"value":942},"# Multiple source tables\n",{"type":60,"tag":314,"props":944,"children":945},{"class":316,"line":345},[946],{"type":60,"tag":314,"props":947,"children":948},{},[949],{"type":66,"value":950},"customers = Dataset(namespace=\"postgres:\u002F\u002Fcrm:5432\", name=\"public.customers\")\n",{"type":60,"tag":314,"props":952,"children":953},{"class":316,"line":354},[954],{"type":60,"tag":314,"props":955,"children":956},{},[957],{"type":66,"value":958},"orders = Dataset(namespace=\"postgres:\u002F\u002Fsales:5432\", name=\"public.orders\")\n",{"type":60,"tag":314,"props":960,"children":961},{"class":316,"line":363},[962],{"type":60,"tag":314,"props":963,"children":964},{},[965],{"type":66,"value":966},"products = Dataset(namespace=\"postgres:\u002F\u002Finventory:5432\", name=\"public.products\")\n",{"type":60,"tag":314,"props":968,"children":969},{"class":316,"line":372},[970],{"type":60,"tag":314,"props":971,"children":972},{"emptyLinePlaceholder":330},[973],{"type":66,"value":333},{"type":60,"tag":314,"props":975,"children":976},{"class":316,"line":381},[977],{"type":60,"tag":314,"props":978,"children":979},{},[980],{"type":66,"value":981},"# Multiple output tables\n",{"type":60,"tag":314,"props":983,"children":984},{"class":316,"line":390},[985],{"type":60,"tag":314,"props":986,"children":987},{},[988],{"type":66,"value":989},"daily_summary = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.daily_summary\")\n",{"type":60,"tag":314,"props":991,"children":992},{"class":316,"line":399},[993],{"type":60,"tag":314,"props":994,"children":995},{},[996],{"type":66,"value":997},"customer_metrics = Dataset(namespace=\"snowflake:\u002F\u002Faccount\", name=\"analytics.customer_metrics\")\n",{"type":60,"tag":314,"props":999,"children":1000},{"class":316,"line":408},[1001],{"type":60,"tag":314,"props":1002,"children":1003},{"emptyLinePlaceholder":330},[1004],{"type":66,"value":333},{"type":60,"tag":314,"props":1006,"children":1007},{"class":316,"line":416},[1008],{"type":60,"tag":314,"props":1009,"children":1010},{},[1011],{"type":66,"value":1012},"aggregate_task = PythonOperator(\n",{"type":60,"tag":314,"props":1014,"children":1015},{"class":316,"line":424},[1016],{"type":60,"tag":314,"props":1017,"children":1018},{},[1019],{"type":66,"value":1020},"    task_id=\"build_daily_aggregates\",\n",{"type":60,"tag":314,"props":1022,"children":1023},{"class":316,"line":433},[1024],{"type":60,"tag":314,"props":1025,"children":1026},{},[1027],{"type":66,"value":1028},"    python_callable=build_aggregates,\n",{"type":60,"tag":314,"props":1030,"children":1031},{"class":316,"line":442},[1032],{"type":60,"tag":314,"props":1033,"children":1034},{},[1035],{"type":66,"value":1036},"    inlets=[customers, orders, products],      # All inputs\n",{"type":60,"tag":314,"props":1038,"children":1039},{"class":316,"line":451},[1040],{"type":60,"tag":314,"props":1041,"children":1042},{},[1043],{"type":66,"value":1044},"    outlets=[daily_summary, customer_metrics], # All outputs\n",{"type":60,"tag":314,"props":1046,"children":1047},{"class":316,"line":460},[1048],{"type":60,"tag":314,"props":1049,"children":1050},{},[1051],{"type":66,"value":378},{"type":60,"tag":268,"props":1053,"children":1054},{},[],{"type":60,"tag":137,"props":1056,"children":1058},{"id":1057},"setting-lineage-in-custom-operators",[1059],{"type":66,"value":1060},"Setting Lineage in Custom Operators",{"type":60,"tag":69,"props":1062,"children":1063},{},[1064],{"type":66,"value":1065},"When building custom operators, you have two options:",{"type":60,"tag":118,"props":1067,"children":1069},{"id":1068},"option-1-implement-openlineage-methods-recommended",[1070],{"type":66,"value":1071},"Option 1: Implement OpenLineage Methods (Recommended)",{"type":60,"tag":69,"props":1073,"children":1074},{},[1075],{"type":66,"value":1076},"This is the preferred approach as it gives you full control over lineage extraction:",{"type":60,"tag":303,"props":1078,"children":1080},{"className":305,"code":1079,"language":307,"meta":308,"style":308},"from airflow.models import BaseOperator\n\n\nclass MyCustomOperator(BaseOperator):\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n\n    def execute(self, context):\n        # ... perform the actual work ...\n        self.log.info(f\"Processing {self.source_table} -> {self.target_table}\")\n\n    def get_openlineage_facets_on_complete(self, task_instance):\n        \"\"\"Return lineage after successful execution.\"\"\"\n        from openlineage.client.event_v2 import Dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)],\n        )\n",[1081],{"type":60,"tag":75,"props":1082,"children":1083},{"__ignoreMap":308},[1084,1092,1099,1106,1114,1122,1130,1138,1146,1153,1161,1169,1177,1184,1192,1200,1208,1216,1223,1231,1239,1247],{"type":60,"tag":314,"props":1085,"children":1086},{"class":316,"line":317},[1087],{"type":60,"tag":314,"props":1088,"children":1089},{},[1090],{"type":66,"value":1091},"from airflow.models import BaseOperator\n",{"type":60,"tag":314,"props":1093,"children":1094},{"class":316,"line":326},[1095],{"type":60,"tag":314,"props":1096,"children":1097},{"emptyLinePlaceholder":330},[1098],{"type":66,"value":333},{"type":60,"tag":314,"props":1100,"children":1101},{"class":316,"line":336},[1102],{"type":60,"tag":314,"props":1103,"children":1104},{"emptyLinePlaceholder":330},[1105],{"type":66,"value":333},{"type":60,"tag":314,"props":1107,"children":1108},{"class":316,"line":345},[1109],{"type":60,"tag":314,"props":1110,"children":1111},{},[1112],{"type":66,"value":1113},"class MyCustomOperator(BaseOperator):\n",{"type":60,"tag":314,"props":1115,"children":1116},{"class":316,"line":354},[1117],{"type":60,"tag":314,"props":1118,"children":1119},{},[1120],{"type":66,"value":1121},"    def __init__(self, source_table: str, target_table: str, **kwargs):\n",{"type":60,"tag":314,"props":1123,"children":1124},{"class":316,"line":363},[1125],{"type":60,"tag":314,"props":1126,"children":1127},{},[1128],{"type":66,"value":1129},"        super().__init__(**kwargs)\n",{"type":60,"tag":314,"props":1131,"children":1132},{"class":316,"line":372},[1133],{"type":60,"tag":314,"props":1134,"children":1135},{},[1136],{"type":66,"value":1137},"        self.source_table = source_table\n",{"type":60,"tag":314,"props":1139,"children":1140},{"class":316,"line":381},[1141],{"type":60,"tag":314,"props":1142,"children":1143},{},[1144],{"type":66,"value":1145},"        self.target_table = target_table\n",{"type":60,"tag":314,"props":1147,"children":1148},{"class":316,"line":390},[1149],{"type":60,"tag":314,"props":1150,"children":1151},{"emptyLinePlaceholder":330},[1152],{"type":66,"value":333},{"type":60,"tag":314,"props":1154,"children":1155},{"class":316,"line":399},[1156],{"type":60,"tag":314,"props":1157,"children":1158},{},[1159],{"type":66,"value":1160},"    def execute(self, context):\n",{"type":60,"tag":314,"props":1162,"children":1163},{"class":316,"line":408},[1164],{"type":60,"tag":314,"props":1165,"children":1166},{},[1167],{"type":66,"value":1168},"        # ... perform the actual work ...\n",{"type":60,"tag":314,"props":1170,"children":1171},{"class":316,"line":416},[1172],{"type":60,"tag":314,"props":1173,"children":1174},{},[1175],{"type":66,"value":1176},"        self.log.info(f\"Processing {self.source_table} -> {self.target_table}\")\n",{"type":60,"tag":314,"props":1178,"children":1179},{"class":316,"line":424},[1180],{"type":60,"tag":314,"props":1181,"children":1182},{"emptyLinePlaceholder":330},[1183],{"type":66,"value":333},{"type":60,"tag":314,"props":1185,"children":1186},{"class":316,"line":433},[1187],{"type":60,"tag":314,"props":1188,"children":1189},{},[1190],{"type":66,"value":1191},"    def get_openlineage_facets_on_complete(self, task_instance):\n",{"type":60,"tag":314,"props":1193,"children":1194},{"class":316,"line":442},[1195],{"type":60,"tag":314,"props":1196,"children":1197},{},[1198],{"type":66,"value":1199},"        \"\"\"Return lineage after successful execution.\"\"\"\n",{"type":60,"tag":314,"props":1201,"children":1202},{"class":316,"line":451},[1203],{"type":60,"tag":314,"props":1204,"children":1205},{},[1206],{"type":66,"value":1207},"        from openlineage.client.event_v2 import Dataset\n",{"type":60,"tag":314,"props":1209,"children":1210},{"class":316,"line":460},[1211],{"type":60,"tag":314,"props":1212,"children":1213},{},[1214],{"type":66,"value":1215},"        from airflow.providers.openlineage.extractors import OperatorLineage\n",{"type":60,"tag":314,"props":1217,"children":1218},{"class":316,"line":704},[1219],{"type":60,"tag":314,"props":1220,"children":1221},{"emptyLinePlaceholder":330},[1222],{"type":66,"value":333},{"type":60,"tag":314,"props":1224,"children":1225},{"class":316,"line":712},[1226],{"type":60,"tag":314,"props":1227,"children":1228},{},[1229],{"type":66,"value":1230},"        return OperatorLineage(\n",{"type":60,"tag":314,"props":1232,"children":1233},{"class":316,"line":720},[1234],{"type":60,"tag":314,"props":1235,"children":1236},{},[1237],{"type":66,"value":1238},"            inputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)],\n",{"type":60,"tag":314,"props":1240,"children":1241},{"class":316,"line":729},[1242],{"type":60,"tag":314,"props":1243,"children":1244},{},[1245],{"type":66,"value":1246},"            outputs=[Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)],\n",{"type":60,"tag":314,"props":1248,"children":1249},{"class":316,"line":738},[1250],{"type":60,"tag":314,"props":1251,"children":1252},{},[1253],{"type":66,"value":1254},"        )\n",{"type":60,"tag":118,"props":1256,"children":1258},{"id":1257},"option-2-set-inletsoutlets-dynamically",[1259],{"type":66,"value":1260},"Option 2: Set Inlets\u002FOutlets Dynamically",{"type":60,"tag":69,"props":1262,"children":1263},{},[1264,1266,1272],{"type":66,"value":1265},"For simpler cases, set lineage within the ",{"type":60,"tag":75,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":66,"value":1271},"execute",{"type":66,"value":1273}," method (non-deferrable operators only):",{"type":60,"tag":303,"props":1275,"children":1277},{"className":305,"code":1276,"language":307,"meta":308,"style":308},"from airflow.models import BaseOperator\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass MyCustomOperator(BaseOperator):\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n\n    def execute(self, context):\n        # Set lineage dynamically based on operator parameters\n        self.inlets = [\n            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)\n        ]\n        self.outlets = [\n            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)\n        ]\n\n        # ... perform the actual work ...\n        self.log.info(f\"Processing {self.source_table} -> {self.target_table}\")\n",[1278],{"type":60,"tag":75,"props":1279,"children":1280},{"__ignoreMap":308},[1281,1288,1295,1302,1309,1316,1323,1330,1337,1344,1351,1358,1366,1374,1382,1390,1398,1406,1413,1420,1427],{"type":60,"tag":314,"props":1282,"children":1283},{"class":316,"line":317},[1284],{"type":60,"tag":314,"props":1285,"children":1286},{},[1287],{"type":66,"value":1091},{"type":60,"tag":314,"props":1289,"children":1290},{"class":316,"line":326},[1291],{"type":60,"tag":314,"props":1292,"children":1293},{},[1294],{"type":66,"value":323},{"type":60,"tag":314,"props":1296,"children":1297},{"class":316,"line":336},[1298],{"type":60,"tag":314,"props":1299,"children":1300},{"emptyLinePlaceholder":330},[1301],{"type":66,"value":333},{"type":60,"tag":314,"props":1303,"children":1304},{"class":316,"line":345},[1305],{"type":60,"tag":314,"props":1306,"children":1307},{"emptyLinePlaceholder":330},[1308],{"type":66,"value":333},{"type":60,"tag":314,"props":1310,"children":1311},{"class":316,"line":354},[1312],{"type":60,"tag":314,"props":1313,"children":1314},{},[1315],{"type":66,"value":1113},{"type":60,"tag":314,"props":1317,"children":1318},{"class":316,"line":363},[1319],{"type":60,"tag":314,"props":1320,"children":1321},{},[1322],{"type":66,"value":1121},{"type":60,"tag":314,"props":1324,"children":1325},{"class":316,"line":372},[1326],{"type":60,"tag":314,"props":1327,"children":1328},{},[1329],{"type":66,"value":1129},{"type":60,"tag":314,"props":1331,"children":1332},{"class":316,"line":381},[1333],{"type":60,"tag":314,"props":1334,"children":1335},{},[1336],{"type":66,"value":1137},{"type":60,"tag":314,"props":1338,"children":1339},{"class":316,"line":390},[1340],{"type":60,"tag":314,"props":1341,"children":1342},{},[1343],{"type":66,"value":1145},{"type":60,"tag":314,"props":1345,"children":1346},{"class":316,"line":399},[1347],{"type":60,"tag":314,"props":1348,"children":1349},{"emptyLinePlaceholder":330},[1350],{"type":66,"value":333},{"type":60,"tag":314,"props":1352,"children":1353},{"class":316,"line":408},[1354],{"type":60,"tag":314,"props":1355,"children":1356},{},[1357],{"type":66,"value":1160},{"type":60,"tag":314,"props":1359,"children":1360},{"class":316,"line":416},[1361],{"type":60,"tag":314,"props":1362,"children":1363},{},[1364],{"type":66,"value":1365},"        # Set lineage dynamically based on operator parameters\n",{"type":60,"tag":314,"props":1367,"children":1368},{"class":316,"line":424},[1369],{"type":60,"tag":314,"props":1370,"children":1371},{},[1372],{"type":66,"value":1373},"        self.inlets = [\n",{"type":60,"tag":314,"props":1375,"children":1376},{"class":316,"line":433},[1377],{"type":60,"tag":314,"props":1378,"children":1379},{},[1380],{"type":66,"value":1381},"            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.source_table)\n",{"type":60,"tag":314,"props":1383,"children":1384},{"class":316,"line":442},[1385],{"type":60,"tag":314,"props":1386,"children":1387},{},[1388],{"type":66,"value":1389},"        ]\n",{"type":60,"tag":314,"props":1391,"children":1392},{"class":316,"line":451},[1393],{"type":60,"tag":314,"props":1394,"children":1395},{},[1396],{"type":66,"value":1397},"        self.outlets = [\n",{"type":60,"tag":314,"props":1399,"children":1400},{"class":316,"line":460},[1401],{"type":60,"tag":314,"props":1402,"children":1403},{},[1404],{"type":66,"value":1405},"            Dataset(namespace=\"warehouse:\u002F\u002Fdb\", name=self.target_table)\n",{"type":60,"tag":314,"props":1407,"children":1408},{"class":316,"line":704},[1409],{"type":60,"tag":314,"props":1410,"children":1411},{},[1412],{"type":66,"value":1389},{"type":60,"tag":314,"props":1414,"children":1415},{"class":316,"line":712},[1416],{"type":60,"tag":314,"props":1417,"children":1418},{"emptyLinePlaceholder":330},[1419],{"type":66,"value":333},{"type":60,"tag":314,"props":1421,"children":1422},{"class":316,"line":720},[1423],{"type":60,"tag":314,"props":1424,"children":1425},{},[1426],{"type":66,"value":1168},{"type":60,"tag":314,"props":1428,"children":1429},{"class":316,"line":729},[1430],{"type":60,"tag":314,"props":1431,"children":1432},{},[1433],{"type":66,"value":1176},{"type":60,"tag":268,"props":1435,"children":1436},{},[],{"type":60,"tag":137,"props":1438,"children":1440},{"id":1439},"dataset-naming-helpers",[1441],{"type":66,"value":1442},"Dataset Naming Helpers",{"type":60,"tag":69,"props":1444,"children":1445},{},[1446,1448,1455],{"type":66,"value":1447},"Use the ",{"type":60,"tag":107,"props":1449,"children":1452},{"href":1450,"rel":1451},"https:\u002F\u002Fopenlineage.io\u002Fdocs\u002Fclient\u002Fpython\u002Fbest-practices#dataset-naming-helpers",[111],[1453],{"type":66,"value":1454},"OpenLineage dataset naming helpers",{"type":66,"value":1456}," to ensure consistent naming across platforms:",{"type":60,"tag":303,"props":1458,"children":1460},{"className":305,"code":1459,"language":307,"meta":308,"style":308},"from openlineage.client.event_v2 import Dataset\n\n# Snowflake\nfrom openlineage.client.naming.snowflake import SnowflakeDatasetNaming\n\nnaming = SnowflakeDatasetNaming(\n    account_identifier=\"myorg-myaccount\",\n    database=\"mydb\",\n    schema=\"myschema\",\n    table=\"mytable\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"snowflake:\u002F\u002Fmyorg-myaccount\", name: \"mydb.myschema.mytable\"\n\n# BigQuery\nfrom openlineage.client.naming.bigquery import BigQueryDatasetNaming\n\nnaming = BigQueryDatasetNaming(\n    project=\"my-project\",\n    dataset=\"my_dataset\",\n    table=\"my_table\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"bigquery\", name: \"my-project.my_dataset.my_table\"\n\n# S3\nfrom openlineage.client.naming.s3 import S3DatasetNaming\n\nnaming = S3DatasetNaming(bucket=\"my-bucket\", key=\"path\u002Fto\u002Ffile.parquet\")\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"s3:\u002F\u002Fmy-bucket\", name: \"path\u002Fto\u002Ffile.parquet\"\n\n# PostgreSQL\nfrom openlineage.client.naming.postgres import PostgresDatasetNaming\n\nnaming = PostgresDatasetNaming(\n    host=\"localhost\",\n    port=5432,\n    database=\"mydb\",\n    schema=\"public\",\n    table=\"users\",\n)\ndataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n# -> namespace: \"postgres:\u002F\u002Flocalhost:5432\", name: \"mydb.public.users\"\n",[1461],{"type":60,"tag":75,"props":1462,"children":1463},{"__ignoreMap":308},[1464,1471,1478,1486,1494,1501,1509,1517,1525,1533,1541,1548,1556,1564,1571,1579,1587,1594,1602,1610,1618,1626,1633,1640,1648,1655,1663,1671,1678,1686,1693,1701,1708,1716,1724,1731,1739,1747,1755,1762,1770,1779,1787,1795],{"type":60,"tag":314,"props":1465,"children":1466},{"class":316,"line":317},[1467],{"type":60,"tag":314,"props":1468,"children":1469},{},[1470],{"type":66,"value":323},{"type":60,"tag":314,"props":1472,"children":1473},{"class":316,"line":326},[1474],{"type":60,"tag":314,"props":1475,"children":1476},{"emptyLinePlaceholder":330},[1477],{"type":66,"value":333},{"type":60,"tag":314,"props":1479,"children":1480},{"class":316,"line":336},[1481],{"type":60,"tag":314,"props":1482,"children":1483},{},[1484],{"type":66,"value":1485},"# Snowflake\n",{"type":60,"tag":314,"props":1487,"children":1488},{"class":316,"line":345},[1489],{"type":60,"tag":314,"props":1490,"children":1491},{},[1492],{"type":66,"value":1493},"from openlineage.client.naming.snowflake import SnowflakeDatasetNaming\n",{"type":60,"tag":314,"props":1495,"children":1496},{"class":316,"line":354},[1497],{"type":60,"tag":314,"props":1498,"children":1499},{"emptyLinePlaceholder":330},[1500],{"type":66,"value":333},{"type":60,"tag":314,"props":1502,"children":1503},{"class":316,"line":363},[1504],{"type":60,"tag":314,"props":1505,"children":1506},{},[1507],{"type":66,"value":1508},"naming = SnowflakeDatasetNaming(\n",{"type":60,"tag":314,"props":1510,"children":1511},{"class":316,"line":372},[1512],{"type":60,"tag":314,"props":1513,"children":1514},{},[1515],{"type":66,"value":1516},"    account_identifier=\"myorg-myaccount\",\n",{"type":60,"tag":314,"props":1518,"children":1519},{"class":316,"line":381},[1520],{"type":60,"tag":314,"props":1521,"children":1522},{},[1523],{"type":66,"value":1524},"    database=\"mydb\",\n",{"type":60,"tag":314,"props":1526,"children":1527},{"class":316,"line":390},[1528],{"type":60,"tag":314,"props":1529,"children":1530},{},[1531],{"type":66,"value":1532},"    schema=\"myschema\",\n",{"type":60,"tag":314,"props":1534,"children":1535},{"class":316,"line":399},[1536],{"type":60,"tag":314,"props":1537,"children":1538},{},[1539],{"type":66,"value":1540},"    table=\"mytable\",\n",{"type":60,"tag":314,"props":1542,"children":1543},{"class":316,"line":408},[1544],{"type":60,"tag":314,"props":1545,"children":1546},{},[1547],{"type":66,"value":378},{"type":60,"tag":314,"props":1549,"children":1550},{"class":316,"line":416},[1551],{"type":60,"tag":314,"props":1552,"children":1553},{},[1554],{"type":66,"value":1555},"dataset = Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n",{"type":60,"tag":314,"props":1557,"children":1558},{"class":316,"line":424},[1559],{"type":60,"tag":314,"props":1560,"children":1561},{},[1562],{"type":66,"value":1563},"# -> namespace: \"snowflake:\u002F\u002Fmyorg-myaccount\", name: \"mydb.myschema.mytable\"\n",{"type":60,"tag":314,"props":1565,"children":1566},{"class":316,"line":433},[1567],{"type":60,"tag":314,"props":1568,"children":1569},{"emptyLinePlaceholder":330},[1570],{"type":66,"value":333},{"type":60,"tag":314,"props":1572,"children":1573},{"class":316,"line":442},[1574],{"type":60,"tag":314,"props":1575,"children":1576},{},[1577],{"type":66,"value":1578},"# BigQuery\n",{"type":60,"tag":314,"props":1580,"children":1581},{"class":316,"line":451},[1582],{"type":60,"tag":314,"props":1583,"children":1584},{},[1585],{"type":66,"value":1586},"from openlineage.client.naming.bigquery import BigQueryDatasetNaming\n",{"type":60,"tag":314,"props":1588,"children":1589},{"class":316,"line":460},[1590],{"type":60,"tag":314,"props":1591,"children":1592},{"emptyLinePlaceholder":330},[1593],{"type":66,"value":333},{"type":60,"tag":314,"props":1595,"children":1596},{"class":316,"line":704},[1597],{"type":60,"tag":314,"props":1598,"children":1599},{},[1600],{"type":66,"value":1601},"naming = BigQueryDatasetNaming(\n",{"type":60,"tag":314,"props":1603,"children":1604},{"class":316,"line":712},[1605],{"type":60,"tag":314,"props":1606,"children":1607},{},[1608],{"type":66,"value":1609},"    project=\"my-project\",\n",{"type":60,"tag":314,"props":1611,"children":1612},{"class":316,"line":720},[1613],{"type":60,"tag":314,"props":1614,"children":1615},{},[1616],{"type":66,"value":1617},"    dataset=\"my_dataset\",\n",{"type":60,"tag":314,"props":1619,"children":1620},{"class":316,"line":729},[1621],{"type":60,"tag":314,"props":1622,"children":1623},{},[1624],{"type":66,"value":1625},"    table=\"my_table\",\n",{"type":60,"tag":314,"props":1627,"children":1628},{"class":316,"line":738},[1629],{"type":60,"tag":314,"props":1630,"children":1631},{},[1632],{"type":66,"value":378},{"type":60,"tag":314,"props":1634,"children":1635},{"class":316,"line":747},[1636],{"type":60,"tag":314,"props":1637,"children":1638},{},[1639],{"type":66,"value":1555},{"type":60,"tag":314,"props":1641,"children":1642},{"class":316,"line":756},[1643],{"type":60,"tag":314,"props":1644,"children":1645},{},[1646],{"type":66,"value":1647},"# -> namespace: \"bigquery\", name: \"my-project.my_dataset.my_table\"\n",{"type":60,"tag":314,"props":1649,"children":1650},{"class":316,"line":765},[1651],{"type":60,"tag":314,"props":1652,"children":1653},{"emptyLinePlaceholder":330},[1654],{"type":66,"value":333},{"type":60,"tag":314,"props":1656,"children":1657},{"class":316,"line":773},[1658],{"type":60,"tag":314,"props":1659,"children":1660},{},[1661],{"type":66,"value":1662},"# S3\n",{"type":60,"tag":314,"props":1664,"children":1665},{"class":316,"line":782},[1666],{"type":60,"tag":314,"props":1667,"children":1668},{},[1669],{"type":66,"value":1670},"from openlineage.client.naming.s3 import S3DatasetNaming\n",{"type":60,"tag":314,"props":1672,"children":1673},{"class":316,"line":791},[1674],{"type":60,"tag":314,"props":1675,"children":1676},{"emptyLinePlaceholder":330},[1677],{"type":66,"value":333},{"type":60,"tag":314,"props":1679,"children":1680},{"class":316,"line":800},[1681],{"type":60,"tag":314,"props":1682,"children":1683},{},[1684],{"type":66,"value":1685},"naming = S3DatasetNaming(bucket=\"my-bucket\", key=\"path\u002Fto\u002Ffile.parquet\")\n",{"type":60,"tag":314,"props":1687,"children":1688},{"class":316,"line":809},[1689],{"type":60,"tag":314,"props":1690,"children":1691},{},[1692],{"type":66,"value":1555},{"type":60,"tag":314,"props":1694,"children":1695},{"class":316,"line":818},[1696],{"type":60,"tag":314,"props":1697,"children":1698},{},[1699],{"type":66,"value":1700},"# -> namespace: \"s3:\u002F\u002Fmy-bucket\", name: \"path\u002Fto\u002Ffile.parquet\"\n",{"type":60,"tag":314,"props":1702,"children":1703},{"class":316,"line":827},[1704],{"type":60,"tag":314,"props":1705,"children":1706},{"emptyLinePlaceholder":330},[1707],{"type":66,"value":333},{"type":60,"tag":314,"props":1709,"children":1710},{"class":316,"line":835},[1711],{"type":60,"tag":314,"props":1712,"children":1713},{},[1714],{"type":66,"value":1715},"# PostgreSQL\n",{"type":60,"tag":314,"props":1717,"children":1718},{"class":316,"line":844},[1719],{"type":60,"tag":314,"props":1720,"children":1721},{},[1722],{"type":66,"value":1723},"from openlineage.client.naming.postgres import PostgresDatasetNaming\n",{"type":60,"tag":314,"props":1725,"children":1726},{"class":316,"line":853},[1727],{"type":60,"tag":314,"props":1728,"children":1729},{"emptyLinePlaceholder":330},[1730],{"type":66,"value":333},{"type":60,"tag":314,"props":1732,"children":1733},{"class":316,"line":862},[1734],{"type":60,"tag":314,"props":1735,"children":1736},{},[1737],{"type":66,"value":1738},"naming = PostgresDatasetNaming(\n",{"type":60,"tag":314,"props":1740,"children":1741},{"class":316,"line":871},[1742],{"type":60,"tag":314,"props":1743,"children":1744},{},[1745],{"type":66,"value":1746},"    host=\"localhost\",\n",{"type":60,"tag":314,"props":1748,"children":1749},{"class":316,"line":880},[1750],{"type":60,"tag":314,"props":1751,"children":1752},{},[1753],{"type":66,"value":1754},"    port=5432,\n",{"type":60,"tag":314,"props":1756,"children":1757},{"class":316,"line":888},[1758],{"type":60,"tag":314,"props":1759,"children":1760},{},[1761],{"type":66,"value":1524},{"type":60,"tag":314,"props":1763,"children":1764},{"class":316,"line":896},[1765],{"type":60,"tag":314,"props":1766,"children":1767},{},[1768],{"type":66,"value":1769},"    schema=\"public\",\n",{"type":60,"tag":314,"props":1771,"children":1773},{"class":316,"line":1772},41,[1774],{"type":60,"tag":314,"props":1775,"children":1776},{},[1777],{"type":66,"value":1778},"    table=\"users\",\n",{"type":60,"tag":314,"props":1780,"children":1782},{"class":316,"line":1781},42,[1783],{"type":60,"tag":314,"props":1784,"children":1785},{},[1786],{"type":66,"value":378},{"type":60,"tag":314,"props":1788,"children":1790},{"class":316,"line":1789},43,[1791],{"type":60,"tag":314,"props":1792,"children":1793},{},[1794],{"type":66,"value":1555},{"type":60,"tag":314,"props":1796,"children":1798},{"class":316,"line":1797},44,[1799],{"type":60,"tag":314,"props":1800,"children":1801},{},[1802],{"type":66,"value":1803},"# -> namespace: \"postgres:\u002F\u002Flocalhost:5432\", name: \"mydb.public.users\"\n",{"type":60,"tag":92,"props":1805,"children":1806},{},[1807],{"type":60,"tag":69,"props":1808,"children":1809},{},[1810,1814,1816,1823],{"type":60,"tag":99,"props":1811,"children":1812},{},[1813],{"type":66,"value":264},{"type":66,"value":1815}," Always use the naming helpers instead of constructing namespaces manually. If a helper is missing for your platform, check the ",{"type":60,"tag":107,"props":1817,"children":1820},{"href":1818,"rel":1819},"https:\u002F\u002Fgithub.com\u002FOpenLineage\u002FOpenLineage",[111],[1821],{"type":66,"value":1822},"OpenLineage repo",{"type":66,"value":1824}," or request it.",{"type":60,"tag":268,"props":1826,"children":1827},{},[],{"type":60,"tag":137,"props":1829,"children":1831},{"id":1830},"precedence-rules",[1832],{"type":66,"value":1833},"Precedence Rules",{"type":60,"tag":69,"props":1835,"children":1836},{},[1837],{"type":66,"value":1838},"OpenLineage uses this precedence for lineage extraction:",{"type":60,"tag":1840,"props":1841,"children":1842},"ol",{},[1843,1854,1871,1887],{"type":60,"tag":1844,"props":1845,"children":1846},"li",{},[1847,1852],{"type":60,"tag":99,"props":1848,"children":1849},{},[1850],{"type":66,"value":1851},"Custom Extractors",{"type":66,"value":1853}," (highest) - User-registered extractors",{"type":60,"tag":1844,"props":1855,"children":1856},{},[1857,1862,1864,1869],{"type":60,"tag":99,"props":1858,"children":1859},{},[1860],{"type":66,"value":1861},"OpenLineage Methods",{"type":66,"value":1863}," - ",{"type":60,"tag":75,"props":1865,"children":1867},{"className":1866},[],[1868],{"type":66,"value":184},{"type":66,"value":1870}," in operator",{"type":60,"tag":1844,"props":1872,"children":1873},{},[1874,1879,1881],{"type":60,"tag":99,"props":1875,"children":1876},{},[1877],{"type":66,"value":1878},"Hook-Level Lineage",{"type":66,"value":1880}," - Lineage collected from hooks via ",{"type":60,"tag":75,"props":1882,"children":1884},{"className":1883},[],[1885],{"type":66,"value":1886},"HookLineageCollector",{"type":60,"tag":1844,"props":1888,"children":1889},{},[1890,1895],{"type":60,"tag":99,"props":1891,"children":1892},{},[1893],{"type":66,"value":1894},"Inlets\u002FOutlets",{"type":66,"value":1896}," (lowest) - Falls back to these if nothing else extracts lineage",{"type":60,"tag":92,"props":1898,"children":1899},{},[1900],{"type":60,"tag":69,"props":1901,"children":1902},{},[1903,1907],{"type":60,"tag":99,"props":1904,"children":1905},{},[1906],{"type":66,"value":264},{"type":66,"value":1908}," If an extractor or method exists but returns no datasets, OpenLineage will check hook-level lineage, then fall back to inlets\u002Foutlets.",{"type":60,"tag":268,"props":1910,"children":1911},{},[],{"type":60,"tag":137,"props":1913,"children":1915},{"id":1914},"best-practices",[1916],{"type":66,"value":1917},"Best Practices",{"type":60,"tag":118,"props":1919,"children":1921},{"id":1920},"use-the-naming-helpers",[1922],{"type":66,"value":1923},"Use the Naming Helpers",{"type":60,"tag":69,"props":1925,"children":1926},{},[1927],{"type":66,"value":1928},"Always use OpenLineage naming helpers for consistent dataset creation:",{"type":60,"tag":303,"props":1930,"children":1932},{"className":305,"code":1931,"language":307,"meta":308,"style":308},"from openlineage.client.event_v2 import Dataset\nfrom openlineage.client.naming.snowflake import SnowflakeDatasetNaming\n\n\ndef snowflake_dataset(schema: str, table: str) -> Dataset:\n    \"\"\"Create a Snowflake Dataset using the naming helper.\"\"\"\n    naming = SnowflakeDatasetNaming(\n        account_identifier=\"mycompany\",\n        database=\"analytics\",\n        schema=schema,\n        table=table,\n    )\n    return Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n\n\n# Usage\nsource = snowflake_dataset(\"raw\", \"orders\")\ntarget = snowflake_dataset(\"staging\", \"orders_clean\")\n",[1933],{"type":60,"tag":75,"props":1934,"children":1935},{"__ignoreMap":308},[1936,1943,1950,1957,1964,1972,1980,1988,1996,2004,2012,2020,2027,2035,2042,2049,2057,2065],{"type":60,"tag":314,"props":1937,"children":1938},{"class":316,"line":317},[1939],{"type":60,"tag":314,"props":1940,"children":1941},{},[1942],{"type":66,"value":323},{"type":60,"tag":314,"props":1944,"children":1945},{"class":316,"line":326},[1946],{"type":60,"tag":314,"props":1947,"children":1948},{},[1949],{"type":66,"value":1493},{"type":60,"tag":314,"props":1951,"children":1952},{"class":316,"line":336},[1953],{"type":60,"tag":314,"props":1954,"children":1955},{"emptyLinePlaceholder":330},[1956],{"type":66,"value":333},{"type":60,"tag":314,"props":1958,"children":1959},{"class":316,"line":345},[1960],{"type":60,"tag":314,"props":1961,"children":1962},{"emptyLinePlaceholder":330},[1963],{"type":66,"value":333},{"type":60,"tag":314,"props":1965,"children":1966},{"class":316,"line":354},[1967],{"type":60,"tag":314,"props":1968,"children":1969},{},[1970],{"type":66,"value":1971},"def snowflake_dataset(schema: str, table: str) -> Dataset:\n",{"type":60,"tag":314,"props":1973,"children":1974},{"class":316,"line":363},[1975],{"type":60,"tag":314,"props":1976,"children":1977},{},[1978],{"type":66,"value":1979},"    \"\"\"Create a Snowflake Dataset using the naming helper.\"\"\"\n",{"type":60,"tag":314,"props":1981,"children":1982},{"class":316,"line":372},[1983],{"type":60,"tag":314,"props":1984,"children":1985},{},[1986],{"type":66,"value":1987},"    naming = SnowflakeDatasetNaming(\n",{"type":60,"tag":314,"props":1989,"children":1990},{"class":316,"line":381},[1991],{"type":60,"tag":314,"props":1992,"children":1993},{},[1994],{"type":66,"value":1995},"        account_identifier=\"mycompany\",\n",{"type":60,"tag":314,"props":1997,"children":1998},{"class":316,"line":390},[1999],{"type":60,"tag":314,"props":2000,"children":2001},{},[2002],{"type":66,"value":2003},"        database=\"analytics\",\n",{"type":60,"tag":314,"props":2005,"children":2006},{"class":316,"line":399},[2007],{"type":60,"tag":314,"props":2008,"children":2009},{},[2010],{"type":66,"value":2011},"        schema=schema,\n",{"type":60,"tag":314,"props":2013,"children":2014},{"class":316,"line":408},[2015],{"type":60,"tag":314,"props":2016,"children":2017},{},[2018],{"type":66,"value":2019},"        table=table,\n",{"type":60,"tag":314,"props":2021,"children":2022},{"class":316,"line":416},[2023],{"type":60,"tag":314,"props":2024,"children":2025},{},[2026],{"type":66,"value":824},{"type":60,"tag":314,"props":2028,"children":2029},{"class":316,"line":424},[2030],{"type":60,"tag":314,"props":2031,"children":2032},{},[2033],{"type":66,"value":2034},"    return Dataset(namespace=naming.get_namespace(), name=naming.get_name())\n",{"type":60,"tag":314,"props":2036,"children":2037},{"class":316,"line":433},[2038],{"type":60,"tag":314,"props":2039,"children":2040},{"emptyLinePlaceholder":330},[2041],{"type":66,"value":333},{"type":60,"tag":314,"props":2043,"children":2044},{"class":316,"line":442},[2045],{"type":60,"tag":314,"props":2046,"children":2047},{"emptyLinePlaceholder":330},[2048],{"type":66,"value":333},{"type":60,"tag":314,"props":2050,"children":2051},{"class":316,"line":451},[2052],{"type":60,"tag":314,"props":2053,"children":2054},{},[2055],{"type":66,"value":2056},"# Usage\n",{"type":60,"tag":314,"props":2058,"children":2059},{"class":316,"line":460},[2060],{"type":60,"tag":314,"props":2061,"children":2062},{},[2063],{"type":66,"value":2064},"source = snowflake_dataset(\"raw\", \"orders\")\n",{"type":60,"tag":314,"props":2066,"children":2067},{"class":316,"line":704},[2068],{"type":60,"tag":314,"props":2069,"children":2070},{},[2071],{"type":66,"value":2072},"target = snowflake_dataset(\"staging\", \"orders_clean\")\n",{"type":60,"tag":118,"props":2074,"children":2076},{"id":2075},"document-your-lineage",[2077],{"type":66,"value":2078},"Document Your Lineage",{"type":60,"tag":69,"props":2080,"children":2081},{},[2082],{"type":66,"value":2083},"Add comments explaining the data flow:",{"type":60,"tag":303,"props":2085,"children":2087},{"className":305,"code":2086,"language":307,"meta":308,"style":308},"transform = SqlOperator(\n    task_id=\"transform_orders\",\n    sql=\"...\",\n    # Lineage: Reads raw orders, joins with customers, writes to staging\n    inlets=[\n        snowflake_dataset(\"raw\", \"orders\"),\n        snowflake_dataset(\"raw\", \"customers\"),\n    ],\n    outlets=[\n        snowflake_dataset(\"staging\", \"order_details\"),\n    ],\n)\n",[2088],{"type":60,"tag":75,"props":2089,"children":2090},{"__ignoreMap":308},[2091,2099,2107,2115,2123,2131,2139,2147,2155,2163,2171,2178],{"type":60,"tag":314,"props":2092,"children":2093},{"class":316,"line":317},[2094],{"type":60,"tag":314,"props":2095,"children":2096},{},[2097],{"type":66,"value":2098},"transform = SqlOperator(\n",{"type":60,"tag":314,"props":2100,"children":2101},{"class":316,"line":326},[2102],{"type":60,"tag":314,"props":2103,"children":2104},{},[2105],{"type":66,"value":2106},"    task_id=\"transform_orders\",\n",{"type":60,"tag":314,"props":2108,"children":2109},{"class":316,"line":336},[2110],{"type":60,"tag":314,"props":2111,"children":2112},{},[2113],{"type":66,"value":2114},"    sql=\"...\",\n",{"type":60,"tag":314,"props":2116,"children":2117},{"class":316,"line":345},[2118],{"type":60,"tag":314,"props":2119,"children":2120},{},[2121],{"type":66,"value":2122},"    # Lineage: Reads raw orders, joins with customers, writes to staging\n",{"type":60,"tag":314,"props":2124,"children":2125},{"class":316,"line":354},[2126],{"type":60,"tag":314,"props":2127,"children":2128},{},[2129],{"type":66,"value":2130},"    inlets=[\n",{"type":60,"tag":314,"props":2132,"children":2133},{"class":316,"line":363},[2134],{"type":60,"tag":314,"props":2135,"children":2136},{},[2137],{"type":66,"value":2138},"        snowflake_dataset(\"raw\", \"orders\"),\n",{"type":60,"tag":314,"props":2140,"children":2141},{"class":316,"line":372},[2142],{"type":60,"tag":314,"props":2143,"children":2144},{},[2145],{"type":66,"value":2146},"        snowflake_dataset(\"raw\", \"customers\"),\n",{"type":60,"tag":314,"props":2148,"children":2149},{"class":316,"line":381},[2150],{"type":60,"tag":314,"props":2151,"children":2152},{},[2153],{"type":66,"value":2154},"    ],\n",{"type":60,"tag":314,"props":2156,"children":2157},{"class":316,"line":390},[2158],{"type":60,"tag":314,"props":2159,"children":2160},{},[2161],{"type":66,"value":2162},"    outlets=[\n",{"type":60,"tag":314,"props":2164,"children":2165},{"class":316,"line":399},[2166],{"type":60,"tag":314,"props":2167,"children":2168},{},[2169],{"type":66,"value":2170},"        snowflake_dataset(\"staging\", \"order_details\"),\n",{"type":60,"tag":314,"props":2172,"children":2173},{"class":316,"line":408},[2174],{"type":60,"tag":314,"props":2175,"children":2176},{},[2177],{"type":66,"value":2154},{"type":60,"tag":314,"props":2179,"children":2180},{"class":316,"line":416},[2181],{"type":60,"tag":314,"props":2182,"children":2183},{},[2184],{"type":66,"value":378},{"type":60,"tag":118,"props":2186,"children":2188},{"id":2187},"keep-lineage-accurate",[2189],{"type":66,"value":2190},"Keep Lineage Accurate",{"type":60,"tag":2192,"props":2193,"children":2194},"ul",{},[2195,2200,2205,2210],{"type":60,"tag":1844,"props":2196,"children":2197},{},[2198],{"type":66,"value":2199},"Update inlets\u002Foutlets when SQL queries change",{"type":60,"tag":1844,"props":2201,"children":2202},{},[2203],{"type":66,"value":2204},"Include all tables referenced in JOINs as inlets",{"type":60,"tag":1844,"props":2206,"children":2207},{},[2208],{"type":66,"value":2209},"Include all tables written to (including temp tables if relevant)",{"type":60,"tag":1844,"props":2211,"children":2212},{},[2213,2218],{"type":60,"tag":99,"props":2214,"children":2215},{},[2216],{"type":66,"value":2217},"Outlet-only and inlet-only annotations are valid.",{"type":66,"value":2219}," One-sided annotations are encouraged for lineage visibility even without a corresponding inlet or outlet in another DAG.",{"type":60,"tag":268,"props":2221,"children":2222},{},[],{"type":60,"tag":137,"props":2224,"children":2226},{"id":2225},"limitations",[2227],{"type":66,"value":2228},"Limitations",{"type":60,"tag":144,"props":2230,"children":2231},{},[2232,2248],{"type":60,"tag":148,"props":2233,"children":2234},{},[2235],{"type":60,"tag":152,"props":2236,"children":2237},{},[2238,2243],{"type":60,"tag":156,"props":2239,"children":2240},{},[2241],{"type":66,"value":2242},"Limitation",{"type":60,"tag":156,"props":2244,"children":2245},{},[2246],{"type":66,"value":2247},"Workaround",{"type":60,"tag":167,"props":2249,"children":2250},{},[2251,2264,2277,2298],{"type":60,"tag":152,"props":2252,"children":2253},{},[2254,2259],{"type":60,"tag":174,"props":2255,"children":2256},{},[2257],{"type":66,"value":2258},"Table-level only (no column lineage)",{"type":60,"tag":174,"props":2260,"children":2261},{},[2262],{"type":66,"value":2263},"Use OpenLineage methods or custom extractor",{"type":60,"tag":152,"props":2265,"children":2266},{},[2267,2272],{"type":60,"tag":174,"props":2268,"children":2269},{},[2270],{"type":66,"value":2271},"Overridden by extractors\u002Fmethods",{"type":60,"tag":174,"props":2273,"children":2274},{},[2275],{"type":66,"value":2276},"Only use for operators without extractors",{"type":60,"tag":152,"props":2278,"children":2279},{},[2280,2285],{"type":60,"tag":174,"props":2281,"children":2282},{},[2283],{"type":66,"value":2284},"Static at DAG parse time",{"type":60,"tag":174,"props":2286,"children":2287},{},[2288,2290,2296],{"type":66,"value":2289},"Set dynamically in ",{"type":60,"tag":75,"props":2291,"children":2293},{"className":2292},[],[2294],{"type":66,"value":2295},"execute()",{"type":66,"value":2297}," or use OL methods",{"type":60,"tag":152,"props":2299,"children":2300},{},[2301,2306],{"type":60,"tag":174,"props":2302,"children":2303},{},[2304],{"type":66,"value":2305},"Deferrable operators lose dynamic lineage",{"type":60,"tag":174,"props":2307,"children":2308},{},[2309,2311,2316],{"type":66,"value":2310},"Use OL methods instead; attributes set in ",{"type":60,"tag":75,"props":2312,"children":2314},{"className":2313},[],[2315],{"type":66,"value":2295},{"type":66,"value":2317}," are lost when deferring",{"type":60,"tag":268,"props":2319,"children":2320},{},[],{"type":60,"tag":137,"props":2322,"children":2324},{"id":2323},"related-skills",[2325],{"type":66,"value":2326},"Related Skills",{"type":60,"tag":2192,"props":2328,"children":2329},{},[2330,2340,2350],{"type":60,"tag":1844,"props":2331,"children":2332},{},[2333,2338],{"type":60,"tag":99,"props":2334,"children":2335},{},[2336],{"type":66,"value":2337},"creating-openlineage-extractors",{"type":66,"value":2339},": For column-level lineage or complex extraction",{"type":60,"tag":1844,"props":2341,"children":2342},{},[2343,2348],{"type":60,"tag":99,"props":2344,"children":2345},{},[2346],{"type":66,"value":2347},"tracing-upstream-lineage",{"type":66,"value":2349},": Investigate where data comes from",{"type":60,"tag":1844,"props":2351,"children":2352},{},[2353,2358],{"type":60,"tag":99,"props":2354,"children":2355},{},[2356],{"type":66,"value":2357},"tracing-downstream-lineage",{"type":66,"value":2359},": Investigate what depends on data",{"type":60,"tag":2361,"props":2362,"children":2363},"style",{},[2364],{"type":66,"value":2365},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2367,"total":844},[2368,2382,2394,2410,2423,2440,2447,2460,2475,2489,2499,2512],{"slug":18,"name":18,"fn":2369,"description":2370,"org":2371,"tags":2372,"stars":25,"repoUrl":26,"updatedAt":2381},"manage and troubleshoot Airflow via CLI","Queries, manages, and troubleshoots Apache Airflow using the `af` CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading task logs, diagnosing failures, debugging import and parse errors, checking connections, variables and pools, exploring the REST API, and monitoring health (for example \"trigger a pipeline\", \"retry a run\", \"list connections\", \"check Airflow health\", \"why did my DAG fail\"). This is the entrypoint that routes to sibling skills for authoring, testing, deploying, and migrating Airflow 2 to 3. Not for warehouse\u002FSQL analytics on Airflow metadata tables (use analyzing-data); for deep root-cause reports use debugging-dags or airflow-investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2373,2374,2377,2378],{"name":17,"slug":18,"type":15},{"name":2375,"slug":2376,"type":15},"CLI","cli",{"name":23,"slug":24,"type":15},{"name":2379,"slug":2380,"type":15},"Debugging","debugging","2026-04-06T18:01:43.992997",{"slug":2383,"name":2383,"fn":2384,"description":2385,"org":2386,"tags":2387,"stars":25,"repoUrl":26,"updatedAt":2393},"airflow-hitl","add human-in-the-loop steps to Airflow DAGs","Builds human-in-the-loop (HITL) Airflow workflows - approval gates, form input, and human-driven branching. Use when a DAG needs a human in the loop - an approval or reject step, sign-off before a task runs, a decision or approval UI, branching on a human choice, or collecting form input mid-run; also on mentions of ApprovalOperator, HITLOperator, HITLBranchOperator, HITLEntryOperator, or HITLTrigger. Requires Airflow 3.1+. Not for AI\u002FLLM task calls (see migrating-ai-sdk-to-common-ai).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2388,2389,2392],{"name":17,"slug":18,"type":15},{"name":2390,"slug":2391,"type":15},"Approvals","approvals",{"name":23,"slug":24,"type":15},"2026-04-06T18:01:46.758548",{"slug":2395,"name":2395,"fn":2396,"description":2397,"org":2398,"tags":2399,"stars":25,"repoUrl":26,"updatedAt":2409},"airflow-plugins","build Airflow UI plugins","Builds Airflow 3.1+ plugins that embed FastAPI apps, custom UI pages, React components, middleware, macros, and operator links directly into the Airflow UI. Use when building anything custom inside Airflow 3.1+ that involves Python and a browser-facing interface - creating an Airflow plugin, adding a custom UI page or nav entry, building FastAPI-backed endpoints inside Airflow, serving static assets from a plugin, embedding a React app, adding middleware to the API server, creating custom operator extra links, or calling the Airflow REST API from inside a plugin; also when AirflowPlugin, fastapi_apps, external_views, react_apps, or plugin registration come up.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2400,2401,2404,2406],{"name":17,"slug":18,"type":15},{"name":2402,"slug":2403,"type":15},"Plugin Development","plugin-development",{"name":2405,"slug":307,"type":15},"Python",{"name":2407,"slug":2408,"type":15},"UI Components","ui-components","2026-04-06T18:01:56.827891",{"slug":2411,"name":2411,"fn":2412,"description":2413,"org":2414,"tags":2415,"stars":25,"repoUrl":26,"updatedAt":2422},"airflow-state-store","persist Airflow task and asset state","Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key\u002Fvalue stores (`task_state_store`, `asset_state_store`) and the crash-safe `ResumableJobMixin`. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or \"what's new in Airflow 3.3\". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state — flag the anti-pattern and recommend task_state_store or asset_state_store instead. Requires Airflow 3.3+.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2416,2417,2418,2419],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":2420,"slug":2421,"type":15},"Operations","operations","2026-07-07T06:43:11.160671",{"slug":2424,"name":2424,"fn":2425,"description":2426,"org":2427,"tags":2428,"stars":25,"repoUrl":26,"updatedAt":2439},"analyzing-data","query data warehouses for business questions","Queries the data warehouse with SQL and answers business questions about data. Use when answering anything that needs warehouse data - counts, metrics, trends, aggregations, joins across tables, data lookups, or ad-hoc SQL analysis (for example \"who uses X\", \"how many Y\", \"show me Z\", \"find customers\", \"what is the count\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2429,2432,2435,2436],{"name":2430,"slug":2431,"type":15},"Analytics","analytics",{"name":2433,"slug":2434,"type":15},"Data Analysis","data-analysis",{"name":20,"slug":21,"type":15},{"name":2437,"slug":2438,"type":15},"SQL","sql","2026-04-06T18:01:49.599775",{"slug":4,"name":4,"fn":5,"description":6,"org":2441,"tags":2442,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2443,2444,2445,2446],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":2448,"name":2448,"fn":2449,"description":2450,"org":2451,"tags":2452,"stars":25,"repoUrl":26,"updatedAt":2459},"authoring-dags","author Airflow DAGs","Workflow and best practices for writing Apache Airflow DAGs. Use when creating a new DAG, write pipeline code, handling questions about DAG patterns and conventions or extending an existing DAG with a follow-up\u002Fdownstream task. ANY request shaped like 'add a DAG named X', 'write a pipeline', 'add a task that runs after Y', or 'extend the DAG'. For testing and debugging DAGs, see the testing-dags skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2453,2454,2455,2458],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2456,"slug":2457,"type":15},"ETL","etl",{"name":2405,"slug":307,"type":15},"2026-04-06T18:01:52.679888",{"slug":2461,"name":2461,"fn":2462,"description":2463,"org":2464,"tags":2465,"stars":25,"repoUrl":26,"updatedAt":2474},"authoring-go-sdk-tasks","implement Airflow tasks in Go","Writes Airflow task logic in Go using the Airflow Go SDK. Use when the user wants to implement Airflow tasks in Go, asks about `BundleProvider`\u002F`RegisterDags`, the `bundlev1` Registry\u002FDag interfaces, registering Go tasks (`AddTask`\u002F`AddTaskWithName`), dependency injection by parameter type (`context.Context`, `sdk.TIRunContext`, `*slog.Logger`, `sdk.Client`), or reading connections\u002Fvariables\u002FXComs from Go. This skill covers the Go-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fpacking\u002Fshipping the bundle see deploying-go-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2466,2467,2470,2471],{"name":17,"slug":18,"type":15},{"name":2468,"slug":2469,"type":15},"API Development","api-development",{"name":23,"slug":24,"type":15},{"name":2472,"slug":2473,"type":15},"Go","go","2026-07-11T05:39:13.552213",{"slug":2476,"name":2476,"fn":2477,"description":2478,"org":2479,"tags":2480,"stars":25,"repoUrl":26,"updatedAt":2488},"authoring-java-sdk-tasks","implement Airflow tasks in Java","Writes Airflow task logic in Java, Kotlin, or any JVM language using the Airflow Java SDK. Use when the user wants to implement Airflow tasks in Java\u002FJVM, asks about `@Builder.Dag`\u002F`@Builder.Task`\u002F`@Builder.XCom`, the `Task`\u002F`BundleBuilder` interfaces, reading connections\u002Fvariables\u002FXComs from Java, the JSON-to-Java type mapping, or logging from Java tasks. This skill covers the Java-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fshipping the bundle see deploying-java-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2481,2482,2485],{"name":23,"slug":24,"type":15},{"name":2483,"slug":2484,"type":15},"Engineering","engineering",{"name":2486,"slug":2487,"type":15},"Java","java","2026-07-18T05:48:13.374003",{"slug":2490,"name":2490,"fn":2491,"description":2492,"org":2493,"tags":2494,"stars":25,"repoUrl":26,"updatedAt":2498},"authoring-language-sdk-tasks","implement non-Python Airflow tasks","The language-neutral foundation for Airflow language SDKs — implement task logic in a non-Python language while the DAG stays in Python. Use when the user wants to run an Airflow task in another language (Java, Kotlin, Go, or other JVM\u002Fnative languages), asks how the Python `@task.stub` pairs with native task code, how task\u002FDAG IDs must match across the two sides, how data passes via XCom as JSON, or which language SDKs exist. This skill owns the shared Python-stub pattern and conceptual model; for a specific language's native API, build, and runtime, use that language's skill (e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2495,2496,2497],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2483,"slug":2484,"type":15},"2026-07-18T05:11:54.496539",{"slug":2500,"name":2500,"fn":2501,"description":2502,"org":2503,"tags":2504,"stars":25,"repoUrl":26,"updatedAt":2511},"blueprint","build reusable Airflow task group templates","Define reusable Airflow task group templates with Pydantic validation and compose DAGs from YAML. Use when creating blueprint templates, composing DAGs from YAML, validating configurations, or enabling no-code DAG authoring for non-engineers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2505,2506,2507,2508],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2456,"slug":2457,"type":15},{"name":2509,"slug":2510,"type":15},"Templates","templates","2026-04-06T18:01:45.361425",{"slug":2513,"name":2513,"fn":2514,"description":2515,"org":2516,"tags":2517,"stars":25,"repoUrl":26,"updatedAt":2524},"checking-freshness","check data freshness in warehouses","Quick data freshness check. Use when the user asks if data is up to date, when a table was last updated, if data is stale, or needs to verify data currency before using it.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2518,2519,2520,2523],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2521,"slug":2522,"type":15},"Data Quality","data-quality",{"name":2456,"slug":2457,"type":15},"2026-04-06T18:02:02.138565",{"items":2526,"total":844},[2527,2534,2540,2547,2554,2561,2568],{"slug":18,"name":18,"fn":2369,"description":2370,"org":2528,"tags":2529,"stars":25,"repoUrl":26,"updatedAt":2381},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2530,2531,2532,2533],{"name":17,"slug":18,"type":15},{"name":2375,"slug":2376,"type":15},{"name":23,"slug":24,"type":15},{"name":2379,"slug":2380,"type":15},{"slug":2383,"name":2383,"fn":2384,"description":2385,"org":2535,"tags":2536,"stars":25,"repoUrl":26,"updatedAt":2393},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2537,2538,2539],{"name":17,"slug":18,"type":15},{"name":2390,"slug":2391,"type":15},{"name":23,"slug":24,"type":15},{"slug":2395,"name":2395,"fn":2396,"description":2397,"org":2541,"tags":2542,"stars":25,"repoUrl":26,"updatedAt":2409},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2543,2544,2545,2546],{"name":17,"slug":18,"type":15},{"name":2402,"slug":2403,"type":15},{"name":2405,"slug":307,"type":15},{"name":2407,"slug":2408,"type":15},{"slug":2411,"name":2411,"fn":2412,"description":2413,"org":2548,"tags":2549,"stars":25,"repoUrl":26,"updatedAt":2422},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2550,2551,2552,2553],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":2420,"slug":2421,"type":15},{"slug":2424,"name":2424,"fn":2425,"description":2426,"org":2555,"tags":2556,"stars":25,"repoUrl":26,"updatedAt":2439},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2557,2558,2559,2560],{"name":2430,"slug":2431,"type":15},{"name":2433,"slug":2434,"type":15},{"name":20,"slug":21,"type":15},{"name":2437,"slug":2438,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2562,"tags":2563,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2564,2565,2566,2567],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":2448,"name":2448,"fn":2449,"description":2450,"org":2569,"tags":2570,"stars":25,"repoUrl":26,"updatedAt":2459},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2571,2572,2573,2574],{"name":17,"slug":18,"type":15},{"name":23,"slug":24,"type":15},{"name":2456,"slug":2457,"type":15},{"name":2405,"slug":307,"type":15}]