[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-astronomer-creating-openlineage-extractors":3,"mdc--y0561a-key":54,"related-org-astronomer-creating-openlineage-extractors":2776,"related-repo-astronomer-creating-openlineage-extractors":2939},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":49,"sourceUrl":52,"mdContent":53},"creating-openlineage-extractors","build custom OpenLineage extractors","Create custom OpenLineage extractors for Airflow operators. Use when the user needs lineage from unsupported or third-party operators, wants column-level lineage, or needs complex extraction logic beyond what inlets\u002Foutlets provide.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"astronomer","Astronomer","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fastronomer.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Airflow","airflow",{"name":20,"slug":21,"type":15},"Data Engineering","data-engineering",{"name":23,"slug":24,"type":15},"Python","python",412,"https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents","2026-04-06T18:02:11.624564",null,55,[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"agentic-workflow","agents","ai","ai-agents","apache-airflow","claude","cursor","dag","data-pipelines","dbt","llm","mcp","orchestrator","skills","workflow-automation","workflow-management","workflow-orchestration","workflows",{"repoUrl":26,"stars":25,"forks":29,"topics":50,"description":51},[31,32,33,34,18,35,36,37,38,21,39,40,41,42,43,44,45,46,47,48],"AI agent tooling for data engineering workflows.","https:\u002F\u002Fgithub.com\u002Fastronomer\u002Fagents\u002Ftree\u002FHEAD\u002Fskills\u002Fcreating-openlineage-extractors","---\nname: creating-openlineage-extractors\ndescription: Create custom OpenLineage extractors for Airflow operators. Use when the user needs lineage from unsupported or third-party operators, wants column-level lineage, or needs complex extraction logic beyond what inlets\u002Foutlets provide.\n---\n\n# Creating OpenLineage Extractors\n\nThis skill guides you through creating custom OpenLineage extractors to capture lineage from Airflow operators that don't have built-in support.\n\n> **Reference:** See the [OpenLineage provider developer guide](https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow-providers-openlineage\u002Fstable\u002Fguides\u002Fdeveloper.html) for the latest patterns and list of supported operators\u002Fhooks.\n\n## When to Use Each Approach\n\n| Scenario | Approach |\n|----------|----------|\n| Operator you own\u002Fmaintain | **OpenLineage Methods** (recommended, simplest) |\n| Third-party operator you can't modify | Custom Extractor |\n| Need column-level lineage | OpenLineage Methods or Custom Extractor |\n| Complex extraction logic | OpenLineage Methods or Custom Extractor |\n| Simple table-level lineage | Inlets\u002FOutlets (simplest, but lowest priority) |\n\n> **Important:** Always prefer OpenLineage methods over custom extractors when possible. Extractors are harder to write, easier to diverge from operator behavior after changes, and harder to debug.\n\n### On Astro\n\nAstro includes built-in OpenLineage integration — no additional transport configuration is needed. Lineage events are automatically collected and displayed in the Astro UI's **Lineage tab**. Custom extractors deployed to an Astro project are automatically picked up, so you only need to register them in `airflow.cfg` or via environment variable and deploy.\n\n---\n\n## Two Approaches\n\n### 1. OpenLineage Methods (Recommended)\n\nUse when you can add methods directly to your custom operator. This is the **go-to solution** for operators you own.\n\n### 2. Custom Extractors\n\nUse when you need lineage from third-party or provider operators that you **cannot modify**.\n\n---\n\n## Approach 1: OpenLineage Methods (Recommended)\n\nWhen you own the operator, add OpenLineage methods directly:\n\n```python\nfrom airflow.models import BaseOperator\n\n\nclass MyCustomOperator(BaseOperator):\n    \"\"\"Custom operator with built-in OpenLineage support.\"\"\"\n\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n        self._rows_processed = 0  # Set during execution\n\n    def execute(self, context):\n        # Do the actual work\n        self._rows_processed = self._process_data()\n        return self._rows_processed\n\n    def get_openlineage_facets_on_start(self):\n        \"\"\"Called when task starts. Return known inputs\u002Foutputs.\"\"\"\n        # Import locally to avoid circular imports\n        from openlineage.client.event_v2 import Dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.target_table)],\n        )\n\n    def get_openlineage_facets_on_complete(self, task_instance):\n        \"\"\"Called after success. Add runtime metadata.\"\"\"\n        from openlineage.client.event_v2 import Dataset\n        from openlineage.client.facet_v2 import output_statistics_output_dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fdb\",\n                    name=self.target_table,\n                    facets={\n                        \"outputStatistics\": output_statistics_output_dataset.OutputStatisticsOutputDatasetFacet(\n                            rowCount=self._rows_processed\n                        )\n                    },\n                )\n            ],\n        )\n\n    def get_openlineage_facets_on_failure(self, task_instance):\n        \"\"\"Called after failure. Optional - for partial lineage.\"\"\"\n        return None\n```\n\n### OpenLineage Methods Reference\n\n| Method | When Called | Required |\n|--------|-------------|----------|\n| `get_openlineage_facets_on_start()` | Task enters RUNNING | No |\n| `get_openlineage_facets_on_complete(ti)` | Task succeeds | No |\n| `get_openlineage_facets_on_failure(ti)` | Task fails | No |\n\n> Implement only the methods you need. Unimplemented methods fall through to Hook-Level Lineage or inlets\u002Foutlets.\n\n---\n\n## Approach 2: Custom Extractors\n\nUse this approach only when you **cannot modify** the operator (e.g., third-party or provider operators).\n\n### Basic Structure\n\n```python\nfrom airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass MyOperatorExtractor(BaseExtractor):\n    \"\"\"Extract lineage from MyCustomOperator.\"\"\"\n\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        \"\"\"Return operator class names this extractor handles.\"\"\"\n        return [\"MyCustomOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        \"\"\"Called BEFORE operator executes. Use for known inputs\u002Foutputs.\"\"\"\n        # Access operator properties via self.operator\n        source_table = self.operator.source_table\n        target_table = self.operator.target_table\n\n        return OperatorLineage(\n            inputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n                    name=f\"public.{source_table}\",\n                )\n            ],\n            outputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n                    name=f\"public.{target_table}\",\n                )\n            ],\n        )\n\n    def extract_on_complete(self, task_instance) -> OperatorLineage | None:\n        \"\"\"Called AFTER operator executes. Use for runtime-determined lineage.\"\"\"\n        # Access properties set during execution\n        # Useful for operators that determine outputs at runtime\n        return None\n```\n\n### OperatorLineage Structure\n\n```python\nfrom airflow.providers.openlineage.extractors.base import OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\nfrom openlineage.client.facet_v2 import sql_job\n\nlineage = OperatorLineage(\n    inputs=[Dataset(namespace=\"...\", name=\"...\")],      # Input datasets\n    outputs=[Dataset(namespace=\"...\", name=\"...\")],     # Output datasets\n    run_facets={\"sql\": sql_job.SQLJobFacet(query=\"SELECT...\")},  # Run metadata\n    job_facets={},                                      # Job metadata\n)\n```\n\n### Extraction Methods\n\n| Method | When Called | Use For |\n|--------|-------------|---------|\n| `_execute_extraction()` | Before operator runs | Static\u002Fknown lineage |\n| `extract_on_complete(task_instance)` | After success | Runtime-determined lineage |\n| `extract_on_failure(task_instance)` | After failure | Partial lineage on errors |\n\n### Registering Extractors\n\n**Option 1: Configuration file (`airflow.cfg`)**\n\n```ini\n[openlineage]\nextractors = mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor\n```\n\n**Option 2: Environment variable**\n\n```bash\nAIRFLOW__OPENLINEAGE__EXTRACTORS='mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor'\n```\n\n> **Important:** The path must be importable from the Airflow worker. Place extractors in your DAGs folder or installed package.\n\n---\n\n## Common Patterns\n\n### SQL Operator Extractor\n\n```python\nfrom airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\nfrom openlineage.client.facet_v2 import sql_job\n\n\nclass MySqlOperatorExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"MySqlOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        sql = self.operator.sql\n        conn_id = self.operator.conn_id\n\n        # Parse SQL to find tables (simplified example)\n        # In practice, use a SQL parser like sqlglot\n        inputs, outputs = self._parse_sql(sql)\n\n        namespace = f\"postgres:\u002F\u002F{conn_id}\"\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=namespace, name=t) for t in inputs],\n            outputs=[Dataset(namespace=namespace, name=t) for t in outputs],\n            job_facets={\n                \"sql\": sql_job.SQLJobFacet(query=sql)\n            },\n        )\n\n    def _parse_sql(self, sql: str) -> tuple[list[str], list[str]]:\n        \"\"\"Parse SQL to extract table names. Use sqlglot for real parsing.\"\"\"\n        # Simplified example - use proper SQL parser in production\n        inputs = []\n        outputs = []\n        # ... parsing logic ...\n        return inputs, outputs\n```\n\n### File Transfer Extractor\n\n```python\nfrom airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass S3ToSnowflakeExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"S3ToSnowflakeOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        s3_bucket = self.operator.s3_bucket\n        s3_key = self.operator.s3_key\n        table = self.operator.table\n        schema = self.operator.schema\n\n        return OperatorLineage(\n            inputs=[\n                Dataset(\n                    namespace=f\"s3:\u002F\u002F{s3_bucket}\",\n                    name=s3_key,\n                )\n            ],\n            outputs=[\n                Dataset(\n                    namespace=\"snowflake:\u002F\u002Fmyaccount.snowflakecomputing.com\",\n                    name=f\"{schema}.{table}\",\n                )\n            ],\n        )\n```\n\n### Dynamic Lineage from Execution\n\n```python\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass DynamicOutputExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"DynamicOutputOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        # Only inputs known before execution\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"...\", name=self.operator.source)],\n        )\n\n    def extract_on_complete(self, task_instance) -> OperatorLineage | None:\n        # Outputs determined during execution\n        # Access via operator properties set in execute()\n        outputs = self.operator.created_tables  # Set during execute()\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"...\", name=self.operator.source)],\n            outputs=[Dataset(namespace=\"...\", name=t) for t in outputs],\n        )\n```\n\n---\n\n## Common Pitfalls\n\n### 1. Circular Imports\n\n**Problem:** Importing Airflow modules at the top level causes circular imports.\n\n```python\n# ❌ BAD - can cause circular import issues\nfrom airflow.models import TaskInstance\nfrom openlineage.client.event_v2 import Dataset\n\nclass MyExtractor(BaseExtractor):\n    ...\n```\n\n```python\n# ✅ GOOD - import inside methods\nclass MyExtractor(BaseExtractor):\n    def _execute_extraction(self):\n        from openlineage.client.event_v2 import Dataset\n        # ...\n```\n\n### 2. Wrong Import Path\n\n**Problem:** Extractor path doesn't match actual module location.\n\n```bash\n# ❌ Wrong - path doesn't exist\nAIRFLOW__OPENLINEAGE__EXTRACTORS='extractors.MyExtractor'\n\n# ✅ Correct - full importable path\nAIRFLOW__OPENLINEAGE__EXTRACTORS='dags.extractors.my_extractor.MyExtractor'\n```\n\n### 3. Not Handling None\n\n**Problem:** Extraction fails when operator properties are None.\n\n```python\n# ✅ Handle optional properties\ndef _execute_extraction(self) -> OperatorLineage | None:\n    if not self.operator.source_table:\n        return None  # Skip extraction\n\n    return OperatorLineage(...)\n```\n\n---\n\n## Testing Extractors\n\n### Unit Testing\n\n```python\nimport pytest\nfrom unittest.mock import MagicMock\nfrom mypackage.extractors import MyOperatorExtractor\n\n\ndef test_extractor():\n    # Mock the operator\n    operator = MagicMock()\n    operator.source_table = \"input_table\"\n    operator.target_table = \"output_table\"\n\n    # Create extractor\n    extractor = MyOperatorExtractor(operator)\n\n    # Test extraction\n    lineage = extractor._execute_extraction()\n\n    assert len(lineage.inputs) == 1\n    assert lineage.inputs[0].name == \"input_table\"\n    assert len(lineage.outputs) == 1\n    assert lineage.outputs[0].name == \"output_table\"\n```\n\n---\n\n## Precedence Rules\n\nOpenLineage checks for lineage in this order:\n\n1. **Custom Extractors** (highest priority)\n2. **OpenLineage Methods** on operator\n3. **Hook-Level Lineage** (from `HookLineageCollector`)\n4. **Inlets\u002FOutlets** (lowest priority)\n\nIf a custom extractor exists, it overrides built-in extraction and inlets\u002Foutlets.\n\n---\n\n## Related Skills\n\n- **annotating-task-lineage**: For simple table-level lineage with inlets\u002Foutlets\n- **tracing-upstream-lineage**: Investigate data origins\n- **tracing-downstream-lineage**: Investigate data dependencies\n",{"data":55,"body":56},{"name":4,"description":6},{"type":57,"children":58},"root",[59,67,73,99,106,203,216,223,244,248,254,260,272,278,290,293,299,304,772,778,869,877,880,886,897,903,1197,1203,1288,1294,1385,1391,1406,1431,1439,1479,1491,1494,1500,1506,1778,1784,2004,2010,2186,2189,2195,2201,2211,2264,2309,2315,2324,2403,2409,2418,2472,2475,2481,2487,2657,2660,2666,2671,2722,2727,2730,2736,2770],{"type":60,"tag":61,"props":62,"children":63},"element","h1",{"id":4},[64],{"type":65,"value":66},"text","Creating OpenLineage Extractors",{"type":60,"tag":68,"props":69,"children":70},"p",{},[71],{"type":65,"value":72},"This skill guides you through creating custom OpenLineage extractors to capture lineage from Airflow operators that don't have built-in support.",{"type":60,"tag":74,"props":75,"children":76},"blockquote",{},[77],{"type":60,"tag":68,"props":78,"children":79},{},[80,86,88,97],{"type":60,"tag":81,"props":82,"children":83},"strong",{},[84],{"type":65,"value":85},"Reference:",{"type":65,"value":87}," See the ",{"type":60,"tag":89,"props":90,"children":94},"a",{"href":91,"rel":92},"https:\u002F\u002Fairflow.apache.org\u002Fdocs\u002Fapache-airflow-providers-openlineage\u002Fstable\u002Fguides\u002Fdeveloper.html",[93],"nofollow",[95],{"type":65,"value":96},"OpenLineage provider developer guide",{"type":65,"value":98}," for the latest patterns and list of supported operators\u002Fhooks.",{"type":60,"tag":100,"props":101,"children":103},"h2",{"id":102},"when-to-use-each-approach",[104],{"type":65,"value":105},"When to Use Each Approach",{"type":60,"tag":107,"props":108,"children":109},"table",{},[110,129],{"type":60,"tag":111,"props":112,"children":113},"thead",{},[114],{"type":60,"tag":115,"props":116,"children":117},"tr",{},[118,124],{"type":60,"tag":119,"props":120,"children":121},"th",{},[122],{"type":65,"value":123},"Scenario",{"type":60,"tag":119,"props":125,"children":126},{},[127],{"type":65,"value":128},"Approach",{"type":60,"tag":130,"props":131,"children":132},"tbody",{},[133,152,165,178,190],{"type":60,"tag":115,"props":134,"children":135},{},[136,142],{"type":60,"tag":137,"props":138,"children":139},"td",{},[140],{"type":65,"value":141},"Operator you own\u002Fmaintain",{"type":60,"tag":137,"props":143,"children":144},{},[145,150],{"type":60,"tag":81,"props":146,"children":147},{},[148],{"type":65,"value":149},"OpenLineage Methods",{"type":65,"value":151}," (recommended, simplest)",{"type":60,"tag":115,"props":153,"children":154},{},[155,160],{"type":60,"tag":137,"props":156,"children":157},{},[158],{"type":65,"value":159},"Third-party operator you can't modify",{"type":60,"tag":137,"props":161,"children":162},{},[163],{"type":65,"value":164},"Custom Extractor",{"type":60,"tag":115,"props":166,"children":167},{},[168,173],{"type":60,"tag":137,"props":169,"children":170},{},[171],{"type":65,"value":172},"Need column-level lineage",{"type":60,"tag":137,"props":174,"children":175},{},[176],{"type":65,"value":177},"OpenLineage Methods or Custom Extractor",{"type":60,"tag":115,"props":179,"children":180},{},[181,186],{"type":60,"tag":137,"props":182,"children":183},{},[184],{"type":65,"value":185},"Complex extraction logic",{"type":60,"tag":137,"props":187,"children":188},{},[189],{"type":65,"value":177},{"type":60,"tag":115,"props":191,"children":192},{},[193,198],{"type":60,"tag":137,"props":194,"children":195},{},[196],{"type":65,"value":197},"Simple table-level lineage",{"type":60,"tag":137,"props":199,"children":200},{},[201],{"type":65,"value":202},"Inlets\u002FOutlets (simplest, but lowest priority)",{"type":60,"tag":74,"props":204,"children":205},{},[206],{"type":60,"tag":68,"props":207,"children":208},{},[209,214],{"type":60,"tag":81,"props":210,"children":211},{},[212],{"type":65,"value":213},"Important:",{"type":65,"value":215}," Always prefer OpenLineage methods over custom extractors when possible. Extractors are harder to write, easier to diverge from operator behavior after changes, and harder to debug.",{"type":60,"tag":217,"props":218,"children":220},"h3",{"id":219},"on-astro",[221],{"type":65,"value":222},"On Astro",{"type":60,"tag":68,"props":224,"children":225},{},[226,228,233,235,242],{"type":65,"value":227},"Astro includes built-in OpenLineage integration — no additional transport configuration is needed. Lineage events are automatically collected and displayed in the Astro UI's ",{"type":60,"tag":81,"props":229,"children":230},{},[231],{"type":65,"value":232},"Lineage tab",{"type":65,"value":234},". Custom extractors deployed to an Astro project are automatically picked up, so you only need to register them in ",{"type":60,"tag":236,"props":237,"children":239},"code",{"className":238},[],[240],{"type":65,"value":241},"airflow.cfg",{"type":65,"value":243}," or via environment variable and deploy.",{"type":60,"tag":245,"props":246,"children":247},"hr",{},[],{"type":60,"tag":100,"props":249,"children":251},{"id":250},"two-approaches",[252],{"type":65,"value":253},"Two Approaches",{"type":60,"tag":217,"props":255,"children":257},{"id":256},"_1-openlineage-methods-recommended",[258],{"type":65,"value":259},"1. OpenLineage Methods (Recommended)",{"type":60,"tag":68,"props":261,"children":262},{},[263,265,270],{"type":65,"value":264},"Use when you can add methods directly to your custom operator. This is the ",{"type":60,"tag":81,"props":266,"children":267},{},[268],{"type":65,"value":269},"go-to solution",{"type":65,"value":271}," for operators you own.",{"type":60,"tag":217,"props":273,"children":275},{"id":274},"_2-custom-extractors",[276],{"type":65,"value":277},"2. Custom Extractors",{"type":60,"tag":68,"props":279,"children":280},{},[281,283,288],{"type":65,"value":282},"Use when you need lineage from third-party or provider operators that you ",{"type":60,"tag":81,"props":284,"children":285},{},[286],{"type":65,"value":287},"cannot modify",{"type":65,"value":289},".",{"type":60,"tag":245,"props":291,"children":292},{},[],{"type":60,"tag":100,"props":294,"children":296},{"id":295},"approach-1-openlineage-methods-recommended",[297],{"type":65,"value":298},"Approach 1: OpenLineage Methods (Recommended)",{"type":60,"tag":68,"props":300,"children":301},{},[302],{"type":65,"value":303},"When you own the operator, add OpenLineage methods directly:",{"type":60,"tag":305,"props":306,"children":310},"pre",{"className":307,"code":308,"language":24,"meta":309,"style":309},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from airflow.models import BaseOperator\n\n\nclass MyCustomOperator(BaseOperator):\n    \"\"\"Custom operator with built-in OpenLineage support.\"\"\"\n\n    def __init__(self, source_table: str, target_table: str, **kwargs):\n        super().__init__(**kwargs)\n        self.source_table = source_table\n        self.target_table = target_table\n        self._rows_processed = 0  # Set during execution\n\n    def execute(self, context):\n        # Do the actual work\n        self._rows_processed = self._process_data()\n        return self._rows_processed\n\n    def get_openlineage_facets_on_start(self):\n        \"\"\"Called when task starts. Return known inputs\u002Foutputs.\"\"\"\n        # Import locally to avoid circular imports\n        from openlineage.client.event_v2 import Dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.target_table)],\n        )\n\n    def get_openlineage_facets_on_complete(self, task_instance):\n        \"\"\"Called after success. Add runtime metadata.\"\"\"\n        from openlineage.client.event_v2 import Dataset\n        from openlineage.client.facet_v2 import output_statistics_output_dataset\n        from airflow.providers.openlineage.extractors import OperatorLineage\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.source_table)],\n            outputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fdb\",\n                    name=self.target_table,\n                    facets={\n                        \"outputStatistics\": output_statistics_output_dataset.OutputStatisticsOutputDatasetFacet(\n                            rowCount=self._rows_processed\n                        )\n                    },\n                )\n            ],\n        )\n\n    def get_openlineage_facets_on_failure(self, task_instance):\n        \"\"\"Called after failure. Optional - for partial lineage.\"\"\"\n        return None\n","",[311],{"type":60,"tag":236,"props":312,"children":313},{"__ignoreMap":309},[314,325,335,343,352,361,369,378,387,396,405,414,422,431,440,449,458,466,475,484,493,502,511,519,528,537,546,555,563,572,581,589,598,606,614,622,630,639,648,657,666,675,684,693,702,711,720,729,737,745,754,763],{"type":60,"tag":315,"props":316,"children":319},"span",{"class":317,"line":318},"line",1,[320],{"type":60,"tag":315,"props":321,"children":322},{},[323],{"type":65,"value":324},"from airflow.models import BaseOperator\n",{"type":60,"tag":315,"props":326,"children":328},{"class":317,"line":327},2,[329],{"type":60,"tag":315,"props":330,"children":332},{"emptyLinePlaceholder":331},true,[333],{"type":65,"value":334},"\n",{"type":60,"tag":315,"props":336,"children":338},{"class":317,"line":337},3,[339],{"type":60,"tag":315,"props":340,"children":341},{"emptyLinePlaceholder":331},[342],{"type":65,"value":334},{"type":60,"tag":315,"props":344,"children":346},{"class":317,"line":345},4,[347],{"type":60,"tag":315,"props":348,"children":349},{},[350],{"type":65,"value":351},"class MyCustomOperator(BaseOperator):\n",{"type":60,"tag":315,"props":353,"children":355},{"class":317,"line":354},5,[356],{"type":60,"tag":315,"props":357,"children":358},{},[359],{"type":65,"value":360},"    \"\"\"Custom operator with built-in OpenLineage support.\"\"\"\n",{"type":60,"tag":315,"props":362,"children":364},{"class":317,"line":363},6,[365],{"type":60,"tag":315,"props":366,"children":367},{"emptyLinePlaceholder":331},[368],{"type":65,"value":334},{"type":60,"tag":315,"props":370,"children":372},{"class":317,"line":371},7,[373],{"type":60,"tag":315,"props":374,"children":375},{},[376],{"type":65,"value":377},"    def __init__(self, source_table: str, target_table: str, **kwargs):\n",{"type":60,"tag":315,"props":379,"children":381},{"class":317,"line":380},8,[382],{"type":60,"tag":315,"props":383,"children":384},{},[385],{"type":65,"value":386},"        super().__init__(**kwargs)\n",{"type":60,"tag":315,"props":388,"children":390},{"class":317,"line":389},9,[391],{"type":60,"tag":315,"props":392,"children":393},{},[394],{"type":65,"value":395},"        self.source_table = source_table\n",{"type":60,"tag":315,"props":397,"children":399},{"class":317,"line":398},10,[400],{"type":60,"tag":315,"props":401,"children":402},{},[403],{"type":65,"value":404},"        self.target_table = target_table\n",{"type":60,"tag":315,"props":406,"children":408},{"class":317,"line":407},11,[409],{"type":60,"tag":315,"props":410,"children":411},{},[412],{"type":65,"value":413},"        self._rows_processed = 0  # Set during execution\n",{"type":60,"tag":315,"props":415,"children":417},{"class":317,"line":416},12,[418],{"type":60,"tag":315,"props":419,"children":420},{"emptyLinePlaceholder":331},[421],{"type":65,"value":334},{"type":60,"tag":315,"props":423,"children":425},{"class":317,"line":424},13,[426],{"type":60,"tag":315,"props":427,"children":428},{},[429],{"type":65,"value":430},"    def execute(self, context):\n",{"type":60,"tag":315,"props":432,"children":434},{"class":317,"line":433},14,[435],{"type":60,"tag":315,"props":436,"children":437},{},[438],{"type":65,"value":439},"        # Do the actual work\n",{"type":60,"tag":315,"props":441,"children":443},{"class":317,"line":442},15,[444],{"type":60,"tag":315,"props":445,"children":446},{},[447],{"type":65,"value":448},"        self._rows_processed = self._process_data()\n",{"type":60,"tag":315,"props":450,"children":452},{"class":317,"line":451},16,[453],{"type":60,"tag":315,"props":454,"children":455},{},[456],{"type":65,"value":457},"        return self._rows_processed\n",{"type":60,"tag":315,"props":459,"children":461},{"class":317,"line":460},17,[462],{"type":60,"tag":315,"props":463,"children":464},{"emptyLinePlaceholder":331},[465],{"type":65,"value":334},{"type":60,"tag":315,"props":467,"children":469},{"class":317,"line":468},18,[470],{"type":60,"tag":315,"props":471,"children":472},{},[473],{"type":65,"value":474},"    def get_openlineage_facets_on_start(self):\n",{"type":60,"tag":315,"props":476,"children":478},{"class":317,"line":477},19,[479],{"type":60,"tag":315,"props":480,"children":481},{},[482],{"type":65,"value":483},"        \"\"\"Called when task starts. Return known inputs\u002Foutputs.\"\"\"\n",{"type":60,"tag":315,"props":485,"children":487},{"class":317,"line":486},20,[488],{"type":60,"tag":315,"props":489,"children":490},{},[491],{"type":65,"value":492},"        # Import locally to avoid circular imports\n",{"type":60,"tag":315,"props":494,"children":496},{"class":317,"line":495},21,[497],{"type":60,"tag":315,"props":498,"children":499},{},[500],{"type":65,"value":501},"        from openlineage.client.event_v2 import Dataset\n",{"type":60,"tag":315,"props":503,"children":505},{"class":317,"line":504},22,[506],{"type":60,"tag":315,"props":507,"children":508},{},[509],{"type":65,"value":510},"        from airflow.providers.openlineage.extractors import OperatorLineage\n",{"type":60,"tag":315,"props":512,"children":514},{"class":317,"line":513},23,[515],{"type":60,"tag":315,"props":516,"children":517},{"emptyLinePlaceholder":331},[518],{"type":65,"value":334},{"type":60,"tag":315,"props":520,"children":522},{"class":317,"line":521},24,[523],{"type":60,"tag":315,"props":524,"children":525},{},[526],{"type":65,"value":527},"        return OperatorLineage(\n",{"type":60,"tag":315,"props":529,"children":531},{"class":317,"line":530},25,[532],{"type":60,"tag":315,"props":533,"children":534},{},[535],{"type":65,"value":536},"            inputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.source_table)],\n",{"type":60,"tag":315,"props":538,"children":540},{"class":317,"line":539},26,[541],{"type":60,"tag":315,"props":542,"children":543},{},[544],{"type":65,"value":545},"            outputs=[Dataset(namespace=\"postgres:\u002F\u002Fdb\", name=self.target_table)],\n",{"type":60,"tag":315,"props":547,"children":549},{"class":317,"line":548},27,[550],{"type":60,"tag":315,"props":551,"children":552},{},[553],{"type":65,"value":554},"        )\n",{"type":60,"tag":315,"props":556,"children":558},{"class":317,"line":557},28,[559],{"type":60,"tag":315,"props":560,"children":561},{"emptyLinePlaceholder":331},[562],{"type":65,"value":334},{"type":60,"tag":315,"props":564,"children":566},{"class":317,"line":565},29,[567],{"type":60,"tag":315,"props":568,"children":569},{},[570],{"type":65,"value":571},"    def get_openlineage_facets_on_complete(self, task_instance):\n",{"type":60,"tag":315,"props":573,"children":575},{"class":317,"line":574},30,[576],{"type":60,"tag":315,"props":577,"children":578},{},[579],{"type":65,"value":580},"        \"\"\"Called after success. Add runtime metadata.\"\"\"\n",{"type":60,"tag":315,"props":582,"children":584},{"class":317,"line":583},31,[585],{"type":60,"tag":315,"props":586,"children":587},{},[588],{"type":65,"value":501},{"type":60,"tag":315,"props":590,"children":592},{"class":317,"line":591},32,[593],{"type":60,"tag":315,"props":594,"children":595},{},[596],{"type":65,"value":597},"        from openlineage.client.facet_v2 import output_statistics_output_dataset\n",{"type":60,"tag":315,"props":599,"children":601},{"class":317,"line":600},33,[602],{"type":60,"tag":315,"props":603,"children":604},{},[605],{"type":65,"value":510},{"type":60,"tag":315,"props":607,"children":609},{"class":317,"line":608},34,[610],{"type":60,"tag":315,"props":611,"children":612},{"emptyLinePlaceholder":331},[613],{"type":65,"value":334},{"type":60,"tag":315,"props":615,"children":617},{"class":317,"line":616},35,[618],{"type":60,"tag":315,"props":619,"children":620},{},[621],{"type":65,"value":527},{"type":60,"tag":315,"props":623,"children":625},{"class":317,"line":624},36,[626],{"type":60,"tag":315,"props":627,"children":628},{},[629],{"type":65,"value":536},{"type":60,"tag":315,"props":631,"children":633},{"class":317,"line":632},37,[634],{"type":60,"tag":315,"props":635,"children":636},{},[637],{"type":65,"value":638},"            outputs=[\n",{"type":60,"tag":315,"props":640,"children":642},{"class":317,"line":641},38,[643],{"type":60,"tag":315,"props":644,"children":645},{},[646],{"type":65,"value":647},"                Dataset(\n",{"type":60,"tag":315,"props":649,"children":651},{"class":317,"line":650},39,[652],{"type":60,"tag":315,"props":653,"children":654},{},[655],{"type":65,"value":656},"                    namespace=\"postgres:\u002F\u002Fdb\",\n",{"type":60,"tag":315,"props":658,"children":660},{"class":317,"line":659},40,[661],{"type":60,"tag":315,"props":662,"children":663},{},[664],{"type":65,"value":665},"                    name=self.target_table,\n",{"type":60,"tag":315,"props":667,"children":669},{"class":317,"line":668},41,[670],{"type":60,"tag":315,"props":671,"children":672},{},[673],{"type":65,"value":674},"                    facets={\n",{"type":60,"tag":315,"props":676,"children":678},{"class":317,"line":677},42,[679],{"type":60,"tag":315,"props":680,"children":681},{},[682],{"type":65,"value":683},"                        \"outputStatistics\": output_statistics_output_dataset.OutputStatisticsOutputDatasetFacet(\n",{"type":60,"tag":315,"props":685,"children":687},{"class":317,"line":686},43,[688],{"type":60,"tag":315,"props":689,"children":690},{},[691],{"type":65,"value":692},"                            rowCount=self._rows_processed\n",{"type":60,"tag":315,"props":694,"children":696},{"class":317,"line":695},44,[697],{"type":60,"tag":315,"props":698,"children":699},{},[700],{"type":65,"value":701},"                        )\n",{"type":60,"tag":315,"props":703,"children":705},{"class":317,"line":704},45,[706],{"type":60,"tag":315,"props":707,"children":708},{},[709],{"type":65,"value":710},"                    },\n",{"type":60,"tag":315,"props":712,"children":714},{"class":317,"line":713},46,[715],{"type":60,"tag":315,"props":716,"children":717},{},[718],{"type":65,"value":719},"                )\n",{"type":60,"tag":315,"props":721,"children":723},{"class":317,"line":722},47,[724],{"type":60,"tag":315,"props":725,"children":726},{},[727],{"type":65,"value":728},"            ],\n",{"type":60,"tag":315,"props":730,"children":732},{"class":317,"line":731},48,[733],{"type":60,"tag":315,"props":734,"children":735},{},[736],{"type":65,"value":554},{"type":60,"tag":315,"props":738,"children":740},{"class":317,"line":739},49,[741],{"type":60,"tag":315,"props":742,"children":743},{"emptyLinePlaceholder":331},[744],{"type":65,"value":334},{"type":60,"tag":315,"props":746,"children":748},{"class":317,"line":747},50,[749],{"type":60,"tag":315,"props":750,"children":751},{},[752],{"type":65,"value":753},"    def get_openlineage_facets_on_failure(self, task_instance):\n",{"type":60,"tag":315,"props":755,"children":757},{"class":317,"line":756},51,[758],{"type":60,"tag":315,"props":759,"children":760},{},[761],{"type":65,"value":762},"        \"\"\"Called after failure. Optional - for partial lineage.\"\"\"\n",{"type":60,"tag":315,"props":764,"children":766},{"class":317,"line":765},52,[767],{"type":60,"tag":315,"props":768,"children":769},{},[770],{"type":65,"value":771},"        return None\n",{"type":60,"tag":217,"props":773,"children":775},{"id":774},"openlineage-methods-reference",[776],{"type":65,"value":777},"OpenLineage Methods Reference",{"type":60,"tag":107,"props":779,"children":780},{},[781,802],{"type":60,"tag":111,"props":782,"children":783},{},[784],{"type":60,"tag":115,"props":785,"children":786},{},[787,792,797],{"type":60,"tag":119,"props":788,"children":789},{},[790],{"type":65,"value":791},"Method",{"type":60,"tag":119,"props":793,"children":794},{},[795],{"type":65,"value":796},"When Called",{"type":60,"tag":119,"props":798,"children":799},{},[800],{"type":65,"value":801},"Required",{"type":60,"tag":130,"props":803,"children":804},{},[805,827,848],{"type":60,"tag":115,"props":806,"children":807},{},[808,817,822],{"type":60,"tag":137,"props":809,"children":810},{},[811],{"type":60,"tag":236,"props":812,"children":814},{"className":813},[],[815],{"type":65,"value":816},"get_openlineage_facets_on_start()",{"type":60,"tag":137,"props":818,"children":819},{},[820],{"type":65,"value":821},"Task enters RUNNING",{"type":60,"tag":137,"props":823,"children":824},{},[825],{"type":65,"value":826},"No",{"type":60,"tag":115,"props":828,"children":829},{},[830,839,844],{"type":60,"tag":137,"props":831,"children":832},{},[833],{"type":60,"tag":236,"props":834,"children":836},{"className":835},[],[837],{"type":65,"value":838},"get_openlineage_facets_on_complete(ti)",{"type":60,"tag":137,"props":840,"children":841},{},[842],{"type":65,"value":843},"Task succeeds",{"type":60,"tag":137,"props":845,"children":846},{},[847],{"type":65,"value":826},{"type":60,"tag":115,"props":849,"children":850},{},[851,860,865],{"type":60,"tag":137,"props":852,"children":853},{},[854],{"type":60,"tag":236,"props":855,"children":857},{"className":856},[],[858],{"type":65,"value":859},"get_openlineage_facets_on_failure(ti)",{"type":60,"tag":137,"props":861,"children":862},{},[863],{"type":65,"value":864},"Task fails",{"type":60,"tag":137,"props":866,"children":867},{},[868],{"type":65,"value":826},{"type":60,"tag":74,"props":870,"children":871},{},[872],{"type":60,"tag":68,"props":873,"children":874},{},[875],{"type":65,"value":876},"Implement only the methods you need. Unimplemented methods fall through to Hook-Level Lineage or inlets\u002Foutlets.",{"type":60,"tag":245,"props":878,"children":879},{},[],{"type":60,"tag":100,"props":881,"children":883},{"id":882},"approach-2-custom-extractors",[884],{"type":65,"value":885},"Approach 2: Custom Extractors",{"type":60,"tag":68,"props":887,"children":888},{},[889,891,895],{"type":65,"value":890},"Use this approach only when you ",{"type":60,"tag":81,"props":892,"children":893},{},[894],{"type":65,"value":287},{"type":65,"value":896}," the operator (e.g., third-party or provider operators).",{"type":60,"tag":217,"props":898,"children":900},{"id":899},"basic-structure",[901],{"type":65,"value":902},"Basic Structure",{"type":60,"tag":305,"props":904,"children":906},{"className":307,"code":905,"language":24,"meta":309,"style":309},"from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass MyOperatorExtractor(BaseExtractor):\n    \"\"\"Extract lineage from MyCustomOperator.\"\"\"\n\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        \"\"\"Return operator class names this extractor handles.\"\"\"\n        return [\"MyCustomOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        \"\"\"Called BEFORE operator executes. Use for known inputs\u002Foutputs.\"\"\"\n        # Access operator properties via self.operator\n        source_table = self.operator.source_table\n        target_table = self.operator.target_table\n\n        return OperatorLineage(\n            inputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n                    name=f\"public.{source_table}\",\n                )\n            ],\n            outputs=[\n                Dataset(\n                    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n                    name=f\"public.{target_table}\",\n                )\n            ],\n        )\n\n    def extract_on_complete(self, task_instance) -> OperatorLineage | None:\n        \"\"\"Called AFTER operator executes. Use for runtime-determined lineage.\"\"\"\n        # Access properties set during execution\n        # Useful for operators that determine outputs at runtime\n        return None\n",[907],{"type":60,"tag":236,"props":908,"children":909},{"__ignoreMap":309},[910,918,926,933,940,948,956,963,971,979,987,995,1002,1010,1018,1026,1034,1042,1049,1056,1064,1071,1079,1087,1094,1101,1108,1115,1122,1130,1137,1144,1151,1158,1166,1174,1182,1190],{"type":60,"tag":315,"props":911,"children":912},{"class":317,"line":318},[913],{"type":60,"tag":315,"props":914,"children":915},{},[916],{"type":65,"value":917},"from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\n",{"type":60,"tag":315,"props":919,"children":920},{"class":317,"line":327},[921],{"type":60,"tag":315,"props":922,"children":923},{},[924],{"type":65,"value":925},"from openlineage.client.event_v2 import Dataset\n",{"type":60,"tag":315,"props":927,"children":928},{"class":317,"line":337},[929],{"type":60,"tag":315,"props":930,"children":931},{"emptyLinePlaceholder":331},[932],{"type":65,"value":334},{"type":60,"tag":315,"props":934,"children":935},{"class":317,"line":345},[936],{"type":60,"tag":315,"props":937,"children":938},{"emptyLinePlaceholder":331},[939],{"type":65,"value":334},{"type":60,"tag":315,"props":941,"children":942},{"class":317,"line":354},[943],{"type":60,"tag":315,"props":944,"children":945},{},[946],{"type":65,"value":947},"class MyOperatorExtractor(BaseExtractor):\n",{"type":60,"tag":315,"props":949,"children":950},{"class":317,"line":363},[951],{"type":60,"tag":315,"props":952,"children":953},{},[954],{"type":65,"value":955},"    \"\"\"Extract lineage from MyCustomOperator.\"\"\"\n",{"type":60,"tag":315,"props":957,"children":958},{"class":317,"line":371},[959],{"type":60,"tag":315,"props":960,"children":961},{"emptyLinePlaceholder":331},[962],{"type":65,"value":334},{"type":60,"tag":315,"props":964,"children":965},{"class":317,"line":380},[966],{"type":60,"tag":315,"props":967,"children":968},{},[969],{"type":65,"value":970},"    @classmethod\n",{"type":60,"tag":315,"props":972,"children":973},{"class":317,"line":389},[974],{"type":60,"tag":315,"props":975,"children":976},{},[977],{"type":65,"value":978},"    def get_operator_classnames(cls) -> list[str]:\n",{"type":60,"tag":315,"props":980,"children":981},{"class":317,"line":398},[982],{"type":60,"tag":315,"props":983,"children":984},{},[985],{"type":65,"value":986},"        \"\"\"Return operator class names this extractor handles.\"\"\"\n",{"type":60,"tag":315,"props":988,"children":989},{"class":317,"line":407},[990],{"type":60,"tag":315,"props":991,"children":992},{},[993],{"type":65,"value":994},"        return [\"MyCustomOperator\"]\n",{"type":60,"tag":315,"props":996,"children":997},{"class":317,"line":416},[998],{"type":60,"tag":315,"props":999,"children":1000},{"emptyLinePlaceholder":331},[1001],{"type":65,"value":334},{"type":60,"tag":315,"props":1003,"children":1004},{"class":317,"line":424},[1005],{"type":60,"tag":315,"props":1006,"children":1007},{},[1008],{"type":65,"value":1009},"    def _execute_extraction(self) -> OperatorLineage | None:\n",{"type":60,"tag":315,"props":1011,"children":1012},{"class":317,"line":433},[1013],{"type":60,"tag":315,"props":1014,"children":1015},{},[1016],{"type":65,"value":1017},"        \"\"\"Called BEFORE operator executes. Use for known inputs\u002Foutputs.\"\"\"\n",{"type":60,"tag":315,"props":1019,"children":1020},{"class":317,"line":442},[1021],{"type":60,"tag":315,"props":1022,"children":1023},{},[1024],{"type":65,"value":1025},"        # Access operator properties via self.operator\n",{"type":60,"tag":315,"props":1027,"children":1028},{"class":317,"line":451},[1029],{"type":60,"tag":315,"props":1030,"children":1031},{},[1032],{"type":65,"value":1033},"        source_table = self.operator.source_table\n",{"type":60,"tag":315,"props":1035,"children":1036},{"class":317,"line":460},[1037],{"type":60,"tag":315,"props":1038,"children":1039},{},[1040],{"type":65,"value":1041},"        target_table = self.operator.target_table\n",{"type":60,"tag":315,"props":1043,"children":1044},{"class":317,"line":468},[1045],{"type":60,"tag":315,"props":1046,"children":1047},{"emptyLinePlaceholder":331},[1048],{"type":65,"value":334},{"type":60,"tag":315,"props":1050,"children":1051},{"class":317,"line":477},[1052],{"type":60,"tag":315,"props":1053,"children":1054},{},[1055],{"type":65,"value":527},{"type":60,"tag":315,"props":1057,"children":1058},{"class":317,"line":486},[1059],{"type":60,"tag":315,"props":1060,"children":1061},{},[1062],{"type":65,"value":1063},"            inputs=[\n",{"type":60,"tag":315,"props":1065,"children":1066},{"class":317,"line":495},[1067],{"type":60,"tag":315,"props":1068,"children":1069},{},[1070],{"type":65,"value":647},{"type":60,"tag":315,"props":1072,"children":1073},{"class":317,"line":504},[1074],{"type":60,"tag":315,"props":1075,"children":1076},{},[1077],{"type":65,"value":1078},"                    namespace=\"postgres:\u002F\u002Fmydb:5432\",\n",{"type":60,"tag":315,"props":1080,"children":1081},{"class":317,"line":513},[1082],{"type":60,"tag":315,"props":1083,"children":1084},{},[1085],{"type":65,"value":1086},"                    name=f\"public.{source_table}\",\n",{"type":60,"tag":315,"props":1088,"children":1089},{"class":317,"line":521},[1090],{"type":60,"tag":315,"props":1091,"children":1092},{},[1093],{"type":65,"value":719},{"type":60,"tag":315,"props":1095,"children":1096},{"class":317,"line":530},[1097],{"type":60,"tag":315,"props":1098,"children":1099},{},[1100],{"type":65,"value":728},{"type":60,"tag":315,"props":1102,"children":1103},{"class":317,"line":539},[1104],{"type":60,"tag":315,"props":1105,"children":1106},{},[1107],{"type":65,"value":638},{"type":60,"tag":315,"props":1109,"children":1110},{"class":317,"line":548},[1111],{"type":60,"tag":315,"props":1112,"children":1113},{},[1114],{"type":65,"value":647},{"type":60,"tag":315,"props":1116,"children":1117},{"class":317,"line":557},[1118],{"type":60,"tag":315,"props":1119,"children":1120},{},[1121],{"type":65,"value":1078},{"type":60,"tag":315,"props":1123,"children":1124},{"class":317,"line":565},[1125],{"type":60,"tag":315,"props":1126,"children":1127},{},[1128],{"type":65,"value":1129},"                    name=f\"public.{target_table}\",\n",{"type":60,"tag":315,"props":1131,"children":1132},{"class":317,"line":574},[1133],{"type":60,"tag":315,"props":1134,"children":1135},{},[1136],{"type":65,"value":719},{"type":60,"tag":315,"props":1138,"children":1139},{"class":317,"line":583},[1140],{"type":60,"tag":315,"props":1141,"children":1142},{},[1143],{"type":65,"value":728},{"type":60,"tag":315,"props":1145,"children":1146},{"class":317,"line":591},[1147],{"type":60,"tag":315,"props":1148,"children":1149},{},[1150],{"type":65,"value":554},{"type":60,"tag":315,"props":1152,"children":1153},{"class":317,"line":600},[1154],{"type":60,"tag":315,"props":1155,"children":1156},{"emptyLinePlaceholder":331},[1157],{"type":65,"value":334},{"type":60,"tag":315,"props":1159,"children":1160},{"class":317,"line":608},[1161],{"type":60,"tag":315,"props":1162,"children":1163},{},[1164],{"type":65,"value":1165},"    def extract_on_complete(self, task_instance) -> OperatorLineage | None:\n",{"type":60,"tag":315,"props":1167,"children":1168},{"class":317,"line":616},[1169],{"type":60,"tag":315,"props":1170,"children":1171},{},[1172],{"type":65,"value":1173},"        \"\"\"Called AFTER operator executes. Use for runtime-determined lineage.\"\"\"\n",{"type":60,"tag":315,"props":1175,"children":1176},{"class":317,"line":624},[1177],{"type":60,"tag":315,"props":1178,"children":1179},{},[1180],{"type":65,"value":1181},"        # Access properties set during execution\n",{"type":60,"tag":315,"props":1183,"children":1184},{"class":317,"line":632},[1185],{"type":60,"tag":315,"props":1186,"children":1187},{},[1188],{"type":65,"value":1189},"        # Useful for operators that determine outputs at runtime\n",{"type":60,"tag":315,"props":1191,"children":1192},{"class":317,"line":641},[1193],{"type":60,"tag":315,"props":1194,"children":1195},{},[1196],{"type":65,"value":771},{"type":60,"tag":217,"props":1198,"children":1200},{"id":1199},"operatorlineage-structure",[1201],{"type":65,"value":1202},"OperatorLineage Structure",{"type":60,"tag":305,"props":1204,"children":1206},{"className":307,"code":1205,"language":24,"meta":309,"style":309},"from airflow.providers.openlineage.extractors.base import OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\nfrom openlineage.client.facet_v2 import sql_job\n\nlineage = OperatorLineage(\n    inputs=[Dataset(namespace=\"...\", name=\"...\")],      # Input datasets\n    outputs=[Dataset(namespace=\"...\", name=\"...\")],     # Output datasets\n    run_facets={\"sql\": sql_job.SQLJobFacet(query=\"SELECT...\")},  # Run metadata\n    job_facets={},                                      # Job metadata\n)\n",[1207],{"type":60,"tag":236,"props":1208,"children":1209},{"__ignoreMap":309},[1210,1218,1225,1233,1240,1248,1256,1264,1272,1280],{"type":60,"tag":315,"props":1211,"children":1212},{"class":317,"line":318},[1213],{"type":60,"tag":315,"props":1214,"children":1215},{},[1216],{"type":65,"value":1217},"from airflow.providers.openlineage.extractors.base import OperatorLineage\n",{"type":60,"tag":315,"props":1219,"children":1220},{"class":317,"line":327},[1221],{"type":60,"tag":315,"props":1222,"children":1223},{},[1224],{"type":65,"value":925},{"type":60,"tag":315,"props":1226,"children":1227},{"class":317,"line":337},[1228],{"type":60,"tag":315,"props":1229,"children":1230},{},[1231],{"type":65,"value":1232},"from openlineage.client.facet_v2 import sql_job\n",{"type":60,"tag":315,"props":1234,"children":1235},{"class":317,"line":345},[1236],{"type":60,"tag":315,"props":1237,"children":1238},{"emptyLinePlaceholder":331},[1239],{"type":65,"value":334},{"type":60,"tag":315,"props":1241,"children":1242},{"class":317,"line":354},[1243],{"type":60,"tag":315,"props":1244,"children":1245},{},[1246],{"type":65,"value":1247},"lineage = OperatorLineage(\n",{"type":60,"tag":315,"props":1249,"children":1250},{"class":317,"line":363},[1251],{"type":60,"tag":315,"props":1252,"children":1253},{},[1254],{"type":65,"value":1255},"    inputs=[Dataset(namespace=\"...\", name=\"...\")],      # Input datasets\n",{"type":60,"tag":315,"props":1257,"children":1258},{"class":317,"line":371},[1259],{"type":60,"tag":315,"props":1260,"children":1261},{},[1262],{"type":65,"value":1263},"    outputs=[Dataset(namespace=\"...\", name=\"...\")],     # Output datasets\n",{"type":60,"tag":315,"props":1265,"children":1266},{"class":317,"line":380},[1267],{"type":60,"tag":315,"props":1268,"children":1269},{},[1270],{"type":65,"value":1271},"    run_facets={\"sql\": sql_job.SQLJobFacet(query=\"SELECT...\")},  # Run metadata\n",{"type":60,"tag":315,"props":1273,"children":1274},{"class":317,"line":389},[1275],{"type":60,"tag":315,"props":1276,"children":1277},{},[1278],{"type":65,"value":1279},"    job_facets={},                                      # Job metadata\n",{"type":60,"tag":315,"props":1281,"children":1282},{"class":317,"line":398},[1283],{"type":60,"tag":315,"props":1284,"children":1285},{},[1286],{"type":65,"value":1287},")\n",{"type":60,"tag":217,"props":1289,"children":1291},{"id":1290},"extraction-methods",[1292],{"type":65,"value":1293},"Extraction Methods",{"type":60,"tag":107,"props":1295,"children":1296},{},[1297,1316],{"type":60,"tag":111,"props":1298,"children":1299},{},[1300],{"type":60,"tag":115,"props":1301,"children":1302},{},[1303,1307,1311],{"type":60,"tag":119,"props":1304,"children":1305},{},[1306],{"type":65,"value":791},{"type":60,"tag":119,"props":1308,"children":1309},{},[1310],{"type":65,"value":796},{"type":60,"tag":119,"props":1312,"children":1313},{},[1314],{"type":65,"value":1315},"Use For",{"type":60,"tag":130,"props":1317,"children":1318},{},[1319,1341,1363],{"type":60,"tag":115,"props":1320,"children":1321},{},[1322,1331,1336],{"type":60,"tag":137,"props":1323,"children":1324},{},[1325],{"type":60,"tag":236,"props":1326,"children":1328},{"className":1327},[],[1329],{"type":65,"value":1330},"_execute_extraction()",{"type":60,"tag":137,"props":1332,"children":1333},{},[1334],{"type":65,"value":1335},"Before operator runs",{"type":60,"tag":137,"props":1337,"children":1338},{},[1339],{"type":65,"value":1340},"Static\u002Fknown lineage",{"type":60,"tag":115,"props":1342,"children":1343},{},[1344,1353,1358],{"type":60,"tag":137,"props":1345,"children":1346},{},[1347],{"type":60,"tag":236,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":65,"value":1352},"extract_on_complete(task_instance)",{"type":60,"tag":137,"props":1354,"children":1355},{},[1356],{"type":65,"value":1357},"After success",{"type":60,"tag":137,"props":1359,"children":1360},{},[1361],{"type":65,"value":1362},"Runtime-determined lineage",{"type":60,"tag":115,"props":1364,"children":1365},{},[1366,1375,1380],{"type":60,"tag":137,"props":1367,"children":1368},{},[1369],{"type":60,"tag":236,"props":1370,"children":1372},{"className":1371},[],[1373],{"type":65,"value":1374},"extract_on_failure(task_instance)",{"type":60,"tag":137,"props":1376,"children":1377},{},[1378],{"type":65,"value":1379},"After failure",{"type":60,"tag":137,"props":1381,"children":1382},{},[1383],{"type":65,"value":1384},"Partial lineage on errors",{"type":60,"tag":217,"props":1386,"children":1388},{"id":1387},"registering-extractors",[1389],{"type":65,"value":1390},"Registering Extractors",{"type":60,"tag":68,"props":1392,"children":1393},{},[1394],{"type":60,"tag":81,"props":1395,"children":1396},{},[1397,1399,1404],{"type":65,"value":1398},"Option 1: Configuration file (",{"type":60,"tag":236,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":65,"value":241},{"type":65,"value":1405},")",{"type":60,"tag":305,"props":1407,"children":1411},{"className":1408,"code":1409,"language":1410,"meta":309,"style":309},"language-ini shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[openlineage]\nextractors = mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor\n","ini",[1412],{"type":60,"tag":236,"props":1413,"children":1414},{"__ignoreMap":309},[1415,1423],{"type":60,"tag":315,"props":1416,"children":1417},{"class":317,"line":318},[1418],{"type":60,"tag":315,"props":1419,"children":1420},{},[1421],{"type":65,"value":1422},"[openlineage]\n",{"type":60,"tag":315,"props":1424,"children":1425},{"class":317,"line":327},[1426],{"type":60,"tag":315,"props":1427,"children":1428},{},[1429],{"type":65,"value":1430},"extractors = mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor\n",{"type":60,"tag":68,"props":1432,"children":1433},{},[1434],{"type":60,"tag":81,"props":1435,"children":1436},{},[1437],{"type":65,"value":1438},"Option 2: Environment variable",{"type":60,"tag":305,"props":1440,"children":1444},{"className":1441,"code":1442,"language":1443,"meta":309,"style":309},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","AIRFLOW__OPENLINEAGE__EXTRACTORS='mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor'\n","bash",[1445],{"type":60,"tag":236,"props":1446,"children":1447},{"__ignoreMap":309},[1448],{"type":60,"tag":315,"props":1449,"children":1450},{"class":317,"line":318},[1451,1457,1463,1468,1474],{"type":60,"tag":315,"props":1452,"children":1454},{"style":1453},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1455],{"type":65,"value":1456},"AIRFLOW__OPENLINEAGE__EXTRACTORS",{"type":60,"tag":315,"props":1458,"children":1460},{"style":1459},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1461],{"type":65,"value":1462},"=",{"type":60,"tag":315,"props":1464,"children":1465},{"style":1459},[1466],{"type":65,"value":1467},"'",{"type":60,"tag":315,"props":1469,"children":1471},{"style":1470},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1472],{"type":65,"value":1473},"mypackage.extractors.MyOperatorExtractor;mypackage.extractors.AnotherExtractor",{"type":60,"tag":315,"props":1475,"children":1476},{"style":1459},[1477],{"type":65,"value":1478},"'\n",{"type":60,"tag":74,"props":1480,"children":1481},{},[1482],{"type":60,"tag":68,"props":1483,"children":1484},{},[1485,1489],{"type":60,"tag":81,"props":1486,"children":1487},{},[1488],{"type":65,"value":213},{"type":65,"value":1490}," The path must be importable from the Airflow worker. Place extractors in your DAGs folder or installed package.",{"type":60,"tag":245,"props":1492,"children":1493},{},[],{"type":60,"tag":100,"props":1495,"children":1497},{"id":1496},"common-patterns",[1498],{"type":65,"value":1499},"Common Patterns",{"type":60,"tag":217,"props":1501,"children":1503},{"id":1502},"sql-operator-extractor",[1504],{"type":65,"value":1505},"SQL Operator Extractor",{"type":60,"tag":305,"props":1507,"children":1509},{"className":307,"code":1508,"language":24,"meta":309,"style":309},"from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\nfrom openlineage.client.facet_v2 import sql_job\n\n\nclass MySqlOperatorExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"MySqlOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        sql = self.operator.sql\n        conn_id = self.operator.conn_id\n\n        # Parse SQL to find tables (simplified example)\n        # In practice, use a SQL parser like sqlglot\n        inputs, outputs = self._parse_sql(sql)\n\n        namespace = f\"postgres:\u002F\u002F{conn_id}\"\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=namespace, name=t) for t in inputs],\n            outputs=[Dataset(namespace=namespace, name=t) for t in outputs],\n            job_facets={\n                \"sql\": sql_job.SQLJobFacet(query=sql)\n            },\n        )\n\n    def _parse_sql(self, sql: str) -> tuple[list[str], list[str]]:\n        \"\"\"Parse SQL to extract table names. Use sqlglot for real parsing.\"\"\"\n        # Simplified example - use proper SQL parser in production\n        inputs = []\n        outputs = []\n        # ... parsing logic ...\n        return inputs, outputs\n",[1510],{"type":60,"tag":236,"props":1511,"children":1512},{"__ignoreMap":309},[1513,1520,1527,1534,1541,1548,1556,1563,1570,1578,1585,1592,1600,1608,1615,1623,1631,1639,1646,1654,1661,1668,1676,1684,1692,1700,1708,1715,1722,1730,1738,1746,1754,1762,1770],{"type":60,"tag":315,"props":1514,"children":1515},{"class":317,"line":318},[1516],{"type":60,"tag":315,"props":1517,"children":1518},{},[1519],{"type":65,"value":917},{"type":60,"tag":315,"props":1521,"children":1522},{"class":317,"line":327},[1523],{"type":60,"tag":315,"props":1524,"children":1525},{},[1526],{"type":65,"value":925},{"type":60,"tag":315,"props":1528,"children":1529},{"class":317,"line":337},[1530],{"type":60,"tag":315,"props":1531,"children":1532},{},[1533],{"type":65,"value":1232},{"type":60,"tag":315,"props":1535,"children":1536},{"class":317,"line":345},[1537],{"type":60,"tag":315,"props":1538,"children":1539},{"emptyLinePlaceholder":331},[1540],{"type":65,"value":334},{"type":60,"tag":315,"props":1542,"children":1543},{"class":317,"line":354},[1544],{"type":60,"tag":315,"props":1545,"children":1546},{"emptyLinePlaceholder":331},[1547],{"type":65,"value":334},{"type":60,"tag":315,"props":1549,"children":1550},{"class":317,"line":363},[1551],{"type":60,"tag":315,"props":1552,"children":1553},{},[1554],{"type":65,"value":1555},"class MySqlOperatorExtractor(BaseExtractor):\n",{"type":60,"tag":315,"props":1557,"children":1558},{"class":317,"line":371},[1559],{"type":60,"tag":315,"props":1560,"children":1561},{},[1562],{"type":65,"value":970},{"type":60,"tag":315,"props":1564,"children":1565},{"class":317,"line":380},[1566],{"type":60,"tag":315,"props":1567,"children":1568},{},[1569],{"type":65,"value":978},{"type":60,"tag":315,"props":1571,"children":1572},{"class":317,"line":389},[1573],{"type":60,"tag":315,"props":1574,"children":1575},{},[1576],{"type":65,"value":1577},"        return [\"MySqlOperator\"]\n",{"type":60,"tag":315,"props":1579,"children":1580},{"class":317,"line":398},[1581],{"type":60,"tag":315,"props":1582,"children":1583},{"emptyLinePlaceholder":331},[1584],{"type":65,"value":334},{"type":60,"tag":315,"props":1586,"children":1587},{"class":317,"line":407},[1588],{"type":60,"tag":315,"props":1589,"children":1590},{},[1591],{"type":65,"value":1009},{"type":60,"tag":315,"props":1593,"children":1594},{"class":317,"line":416},[1595],{"type":60,"tag":315,"props":1596,"children":1597},{},[1598],{"type":65,"value":1599},"        sql = self.operator.sql\n",{"type":60,"tag":315,"props":1601,"children":1602},{"class":317,"line":424},[1603],{"type":60,"tag":315,"props":1604,"children":1605},{},[1606],{"type":65,"value":1607},"        conn_id = self.operator.conn_id\n",{"type":60,"tag":315,"props":1609,"children":1610},{"class":317,"line":433},[1611],{"type":60,"tag":315,"props":1612,"children":1613},{"emptyLinePlaceholder":331},[1614],{"type":65,"value":334},{"type":60,"tag":315,"props":1616,"children":1617},{"class":317,"line":442},[1618],{"type":60,"tag":315,"props":1619,"children":1620},{},[1621],{"type":65,"value":1622},"        # Parse SQL to find tables (simplified example)\n",{"type":60,"tag":315,"props":1624,"children":1625},{"class":317,"line":451},[1626],{"type":60,"tag":315,"props":1627,"children":1628},{},[1629],{"type":65,"value":1630},"        # In practice, use a SQL parser like sqlglot\n",{"type":60,"tag":315,"props":1632,"children":1633},{"class":317,"line":460},[1634],{"type":60,"tag":315,"props":1635,"children":1636},{},[1637],{"type":65,"value":1638},"        inputs, outputs = self._parse_sql(sql)\n",{"type":60,"tag":315,"props":1640,"children":1641},{"class":317,"line":468},[1642],{"type":60,"tag":315,"props":1643,"children":1644},{"emptyLinePlaceholder":331},[1645],{"type":65,"value":334},{"type":60,"tag":315,"props":1647,"children":1648},{"class":317,"line":477},[1649],{"type":60,"tag":315,"props":1650,"children":1651},{},[1652],{"type":65,"value":1653},"        namespace = f\"postgres:\u002F\u002F{conn_id}\"\n",{"type":60,"tag":315,"props":1655,"children":1656},{"class":317,"line":486},[1657],{"type":60,"tag":315,"props":1658,"children":1659},{"emptyLinePlaceholder":331},[1660],{"type":65,"value":334},{"type":60,"tag":315,"props":1662,"children":1663},{"class":317,"line":495},[1664],{"type":60,"tag":315,"props":1665,"children":1666},{},[1667],{"type":65,"value":527},{"type":60,"tag":315,"props":1669,"children":1670},{"class":317,"line":504},[1671],{"type":60,"tag":315,"props":1672,"children":1673},{},[1674],{"type":65,"value":1675},"            inputs=[Dataset(namespace=namespace, name=t) for t in inputs],\n",{"type":60,"tag":315,"props":1677,"children":1678},{"class":317,"line":513},[1679],{"type":60,"tag":315,"props":1680,"children":1681},{},[1682],{"type":65,"value":1683},"            outputs=[Dataset(namespace=namespace, name=t) for t in outputs],\n",{"type":60,"tag":315,"props":1685,"children":1686},{"class":317,"line":521},[1687],{"type":60,"tag":315,"props":1688,"children":1689},{},[1690],{"type":65,"value":1691},"            job_facets={\n",{"type":60,"tag":315,"props":1693,"children":1694},{"class":317,"line":530},[1695],{"type":60,"tag":315,"props":1696,"children":1697},{},[1698],{"type":65,"value":1699},"                \"sql\": sql_job.SQLJobFacet(query=sql)\n",{"type":60,"tag":315,"props":1701,"children":1702},{"class":317,"line":539},[1703],{"type":60,"tag":315,"props":1704,"children":1705},{},[1706],{"type":65,"value":1707},"            },\n",{"type":60,"tag":315,"props":1709,"children":1710},{"class":317,"line":548},[1711],{"type":60,"tag":315,"props":1712,"children":1713},{},[1714],{"type":65,"value":554},{"type":60,"tag":315,"props":1716,"children":1717},{"class":317,"line":557},[1718],{"type":60,"tag":315,"props":1719,"children":1720},{"emptyLinePlaceholder":331},[1721],{"type":65,"value":334},{"type":60,"tag":315,"props":1723,"children":1724},{"class":317,"line":565},[1725],{"type":60,"tag":315,"props":1726,"children":1727},{},[1728],{"type":65,"value":1729},"    def _parse_sql(self, sql: str) -> tuple[list[str], list[str]]:\n",{"type":60,"tag":315,"props":1731,"children":1732},{"class":317,"line":574},[1733],{"type":60,"tag":315,"props":1734,"children":1735},{},[1736],{"type":65,"value":1737},"        \"\"\"Parse SQL to extract table names. Use sqlglot for real parsing.\"\"\"\n",{"type":60,"tag":315,"props":1739,"children":1740},{"class":317,"line":583},[1741],{"type":60,"tag":315,"props":1742,"children":1743},{},[1744],{"type":65,"value":1745},"        # Simplified example - use proper SQL parser in production\n",{"type":60,"tag":315,"props":1747,"children":1748},{"class":317,"line":591},[1749],{"type":60,"tag":315,"props":1750,"children":1751},{},[1752],{"type":65,"value":1753},"        inputs = []\n",{"type":60,"tag":315,"props":1755,"children":1756},{"class":317,"line":600},[1757],{"type":60,"tag":315,"props":1758,"children":1759},{},[1760],{"type":65,"value":1761},"        outputs = []\n",{"type":60,"tag":315,"props":1763,"children":1764},{"class":317,"line":608},[1765],{"type":60,"tag":315,"props":1766,"children":1767},{},[1768],{"type":65,"value":1769},"        # ... parsing logic ...\n",{"type":60,"tag":315,"props":1771,"children":1772},{"class":317,"line":616},[1773],{"type":60,"tag":315,"props":1774,"children":1775},{},[1776],{"type":65,"value":1777},"        return inputs, outputs\n",{"type":60,"tag":217,"props":1779,"children":1781},{"id":1780},"file-transfer-extractor",[1782],{"type":65,"value":1783},"File Transfer Extractor",{"type":60,"tag":305,"props":1785,"children":1787},{"className":307,"code":1786,"language":24,"meta":309,"style":309},"from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage\nfrom openlineage.client.event_v2 import Dataset\n\n\nclass S3ToSnowflakeExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"S3ToSnowflakeOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        s3_bucket = self.operator.s3_bucket\n        s3_key = self.operator.s3_key\n        table = self.operator.table\n        schema = self.operator.schema\n\n        return OperatorLineage(\n            inputs=[\n                Dataset(\n                    namespace=f\"s3:\u002F\u002F{s3_bucket}\",\n                    name=s3_key,\n                )\n            ],\n            outputs=[\n                Dataset(\n                    namespace=\"snowflake:\u002F\u002Fmyaccount.snowflakecomputing.com\",\n                    name=f\"{schema}.{table}\",\n                )\n            ],\n        )\n",[1788],{"type":60,"tag":236,"props":1789,"children":1790},{"__ignoreMap":309},[1791,1798,1805,1812,1819,1827,1834,1841,1849,1856,1863,1871,1879,1887,1895,1902,1909,1916,1923,1931,1939,1946,1953,1960,1967,1975,1983,1990,1997],{"type":60,"tag":315,"props":1792,"children":1793},{"class":317,"line":318},[1794],{"type":60,"tag":315,"props":1795,"children":1796},{},[1797],{"type":65,"value":917},{"type":60,"tag":315,"props":1799,"children":1800},{"class":317,"line":327},[1801],{"type":60,"tag":315,"props":1802,"children":1803},{},[1804],{"type":65,"value":925},{"type":60,"tag":315,"props":1806,"children":1807},{"class":317,"line":337},[1808],{"type":60,"tag":315,"props":1809,"children":1810},{"emptyLinePlaceholder":331},[1811],{"type":65,"value":334},{"type":60,"tag":315,"props":1813,"children":1814},{"class":317,"line":345},[1815],{"type":60,"tag":315,"props":1816,"children":1817},{"emptyLinePlaceholder":331},[1818],{"type":65,"value":334},{"type":60,"tag":315,"props":1820,"children":1821},{"class":317,"line":354},[1822],{"type":60,"tag":315,"props":1823,"children":1824},{},[1825],{"type":65,"value":1826},"class S3ToSnowflakeExtractor(BaseExtractor):\n",{"type":60,"tag":315,"props":1828,"children":1829},{"class":317,"line":363},[1830],{"type":60,"tag":315,"props":1831,"children":1832},{},[1833],{"type":65,"value":970},{"type":60,"tag":315,"props":1835,"children":1836},{"class":317,"line":371},[1837],{"type":60,"tag":315,"props":1838,"children":1839},{},[1840],{"type":65,"value":978},{"type":60,"tag":315,"props":1842,"children":1843},{"class":317,"line":380},[1844],{"type":60,"tag":315,"props":1845,"children":1846},{},[1847],{"type":65,"value":1848},"        return [\"S3ToSnowflakeOperator\"]\n",{"type":60,"tag":315,"props":1850,"children":1851},{"class":317,"line":389},[1852],{"type":60,"tag":315,"props":1853,"children":1854},{"emptyLinePlaceholder":331},[1855],{"type":65,"value":334},{"type":60,"tag":315,"props":1857,"children":1858},{"class":317,"line":398},[1859],{"type":60,"tag":315,"props":1860,"children":1861},{},[1862],{"type":65,"value":1009},{"type":60,"tag":315,"props":1864,"children":1865},{"class":317,"line":407},[1866],{"type":60,"tag":315,"props":1867,"children":1868},{},[1869],{"type":65,"value":1870},"        s3_bucket = self.operator.s3_bucket\n",{"type":60,"tag":315,"props":1872,"children":1873},{"class":317,"line":416},[1874],{"type":60,"tag":315,"props":1875,"children":1876},{},[1877],{"type":65,"value":1878},"        s3_key = self.operator.s3_key\n",{"type":60,"tag":315,"props":1880,"children":1881},{"class":317,"line":424},[1882],{"type":60,"tag":315,"props":1883,"children":1884},{},[1885],{"type":65,"value":1886},"        table = self.operator.table\n",{"type":60,"tag":315,"props":1888,"children":1889},{"class":317,"line":433},[1890],{"type":60,"tag":315,"props":1891,"children":1892},{},[1893],{"type":65,"value":1894},"        schema = self.operator.schema\n",{"type":60,"tag":315,"props":1896,"children":1897},{"class":317,"line":442},[1898],{"type":60,"tag":315,"props":1899,"children":1900},{"emptyLinePlaceholder":331},[1901],{"type":65,"value":334},{"type":60,"tag":315,"props":1903,"children":1904},{"class":317,"line":451},[1905],{"type":60,"tag":315,"props":1906,"children":1907},{},[1908],{"type":65,"value":527},{"type":60,"tag":315,"props":1910,"children":1911},{"class":317,"line":460},[1912],{"type":60,"tag":315,"props":1913,"children":1914},{},[1915],{"type":65,"value":1063},{"type":60,"tag":315,"props":1917,"children":1918},{"class":317,"line":468},[1919],{"type":60,"tag":315,"props":1920,"children":1921},{},[1922],{"type":65,"value":647},{"type":60,"tag":315,"props":1924,"children":1925},{"class":317,"line":477},[1926],{"type":60,"tag":315,"props":1927,"children":1928},{},[1929],{"type":65,"value":1930},"                    namespace=f\"s3:\u002F\u002F{s3_bucket}\",\n",{"type":60,"tag":315,"props":1932,"children":1933},{"class":317,"line":486},[1934],{"type":60,"tag":315,"props":1935,"children":1936},{},[1937],{"type":65,"value":1938},"                    name=s3_key,\n",{"type":60,"tag":315,"props":1940,"children":1941},{"class":317,"line":495},[1942],{"type":60,"tag":315,"props":1943,"children":1944},{},[1945],{"type":65,"value":719},{"type":60,"tag":315,"props":1947,"children":1948},{"class":317,"line":504},[1949],{"type":60,"tag":315,"props":1950,"children":1951},{},[1952],{"type":65,"value":728},{"type":60,"tag":315,"props":1954,"children":1955},{"class":317,"line":513},[1956],{"type":60,"tag":315,"props":1957,"children":1958},{},[1959],{"type":65,"value":638},{"type":60,"tag":315,"props":1961,"children":1962},{"class":317,"line":521},[1963],{"type":60,"tag":315,"props":1964,"children":1965},{},[1966],{"type":65,"value":647},{"type":60,"tag":315,"props":1968,"children":1969},{"class":317,"line":530},[1970],{"type":60,"tag":315,"props":1971,"children":1972},{},[1973],{"type":65,"value":1974},"                    namespace=\"snowflake:\u002F\u002Fmyaccount.snowflakecomputing.com\",\n",{"type":60,"tag":315,"props":1976,"children":1977},{"class":317,"line":539},[1978],{"type":60,"tag":315,"props":1979,"children":1980},{},[1981],{"type":65,"value":1982},"                    name=f\"{schema}.{table}\",\n",{"type":60,"tag":315,"props":1984,"children":1985},{"class":317,"line":548},[1986],{"type":60,"tag":315,"props":1987,"children":1988},{},[1989],{"type":65,"value":719},{"type":60,"tag":315,"props":1991,"children":1992},{"class":317,"line":557},[1993],{"type":60,"tag":315,"props":1994,"children":1995},{},[1996],{"type":65,"value":728},{"type":60,"tag":315,"props":1998,"children":1999},{"class":317,"line":565},[2000],{"type":60,"tag":315,"props":2001,"children":2002},{},[2003],{"type":65,"value":554},{"type":60,"tag":217,"props":2005,"children":2007},{"id":2006},"dynamic-lineage-from-execution",[2008],{"type":65,"value":2009},"Dynamic Lineage from Execution",{"type":60,"tag":305,"props":2011,"children":2013},{"className":307,"code":2012,"language":24,"meta":309,"style":309},"from openlineage.client.event_v2 import Dataset\n\n\nclass DynamicOutputExtractor(BaseExtractor):\n    @classmethod\n    def get_operator_classnames(cls) -> list[str]:\n        return [\"DynamicOutputOperator\"]\n\n    def _execute_extraction(self) -> OperatorLineage | None:\n        # Only inputs known before execution\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"...\", name=self.operator.source)],\n        )\n\n    def extract_on_complete(self, task_instance) -> OperatorLineage | None:\n        # Outputs determined during execution\n        # Access via operator properties set in execute()\n        outputs = self.operator.created_tables  # Set during execute()\n\n        return OperatorLineage(\n            inputs=[Dataset(namespace=\"...\", name=self.operator.source)],\n            outputs=[Dataset(namespace=\"...\", name=t) for t in outputs],\n        )\n",[2014],{"type":60,"tag":236,"props":2015,"children":2016},{"__ignoreMap":309},[2017,2024,2031,2038,2046,2053,2060,2068,2075,2082,2090,2097,2105,2112,2119,2126,2134,2142,2150,2157,2164,2171,2179],{"type":60,"tag":315,"props":2018,"children":2019},{"class":317,"line":318},[2020],{"type":60,"tag":315,"props":2021,"children":2022},{},[2023],{"type":65,"value":925},{"type":60,"tag":315,"props":2025,"children":2026},{"class":317,"line":327},[2027],{"type":60,"tag":315,"props":2028,"children":2029},{"emptyLinePlaceholder":331},[2030],{"type":65,"value":334},{"type":60,"tag":315,"props":2032,"children":2033},{"class":317,"line":337},[2034],{"type":60,"tag":315,"props":2035,"children":2036},{"emptyLinePlaceholder":331},[2037],{"type":65,"value":334},{"type":60,"tag":315,"props":2039,"children":2040},{"class":317,"line":345},[2041],{"type":60,"tag":315,"props":2042,"children":2043},{},[2044],{"type":65,"value":2045},"class DynamicOutputExtractor(BaseExtractor):\n",{"type":60,"tag":315,"props":2047,"children":2048},{"class":317,"line":354},[2049],{"type":60,"tag":315,"props":2050,"children":2051},{},[2052],{"type":65,"value":970},{"type":60,"tag":315,"props":2054,"children":2055},{"class":317,"line":363},[2056],{"type":60,"tag":315,"props":2057,"children":2058},{},[2059],{"type":65,"value":978},{"type":60,"tag":315,"props":2061,"children":2062},{"class":317,"line":371},[2063],{"type":60,"tag":315,"props":2064,"children":2065},{},[2066],{"type":65,"value":2067},"        return [\"DynamicOutputOperator\"]\n",{"type":60,"tag":315,"props":2069,"children":2070},{"class":317,"line":380},[2071],{"type":60,"tag":315,"props":2072,"children":2073},{"emptyLinePlaceholder":331},[2074],{"type":65,"value":334},{"type":60,"tag":315,"props":2076,"children":2077},{"class":317,"line":389},[2078],{"type":60,"tag":315,"props":2079,"children":2080},{},[2081],{"type":65,"value":1009},{"type":60,"tag":315,"props":2083,"children":2084},{"class":317,"line":398},[2085],{"type":60,"tag":315,"props":2086,"children":2087},{},[2088],{"type":65,"value":2089},"        # Only inputs known before execution\n",{"type":60,"tag":315,"props":2091,"children":2092},{"class":317,"line":407},[2093],{"type":60,"tag":315,"props":2094,"children":2095},{},[2096],{"type":65,"value":527},{"type":60,"tag":315,"props":2098,"children":2099},{"class":317,"line":416},[2100],{"type":60,"tag":315,"props":2101,"children":2102},{},[2103],{"type":65,"value":2104},"            inputs=[Dataset(namespace=\"...\", name=self.operator.source)],\n",{"type":60,"tag":315,"props":2106,"children":2107},{"class":317,"line":424},[2108],{"type":60,"tag":315,"props":2109,"children":2110},{},[2111],{"type":65,"value":554},{"type":60,"tag":315,"props":2113,"children":2114},{"class":317,"line":433},[2115],{"type":60,"tag":315,"props":2116,"children":2117},{"emptyLinePlaceholder":331},[2118],{"type":65,"value":334},{"type":60,"tag":315,"props":2120,"children":2121},{"class":317,"line":442},[2122],{"type":60,"tag":315,"props":2123,"children":2124},{},[2125],{"type":65,"value":1165},{"type":60,"tag":315,"props":2127,"children":2128},{"class":317,"line":451},[2129],{"type":60,"tag":315,"props":2130,"children":2131},{},[2132],{"type":65,"value":2133},"        # Outputs determined during execution\n",{"type":60,"tag":315,"props":2135,"children":2136},{"class":317,"line":460},[2137],{"type":60,"tag":315,"props":2138,"children":2139},{},[2140],{"type":65,"value":2141},"        # Access via operator properties set in execute()\n",{"type":60,"tag":315,"props":2143,"children":2144},{"class":317,"line":468},[2145],{"type":60,"tag":315,"props":2146,"children":2147},{},[2148],{"type":65,"value":2149},"        outputs = self.operator.created_tables  # Set during execute()\n",{"type":60,"tag":315,"props":2151,"children":2152},{"class":317,"line":477},[2153],{"type":60,"tag":315,"props":2154,"children":2155},{"emptyLinePlaceholder":331},[2156],{"type":65,"value":334},{"type":60,"tag":315,"props":2158,"children":2159},{"class":317,"line":486},[2160],{"type":60,"tag":315,"props":2161,"children":2162},{},[2163],{"type":65,"value":527},{"type":60,"tag":315,"props":2165,"children":2166},{"class":317,"line":495},[2167],{"type":60,"tag":315,"props":2168,"children":2169},{},[2170],{"type":65,"value":2104},{"type":60,"tag":315,"props":2172,"children":2173},{"class":317,"line":504},[2174],{"type":60,"tag":315,"props":2175,"children":2176},{},[2177],{"type":65,"value":2178},"            outputs=[Dataset(namespace=\"...\", name=t) for t in outputs],\n",{"type":60,"tag":315,"props":2180,"children":2181},{"class":317,"line":513},[2182],{"type":60,"tag":315,"props":2183,"children":2184},{},[2185],{"type":65,"value":554},{"type":60,"tag":245,"props":2187,"children":2188},{},[],{"type":60,"tag":100,"props":2190,"children":2192},{"id":2191},"common-pitfalls",[2193],{"type":65,"value":2194},"Common Pitfalls",{"type":60,"tag":217,"props":2196,"children":2198},{"id":2197},"_1-circular-imports",[2199],{"type":65,"value":2200},"1. Circular Imports",{"type":60,"tag":68,"props":2202,"children":2203},{},[2204,2209],{"type":60,"tag":81,"props":2205,"children":2206},{},[2207],{"type":65,"value":2208},"Problem:",{"type":65,"value":2210}," Importing Airflow modules at the top level causes circular imports.",{"type":60,"tag":305,"props":2212,"children":2214},{"className":307,"code":2213,"language":24,"meta":309,"style":309},"# ❌ BAD - can cause circular import issues\nfrom airflow.models import TaskInstance\nfrom openlineage.client.event_v2 import Dataset\n\nclass MyExtractor(BaseExtractor):\n    ...\n",[2215],{"type":60,"tag":236,"props":2216,"children":2217},{"__ignoreMap":309},[2218,2226,2234,2241,2248,2256],{"type":60,"tag":315,"props":2219,"children":2220},{"class":317,"line":318},[2221],{"type":60,"tag":315,"props":2222,"children":2223},{},[2224],{"type":65,"value":2225},"# ❌ BAD - can cause circular import issues\n",{"type":60,"tag":315,"props":2227,"children":2228},{"class":317,"line":327},[2229],{"type":60,"tag":315,"props":2230,"children":2231},{},[2232],{"type":65,"value":2233},"from airflow.models import TaskInstance\n",{"type":60,"tag":315,"props":2235,"children":2236},{"class":317,"line":337},[2237],{"type":60,"tag":315,"props":2238,"children":2239},{},[2240],{"type":65,"value":925},{"type":60,"tag":315,"props":2242,"children":2243},{"class":317,"line":345},[2244],{"type":60,"tag":315,"props":2245,"children":2246},{"emptyLinePlaceholder":331},[2247],{"type":65,"value":334},{"type":60,"tag":315,"props":2249,"children":2250},{"class":317,"line":354},[2251],{"type":60,"tag":315,"props":2252,"children":2253},{},[2254],{"type":65,"value":2255},"class MyExtractor(BaseExtractor):\n",{"type":60,"tag":315,"props":2257,"children":2258},{"class":317,"line":363},[2259],{"type":60,"tag":315,"props":2260,"children":2261},{},[2262],{"type":65,"value":2263},"    ...\n",{"type":60,"tag":305,"props":2265,"children":2267},{"className":307,"code":2266,"language":24,"meta":309,"style":309},"# ✅ GOOD - import inside methods\nclass MyExtractor(BaseExtractor):\n    def _execute_extraction(self):\n        from openlineage.client.event_v2 import Dataset\n        # ...\n",[2268],{"type":60,"tag":236,"props":2269,"children":2270},{"__ignoreMap":309},[2271,2279,2286,2294,2301],{"type":60,"tag":315,"props":2272,"children":2273},{"class":317,"line":318},[2274],{"type":60,"tag":315,"props":2275,"children":2276},{},[2277],{"type":65,"value":2278},"# ✅ GOOD - import inside methods\n",{"type":60,"tag":315,"props":2280,"children":2281},{"class":317,"line":327},[2282],{"type":60,"tag":315,"props":2283,"children":2284},{},[2285],{"type":65,"value":2255},{"type":60,"tag":315,"props":2287,"children":2288},{"class":317,"line":337},[2289],{"type":60,"tag":315,"props":2290,"children":2291},{},[2292],{"type":65,"value":2293},"    def _execute_extraction(self):\n",{"type":60,"tag":315,"props":2295,"children":2296},{"class":317,"line":345},[2297],{"type":60,"tag":315,"props":2298,"children":2299},{},[2300],{"type":65,"value":501},{"type":60,"tag":315,"props":2302,"children":2303},{"class":317,"line":354},[2304],{"type":60,"tag":315,"props":2305,"children":2306},{},[2307],{"type":65,"value":2308},"        # ...\n",{"type":60,"tag":217,"props":2310,"children":2312},{"id":2311},"_2-wrong-import-path",[2313],{"type":65,"value":2314},"2. Wrong Import Path",{"type":60,"tag":68,"props":2316,"children":2317},{},[2318,2322],{"type":60,"tag":81,"props":2319,"children":2320},{},[2321],{"type":65,"value":2208},{"type":65,"value":2323}," Extractor path doesn't match actual module location.",{"type":60,"tag":305,"props":2325,"children":2327},{"className":1441,"code":2326,"language":1443,"meta":309,"style":309},"# ❌ Wrong - path doesn't exist\nAIRFLOW__OPENLINEAGE__EXTRACTORS='extractors.MyExtractor'\n\n# ✅ Correct - full importable path\nAIRFLOW__OPENLINEAGE__EXTRACTORS='dags.extractors.my_extractor.MyExtractor'\n",[2328],{"type":60,"tag":236,"props":2329,"children":2330},{"__ignoreMap":309},[2331,2340,2364,2371,2379],{"type":60,"tag":315,"props":2332,"children":2333},{"class":317,"line":318},[2334],{"type":60,"tag":315,"props":2335,"children":2337},{"style":2336},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2338],{"type":65,"value":2339},"# ❌ Wrong - path doesn't exist\n",{"type":60,"tag":315,"props":2341,"children":2342},{"class":317,"line":327},[2343,2347,2351,2355,2360],{"type":60,"tag":315,"props":2344,"children":2345},{"style":1453},[2346],{"type":65,"value":1456},{"type":60,"tag":315,"props":2348,"children":2349},{"style":1459},[2350],{"type":65,"value":1462},{"type":60,"tag":315,"props":2352,"children":2353},{"style":1459},[2354],{"type":65,"value":1467},{"type":60,"tag":315,"props":2356,"children":2357},{"style":1470},[2358],{"type":65,"value":2359},"extractors.MyExtractor",{"type":60,"tag":315,"props":2361,"children":2362},{"style":1459},[2363],{"type":65,"value":1478},{"type":60,"tag":315,"props":2365,"children":2366},{"class":317,"line":337},[2367],{"type":60,"tag":315,"props":2368,"children":2369},{"emptyLinePlaceholder":331},[2370],{"type":65,"value":334},{"type":60,"tag":315,"props":2372,"children":2373},{"class":317,"line":345},[2374],{"type":60,"tag":315,"props":2375,"children":2376},{"style":2336},[2377],{"type":65,"value":2378},"# ✅ Correct - full importable path\n",{"type":60,"tag":315,"props":2380,"children":2381},{"class":317,"line":354},[2382,2386,2390,2394,2399],{"type":60,"tag":315,"props":2383,"children":2384},{"style":1453},[2385],{"type":65,"value":1456},{"type":60,"tag":315,"props":2387,"children":2388},{"style":1459},[2389],{"type":65,"value":1462},{"type":60,"tag":315,"props":2391,"children":2392},{"style":1459},[2393],{"type":65,"value":1467},{"type":60,"tag":315,"props":2395,"children":2396},{"style":1470},[2397],{"type":65,"value":2398},"dags.extractors.my_extractor.MyExtractor",{"type":60,"tag":315,"props":2400,"children":2401},{"style":1459},[2402],{"type":65,"value":1478},{"type":60,"tag":217,"props":2404,"children":2406},{"id":2405},"_3-not-handling-none",[2407],{"type":65,"value":2408},"3. Not Handling None",{"type":60,"tag":68,"props":2410,"children":2411},{},[2412,2416],{"type":60,"tag":81,"props":2413,"children":2414},{},[2415],{"type":65,"value":2208},{"type":65,"value":2417}," Extraction fails when operator properties are None.",{"type":60,"tag":305,"props":2419,"children":2421},{"className":307,"code":2420,"language":24,"meta":309,"style":309},"# ✅ Handle optional properties\ndef _execute_extraction(self) -> OperatorLineage | None:\n    if not self.operator.source_table:\n        return None  # Skip extraction\n\n    return OperatorLineage(...)\n",[2422],{"type":60,"tag":236,"props":2423,"children":2424},{"__ignoreMap":309},[2425,2433,2441,2449,2457,2464],{"type":60,"tag":315,"props":2426,"children":2427},{"class":317,"line":318},[2428],{"type":60,"tag":315,"props":2429,"children":2430},{},[2431],{"type":65,"value":2432},"# ✅ Handle optional properties\n",{"type":60,"tag":315,"props":2434,"children":2435},{"class":317,"line":327},[2436],{"type":60,"tag":315,"props":2437,"children":2438},{},[2439],{"type":65,"value":2440},"def _execute_extraction(self) -> OperatorLineage | None:\n",{"type":60,"tag":315,"props":2442,"children":2443},{"class":317,"line":337},[2444],{"type":60,"tag":315,"props":2445,"children":2446},{},[2447],{"type":65,"value":2448},"    if not self.operator.source_table:\n",{"type":60,"tag":315,"props":2450,"children":2451},{"class":317,"line":345},[2452],{"type":60,"tag":315,"props":2453,"children":2454},{},[2455],{"type":65,"value":2456},"        return None  # Skip extraction\n",{"type":60,"tag":315,"props":2458,"children":2459},{"class":317,"line":354},[2460],{"type":60,"tag":315,"props":2461,"children":2462},{"emptyLinePlaceholder":331},[2463],{"type":65,"value":334},{"type":60,"tag":315,"props":2465,"children":2466},{"class":317,"line":363},[2467],{"type":60,"tag":315,"props":2468,"children":2469},{},[2470],{"type":65,"value":2471},"    return OperatorLineage(...)\n",{"type":60,"tag":245,"props":2473,"children":2474},{},[],{"type":60,"tag":100,"props":2476,"children":2478},{"id":2477},"testing-extractors",[2479],{"type":65,"value":2480},"Testing Extractors",{"type":60,"tag":217,"props":2482,"children":2484},{"id":2483},"unit-testing",[2485],{"type":65,"value":2486},"Unit Testing",{"type":60,"tag":305,"props":2488,"children":2490},{"className":307,"code":2489,"language":24,"meta":309,"style":309},"import pytest\nfrom unittest.mock import MagicMock\nfrom mypackage.extractors import MyOperatorExtractor\n\n\ndef test_extractor():\n    # Mock the operator\n    operator = MagicMock()\n    operator.source_table = \"input_table\"\n    operator.target_table = \"output_table\"\n\n    # Create extractor\n    extractor = MyOperatorExtractor(operator)\n\n    # Test extraction\n    lineage = extractor._execute_extraction()\n\n    assert len(lineage.inputs) == 1\n    assert lineage.inputs[0].name == \"input_table\"\n    assert len(lineage.outputs) == 1\n    assert lineage.outputs[0].name == \"output_table\"\n",[2491],{"type":60,"tag":236,"props":2492,"children":2493},{"__ignoreMap":309},[2494,2502,2510,2518,2525,2532,2540,2548,2556,2564,2572,2579,2587,2595,2602,2610,2618,2625,2633,2641,2649],{"type":60,"tag":315,"props":2495,"children":2496},{"class":317,"line":318},[2497],{"type":60,"tag":315,"props":2498,"children":2499},{},[2500],{"type":65,"value":2501},"import pytest\n",{"type":60,"tag":315,"props":2503,"children":2504},{"class":317,"line":327},[2505],{"type":60,"tag":315,"props":2506,"children":2507},{},[2508],{"type":65,"value":2509},"from unittest.mock import MagicMock\n",{"type":60,"tag":315,"props":2511,"children":2512},{"class":317,"line":337},[2513],{"type":60,"tag":315,"props":2514,"children":2515},{},[2516],{"type":65,"value":2517},"from mypackage.extractors import MyOperatorExtractor\n",{"type":60,"tag":315,"props":2519,"children":2520},{"class":317,"line":345},[2521],{"type":60,"tag":315,"props":2522,"children":2523},{"emptyLinePlaceholder":331},[2524],{"type":65,"value":334},{"type":60,"tag":315,"props":2526,"children":2527},{"class":317,"line":354},[2528],{"type":60,"tag":315,"props":2529,"children":2530},{"emptyLinePlaceholder":331},[2531],{"type":65,"value":334},{"type":60,"tag":315,"props":2533,"children":2534},{"class":317,"line":363},[2535],{"type":60,"tag":315,"props":2536,"children":2537},{},[2538],{"type":65,"value":2539},"def test_extractor():\n",{"type":60,"tag":315,"props":2541,"children":2542},{"class":317,"line":371},[2543],{"type":60,"tag":315,"props":2544,"children":2545},{},[2546],{"type":65,"value":2547},"    # Mock the operator\n",{"type":60,"tag":315,"props":2549,"children":2550},{"class":317,"line":380},[2551],{"type":60,"tag":315,"props":2552,"children":2553},{},[2554],{"type":65,"value":2555},"    operator = MagicMock()\n",{"type":60,"tag":315,"props":2557,"children":2558},{"class":317,"line":389},[2559],{"type":60,"tag":315,"props":2560,"children":2561},{},[2562],{"type":65,"value":2563},"    operator.source_table = \"input_table\"\n",{"type":60,"tag":315,"props":2565,"children":2566},{"class":317,"line":398},[2567],{"type":60,"tag":315,"props":2568,"children":2569},{},[2570],{"type":65,"value":2571},"    operator.target_table = \"output_table\"\n",{"type":60,"tag":315,"props":2573,"children":2574},{"class":317,"line":407},[2575],{"type":60,"tag":315,"props":2576,"children":2577},{"emptyLinePlaceholder":331},[2578],{"type":65,"value":334},{"type":60,"tag":315,"props":2580,"children":2581},{"class":317,"line":416},[2582],{"type":60,"tag":315,"props":2583,"children":2584},{},[2585],{"type":65,"value":2586},"    # Create extractor\n",{"type":60,"tag":315,"props":2588,"children":2589},{"class":317,"line":424},[2590],{"type":60,"tag":315,"props":2591,"children":2592},{},[2593],{"type":65,"value":2594},"    extractor = MyOperatorExtractor(operator)\n",{"type":60,"tag":315,"props":2596,"children":2597},{"class":317,"line":433},[2598],{"type":60,"tag":315,"props":2599,"children":2600},{"emptyLinePlaceholder":331},[2601],{"type":65,"value":334},{"type":60,"tag":315,"props":2603,"children":2604},{"class":317,"line":442},[2605],{"type":60,"tag":315,"props":2606,"children":2607},{},[2608],{"type":65,"value":2609},"    # Test extraction\n",{"type":60,"tag":315,"props":2611,"children":2612},{"class":317,"line":451},[2613],{"type":60,"tag":315,"props":2614,"children":2615},{},[2616],{"type":65,"value":2617},"    lineage = extractor._execute_extraction()\n",{"type":60,"tag":315,"props":2619,"children":2620},{"class":317,"line":460},[2621],{"type":60,"tag":315,"props":2622,"children":2623},{"emptyLinePlaceholder":331},[2624],{"type":65,"value":334},{"type":60,"tag":315,"props":2626,"children":2627},{"class":317,"line":468},[2628],{"type":60,"tag":315,"props":2629,"children":2630},{},[2631],{"type":65,"value":2632},"    assert len(lineage.inputs) == 1\n",{"type":60,"tag":315,"props":2634,"children":2635},{"class":317,"line":477},[2636],{"type":60,"tag":315,"props":2637,"children":2638},{},[2639],{"type":65,"value":2640},"    assert lineage.inputs[0].name == \"input_table\"\n",{"type":60,"tag":315,"props":2642,"children":2643},{"class":317,"line":486},[2644],{"type":60,"tag":315,"props":2645,"children":2646},{},[2647],{"type":65,"value":2648},"    assert len(lineage.outputs) == 1\n",{"type":60,"tag":315,"props":2650,"children":2651},{"class":317,"line":495},[2652],{"type":60,"tag":315,"props":2653,"children":2654},{},[2655],{"type":65,"value":2656},"    assert lineage.outputs[0].name == \"output_table\"\n",{"type":60,"tag":245,"props":2658,"children":2659},{},[],{"type":60,"tag":100,"props":2661,"children":2663},{"id":2662},"precedence-rules",[2664],{"type":65,"value":2665},"Precedence Rules",{"type":60,"tag":68,"props":2667,"children":2668},{},[2669],{"type":65,"value":2670},"OpenLineage checks for lineage in this order:",{"type":60,"tag":2672,"props":2673,"children":2674},"ol",{},[2675,2686,2695,2712],{"type":60,"tag":2676,"props":2677,"children":2678},"li",{},[2679,2684],{"type":60,"tag":81,"props":2680,"children":2681},{},[2682],{"type":65,"value":2683},"Custom Extractors",{"type":65,"value":2685}," (highest priority)",{"type":60,"tag":2676,"props":2687,"children":2688},{},[2689,2693],{"type":60,"tag":81,"props":2690,"children":2691},{},[2692],{"type":65,"value":149},{"type":65,"value":2694}," on operator",{"type":60,"tag":2676,"props":2696,"children":2697},{},[2698,2703,2705,2711],{"type":60,"tag":81,"props":2699,"children":2700},{},[2701],{"type":65,"value":2702},"Hook-Level Lineage",{"type":65,"value":2704}," (from ",{"type":60,"tag":236,"props":2706,"children":2708},{"className":2707},[],[2709],{"type":65,"value":2710},"HookLineageCollector",{"type":65,"value":1405},{"type":60,"tag":2676,"props":2713,"children":2714},{},[2715,2720],{"type":60,"tag":81,"props":2716,"children":2717},{},[2718],{"type":65,"value":2719},"Inlets\u002FOutlets",{"type":65,"value":2721}," (lowest priority)",{"type":60,"tag":68,"props":2723,"children":2724},{},[2725],{"type":65,"value":2726},"If a custom extractor exists, it overrides built-in extraction and inlets\u002Foutlets.",{"type":60,"tag":245,"props":2728,"children":2729},{},[],{"type":60,"tag":100,"props":2731,"children":2733},{"id":2732},"related-skills",[2734],{"type":65,"value":2735},"Related Skills",{"type":60,"tag":2737,"props":2738,"children":2739},"ul",{},[2740,2750,2760],{"type":60,"tag":2676,"props":2741,"children":2742},{},[2743,2748],{"type":60,"tag":81,"props":2744,"children":2745},{},[2746],{"type":65,"value":2747},"annotating-task-lineage",{"type":65,"value":2749},": For simple table-level lineage with inlets\u002Foutlets",{"type":60,"tag":2676,"props":2751,"children":2752},{},[2753,2758],{"type":60,"tag":81,"props":2754,"children":2755},{},[2756],{"type":65,"value":2757},"tracing-upstream-lineage",{"type":65,"value":2759},": Investigate data origins",{"type":60,"tag":2676,"props":2761,"children":2762},{},[2763,2768],{"type":60,"tag":81,"props":2764,"children":2765},{},[2766],{"type":65,"value":2767},"tracing-downstream-lineage",{"type":65,"value":2769},": Investigate data dependencies",{"type":60,"tag":2771,"props":2772,"children":2773},"style",{},[2774],{"type":65,"value":2775},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2777,"total":608},[2778,2794,2806,2821,2834,2851,2861,2874,2889,2903,2913,2926],{"slug":18,"name":18,"fn":2779,"description":2780,"org":2781,"tags":2782,"stars":25,"repoUrl":26,"updatedAt":2793},"manage and troubleshoot Airflow via CLI","Queries, manages, and troubleshoots Apache Airflow using the `af` CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading task logs, diagnosing failures, debugging import and parse errors, checking connections, variables and pools, exploring the REST API, and monitoring health (for example \"trigger a pipeline\", \"retry a run\", \"list connections\", \"check Airflow health\", \"why did my DAG fail\"). This is the entrypoint that routes to sibling skills for authoring, testing, deploying, and migrating Airflow 2 to 3. Not for warehouse\u002FSQL analytics on Airflow metadata tables (use analyzing-data); for deep root-cause reports use debugging-dags or airflow-investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2783,2784,2787,2790],{"name":17,"slug":18,"type":15},{"name":2785,"slug":2786,"type":15},"CLI","cli",{"name":2788,"slug":2789,"type":15},"Data Pipeline","data-pipeline",{"name":2791,"slug":2792,"type":15},"Debugging","debugging","2026-04-06T18:01:43.992997",{"slug":2795,"name":2795,"fn":2796,"description":2797,"org":2798,"tags":2799,"stars":25,"repoUrl":26,"updatedAt":2805},"airflow-hitl","add human-in-the-loop steps to Airflow DAGs","Builds human-in-the-loop (HITL) Airflow workflows - approval gates, form input, and human-driven branching. Use when a DAG needs a human in the loop - an approval or reject step, sign-off before a task runs, a decision or approval UI, branching on a human choice, or collecting form input mid-run; also on mentions of ApprovalOperator, HITLOperator, HITLBranchOperator, HITLEntryOperator, or HITLTrigger. Requires Airflow 3.1+. Not for AI\u002FLLM task calls (see migrating-ai-sdk-to-common-ai).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2800,2801,2804],{"name":17,"slug":18,"type":15},{"name":2802,"slug":2803,"type":15},"Approvals","approvals",{"name":2788,"slug":2789,"type":15},"2026-04-06T18:01:46.758548",{"slug":2807,"name":2807,"fn":2808,"description":2809,"org":2810,"tags":2811,"stars":25,"repoUrl":26,"updatedAt":2820},"airflow-plugins","build Airflow UI plugins","Builds Airflow 3.1+ plugins that embed FastAPI apps, custom UI pages, React components, middleware, macros, and operator links directly into the Airflow UI. Use when building anything custom inside Airflow 3.1+ that involves Python and a browser-facing interface - creating an Airflow plugin, adding a custom UI page or nav entry, building FastAPI-backed endpoints inside Airflow, serving static assets from a plugin, embedding a React app, adding middleware to the API server, creating custom operator extra links, or calling the Airflow REST API from inside a plugin; also when AirflowPlugin, fastapi_apps, external_views, react_apps, or plugin registration come up.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2812,2813,2816,2817],{"name":17,"slug":18,"type":15},{"name":2814,"slug":2815,"type":15},"Plugin Development","plugin-development",{"name":23,"slug":24,"type":15},{"name":2818,"slug":2819,"type":15},"UI Components","ui-components","2026-04-06T18:01:56.827891",{"slug":2822,"name":2822,"fn":2823,"description":2824,"org":2825,"tags":2826,"stars":25,"repoUrl":26,"updatedAt":2833},"airflow-state-store","persist Airflow task and asset state","Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key\u002Fvalue stores (`task_state_store`, `asset_state_store`) and the crash-safe `ResumableJobMixin`. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or \"what's new in Airflow 3.3\". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state — flag the anti-pattern and recommend task_state_store or asset_state_store instead. Requires Airflow 3.3+.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2827,2828,2829,2830],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2831,"slug":2832,"type":15},"Operations","operations","2026-07-07T06:43:11.160671",{"slug":2835,"name":2835,"fn":2836,"description":2837,"org":2838,"tags":2839,"stars":25,"repoUrl":26,"updatedAt":2850},"analyzing-data","query data warehouses for business questions","Queries the data warehouse with SQL and answers business questions about data. Use when answering anything that needs warehouse data - counts, metrics, trends, aggregations, joins across tables, data lookups, or ad-hoc SQL analysis (for example \"who uses X\", \"how many Y\", \"show me Z\", \"find customers\", \"what is the count\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2840,2843,2846,2847],{"name":2841,"slug":2842,"type":15},"Analytics","analytics",{"name":2844,"slug":2845,"type":15},"Data Analysis","data-analysis",{"name":20,"slug":21,"type":15},{"name":2848,"slug":2849,"type":15},"SQL","sql","2026-04-06T18:01:49.599775",{"slug":2747,"name":2747,"fn":2852,"description":2853,"org":2854,"tags":2855,"stars":25,"repoUrl":26,"updatedAt":2860},"annotate Airflow tasks with data lineage","Annotate Airflow tasks with data lineage using inlets and outlets. Use when the user wants to add lineage metadata to tasks, specify input\u002Foutput datasets, or enable lineage tracking for operators without built-in OpenLineage extraction.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2856,2857,2858,2859],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2788,"slug":2789,"type":15},{"name":13,"slug":14,"type":15},"2026-04-06T18:02:03.487365",{"slug":2862,"name":2862,"fn":2863,"description":2864,"org":2865,"tags":2866,"stars":25,"repoUrl":26,"updatedAt":2873},"authoring-dags","author Airflow DAGs","Workflow and best practices for writing Apache Airflow DAGs. Use when creating a new DAG, write pipeline code, handling questions about DAG patterns and conventions or extending an existing DAG with a follow-up\u002Fdownstream task. ANY request shaped like 'add a DAG named X', 'write a pipeline', 'add a task that runs after Y', or 'extend the DAG'. For testing and debugging DAGs, see the testing-dags skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2867,2868,2869,2872],{"name":17,"slug":18,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2870,"slug":2871,"type":15},"ETL","etl",{"name":23,"slug":24,"type":15},"2026-04-06T18:01:52.679888",{"slug":2875,"name":2875,"fn":2876,"description":2877,"org":2878,"tags":2879,"stars":25,"repoUrl":26,"updatedAt":2888},"authoring-go-sdk-tasks","implement Airflow tasks in Go","Writes Airflow task logic in Go using the Airflow Go SDK. Use when the user wants to implement Airflow tasks in Go, asks about `BundleProvider`\u002F`RegisterDags`, the `bundlev1` Registry\u002FDag interfaces, registering Go tasks (`AddTask`\u002F`AddTaskWithName`), dependency injection by parameter type (`context.Context`, `sdk.TIRunContext`, `*slog.Logger`, `sdk.Client`), or reading connections\u002Fvariables\u002FXComs from Go. This skill covers the Go-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fpacking\u002Fshipping the bundle see deploying-go-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2880,2881,2884,2885],{"name":17,"slug":18,"type":15},{"name":2882,"slug":2883,"type":15},"API Development","api-development",{"name":2788,"slug":2789,"type":15},{"name":2886,"slug":2887,"type":15},"Go","go","2026-07-11T05:39:13.552213",{"slug":2890,"name":2890,"fn":2891,"description":2892,"org":2893,"tags":2894,"stars":25,"repoUrl":26,"updatedAt":2902},"authoring-java-sdk-tasks","implement Airflow tasks in Java","Writes Airflow task logic in Java, Kotlin, or any JVM language using the Airflow Java SDK. Use when the user wants to implement Airflow tasks in Java\u002FJVM, asks about `@Builder.Dag`\u002F`@Builder.Task`\u002F`@Builder.XCom`, the `Task`\u002F`BundleBuilder` interfaces, reading connections\u002Fvariables\u002FXComs from Java, the JSON-to-Java type mapping, or logging from Java tasks. This skill covers the Java-specific native API; the shared Python-stub pattern and conceptual model live in authoring-language-sdk-tasks. For building\u002Fshipping the bundle see deploying-java-sdk-bundles; for coordinator config see configuring-airflow-language-sdks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2895,2896,2899],{"name":2788,"slug":2789,"type":15},{"name":2897,"slug":2898,"type":15},"Engineering","engineering",{"name":2900,"slug":2901,"type":15},"Java","java","2026-07-18T05:48:13.374003",{"slug":2904,"name":2904,"fn":2905,"description":2906,"org":2907,"tags":2908,"stars":25,"repoUrl":26,"updatedAt":2912},"authoring-language-sdk-tasks","implement non-Python Airflow tasks","The language-neutral foundation for Airflow language SDKs — implement task logic in a non-Python language while the DAG stays in Python. Use when the user wants to run an Airflow task in another language (Java, Kotlin, Go, or other JVM\u002Fnative languages), asks how the Python `@task.stub` pairs with native task code, how task\u002FDAG IDs must match across the two sides, how data passes via XCom as JSON, or which language SDKs exist. This skill owns the shared Python-stub pattern and conceptual model; for a specific language's native API, build, and runtime, use that language's skill (e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2909,2910,2911],{"name":17,"slug":18,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2897,"slug":2898,"type":15},"2026-07-18T05:11:54.496539",{"slug":2914,"name":2914,"fn":2915,"description":2916,"org":2917,"tags":2918,"stars":25,"repoUrl":26,"updatedAt":2925},"blueprint","build reusable Airflow task group templates","Define reusable Airflow task group templates with Pydantic validation and compose DAGs from YAML. Use when creating blueprint templates, composing DAGs from YAML, validating configurations, or enabling no-code DAG authoring for non-engineers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2919,2920,2921,2922],{"name":17,"slug":18,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2870,"slug":2871,"type":15},{"name":2923,"slug":2924,"type":15},"Templates","templates","2026-04-06T18:01:45.361425",{"slug":2927,"name":2927,"fn":2928,"description":2929,"org":2930,"tags":2931,"stars":25,"repoUrl":26,"updatedAt":2938},"checking-freshness","check data freshness in warehouses","Quick data freshness check. Use when the user asks if data is up to date, when a table was last updated, if data is stale, or needs to verify data currency before using it.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2932,2933,2934,2937],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2935,"slug":2936,"type":15},"Data Quality","data-quality",{"name":2870,"slug":2871,"type":15},"2026-04-06T18:02:02.138565",{"items":2940,"total":608},[2941,2948,2954,2961,2968,2975,2982],{"slug":18,"name":18,"fn":2779,"description":2780,"org":2942,"tags":2943,"stars":25,"repoUrl":26,"updatedAt":2793},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2944,2945,2946,2947],{"name":17,"slug":18,"type":15},{"name":2785,"slug":2786,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2791,"slug":2792,"type":15},{"slug":2795,"name":2795,"fn":2796,"description":2797,"org":2949,"tags":2950,"stars":25,"repoUrl":26,"updatedAt":2805},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2951,2952,2953],{"name":17,"slug":18,"type":15},{"name":2802,"slug":2803,"type":15},{"name":2788,"slug":2789,"type":15},{"slug":2807,"name":2807,"fn":2808,"description":2809,"org":2955,"tags":2956,"stars":25,"repoUrl":26,"updatedAt":2820},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2957,2958,2959,2960],{"name":17,"slug":18,"type":15},{"name":2814,"slug":2815,"type":15},{"name":23,"slug":24,"type":15},{"name":2818,"slug":2819,"type":15},{"slug":2822,"name":2822,"fn":2823,"description":2824,"org":2962,"tags":2963,"stars":25,"repoUrl":26,"updatedAt":2833},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2964,2965,2966,2967],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2831,"slug":2832,"type":15},{"slug":2835,"name":2835,"fn":2836,"description":2837,"org":2969,"tags":2970,"stars":25,"repoUrl":26,"updatedAt":2850},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2971,2972,2973,2974],{"name":2841,"slug":2842,"type":15},{"name":2844,"slug":2845,"type":15},{"name":20,"slug":21,"type":15},{"name":2848,"slug":2849,"type":15},{"slug":2747,"name":2747,"fn":2852,"description":2853,"org":2976,"tags":2977,"stars":25,"repoUrl":26,"updatedAt":2860},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2978,2979,2980,2981],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2788,"slug":2789,"type":15},{"name":13,"slug":14,"type":15},{"slug":2862,"name":2862,"fn":2863,"description":2864,"org":2983,"tags":2984,"stars":25,"repoUrl":26,"updatedAt":2873},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2985,2986,2987,2988],{"name":17,"slug":18,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2870,"slug":2871,"type":15},{"name":23,"slug":24,"type":15}]