[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-apache-datafusion-python":3,"mdc--l4jtre-key":33,"related-repo-apache-datafusion-python":6949,"related-org-apache-datafusion-python":6957},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":28,"sourceUrl":31,"mdContent":32},"datafusion-python","write Apache DataFusion Python code","Use when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code. Covers imports, data loading, DataFrame operations, expression building, SQL-to-DataFrame mappings, idiomatic patterns, and common pitfalls.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"apache","Apache Software Foundation","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fapache.png",[12,16,19],{"name":13,"slug":14,"type":15},"Data Analysis","data-analysis","tag",{"name":17,"slug":18,"type":15},"Python","python",{"name":20,"slug":21,"type":15},"SQL","sql",593,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python","2026-07-12T08:36:04.957626",null,161,[],{"repoUrl":23,"stars":22,"forks":26,"topics":29,"description":30},[],"Apache DataFusion Python Bindings","https:\u002F\u002Fgithub.com\u002Fapache\u002Fdatafusion-python\u002Ftree\u002FHEAD\u002Fskills\u002Fdatafusion_python","---\nname: datafusion-python\ndescription: Use when the user is writing datafusion-python (Apache DataFusion Python bindings) DataFrame or SQL code. Covers imports, data loading, DataFrame operations, expression building, SQL-to-DataFrame mappings, idiomatic patterns, and common pitfalls.\n---\n\n# DataFusion Python DataFrame API Guide\n\n## What Is DataFusion?\n\nDataFusion is an **in-process query engine** built on Apache Arrow. It is not a\ndatabase -- there is no server, no connection string, and no external\ndependencies. You create a `SessionContext`, point it at data (Parquet, CSV,\nJSON, Arrow IPC, Pandas, Polars, or raw Python dicts\u002Flists), and run queries\nusing either SQL or the DataFrame API described below.\n\nAll data flows through **Apache Arrow**. The canonical Python implementation is\nPyArrow (`pyarrow.RecordBatch` \u002F `pyarrow.Table`), but any library that\nconforms to the [Arrow C Data Interface](https:\u002F\u002Farrow.apache.org\u002Fdocs\u002Fformat\u002FCDataInterface.html)\ncan interoperate with DataFusion.\n\n## Core Abstractions\n\n| Abstraction | Role | Key import |\n|---|---|---|\n| `SessionContext` | Entry point. Loads data, runs SQL, produces DataFrames. | `from datafusion import SessionContext` |\n| `DataFrame` | Lazy query builder. Each method returns a new DataFrame. | Returned by context methods |\n| `Expr` | Expression tree node (column ref, literal, function call, ...). | `from datafusion import col, lit` |\n| `functions` | 290+ built-in scalar, aggregate, and window functions. | `from datafusion import functions as F` |\n| `functions.spark` | PySpark-compatible function surface (parameter names match `pyspark.sql.functions`). | `from datafusion.functions import spark` |\n\n## Import Conventions\n\n```python\nfrom datafusion import SessionContext, col, lit\nfrom datafusion import functions as F\nfrom datafusion.functions import spark # only when porting pyspark code\n```\n\n## Data Loading\n\n```python\nctx = SessionContext()\n\n# From files\ndf = ctx.read_parquet(\"path\u002Fto\u002Fdata.parquet\")\ndf = ctx.read_csv(\"path\u002Fto\u002Fdata.csv\")\ndf = ctx.read_json(\"path\u002Fto\u002Fdata.json\")\n\n# From Python objects\ndf = ctx.from_pydict({\"a\": [1, 2, 3], \"b\": [\"x\", \"y\", \"z\"]})\ndf = ctx.from_pylist([{\"a\": 1, \"b\": \"x\"}, {\"a\": 2, \"b\": \"y\"}])\ndf = ctx.from_pandas(pandas_df)\ndf = ctx.from_polars(polars_df)\ndf = ctx.from_arrow(arrow_table)\ndf = ctx.read_batch(record_batch) # one pa.RecordBatch, no named table\ndf = ctx.read_batches([batch1, batch2]) # several pa.RecordBatch\n\n# From SQL\ndf = ctx.sql(\"SELECT a, b FROM my_table WHERE a > 1\")\n```\n\nTo make a DataFrame queryable by name in SQL, register it first:\n\n```python\nctx.register_parquet(\"my_table\", \"path\u002Fto\u002Fdata.parquet\")\nctx.register_csv(\"my_table\", \"path\u002Fto\u002Fdata.csv\")\n```\n\n## DataFrame Operations Quick Reference\n\nEvery method returns a **new** DataFrame (immutable\u002Flazy). Chain them fluently.\n\n### Projection\n\n```python\ndf.select(\"a\", \"b\") # preferred: plain names as strings\ndf.select(col(\"a\"), (col(\"b\") + 1).alias(\"b_plus_1\")) # use col()\u002FExpr only when you need an expression\n\ndf.with_column(\"new_col\", col(\"a\") + lit(10)) # add one column\ndf.with_columns(\n col(\"a\").alias(\"x\"),\n y=col(\"b\") + lit(1), # named keyword form\n)\n\ndf.drop(\"unwanted_col\")\ndf.with_column_renamed(\"old_name\", \"new_name\")\n```\n\nWhen a column is referenced by name alone, pass the name as a string rather\nthan wrapping it in `col()`. Reach for `col()` only when the projection needs\narithmetic, aliasing, casting, or another expression operation.\n\n**Case sensitivity**: both `select(\"Name\")` and `col(\"Name\")` lowercase the\nidentifier. For a column whose real name has uppercase letters, embed double\nquotes inside the string: `select('\"MyCol\"')` or `col('\"MyCol\"')`. Without the\ninner quotes the lookup will fail with `No field named mycol`.\n\n### Filtering\n\n```python\ndf.filter(col(\"a\") > 10)\ndf.filter(col(\"a\") > 10, col(\"b\") == \"x\") # multiple = AND\ndf.filter(\"a > 10\") # SQL expression string\n```\n\nRaw Python values on the right-hand side of a comparison are auto-wrapped\ninto literals by the `Expr` operators, so prefer `col(\"a\") > 10` over\n`col(\"a\") > lit(10)`. See the Comparisons section and pitfall #2 for the\nfull rule.\n\n### Aggregation\n\n```python\n# GROUP BY a, compute sum(b) and count(*)\ndf.aggregate([\"a\"], [F.sum(col(\"b\")), F.count(col(\"a\"))])\n\n# HAVING equivalent: use the filter keyword on the aggregate function\ndf.aggregate(\n [\"region\"],\n [F.sum(col(\"sales\"), filter=col(\"sales\") > 1000).alias(\"large_sales\")],\n)\n```\n\nAs with `select()`, group keys can be passed as plain name strings. Reach for\n`col(...)` only when the grouping expression needs arithmetic, aliasing,\ncasting, or another expression operation.\n\nMost aggregate functions accept an optional `filter` keyword argument. When\nprovided, only rows where the filter expression is true contribute to the\naggregate.\n\n### Sorting\n\n```python\ndf.sort(\"a\") # ascending (plain name, preferred)\ndf.sort(col(\"a\")) # ascending via col()\ndf.sort(col(\"a\").sort(ascending=False)) # descending\ndf.sort(col(\"a\").sort(nulls_first=False)) # override null placement\n\ndf.sort_by(\"a\", \"b\") # ascending-only shortcut\n```\n\nAs with `select()` and `aggregate()`, bare column references can be passed as\nplain name strings. A plain expression passed to `sort()` is already treated\nas ascending, so reach for `col(...).sort(...)` only when you need to override\na default (descending order or null placement). Writing\n`col(\"a\").sort(ascending=True)` is redundant.\n\nFor ascending-only sorts with no null-placement override, `df.sort_by(...)` is\na shorter alias for `df.sort(...)`.\n\n### Joining\n\n```python\n# Equi-join on shared column name\ndf1.join(df2, on=\"key\")\ndf1.join(df2, on=\"key\", how=\"left\")\n\n# Different column names\ndf1.join(df2, left_on=\"id\", right_on=\"fk_id\", how=\"inner\")\n\n# Expression-based join (supports inequality predicates)\ndf1.join_on(df2, col(\"a\") == col(\"b\"), how=\"inner\")\n\n# Semi join: keep rows from left where a match exists in right (like EXISTS)\ndf1.join(df2, on=\"key\", how=\"semi\")\n\n# Anti join: keep rows from left where NO match exists in right (like NOT EXISTS)\ndf1.join(df2, on=\"key\", how=\"anti\")\n```\n\nJoin types: `\"inner\"`, `\"left\"`, `\"right\"`, `\"full\"`, `\"semi\"`, `\"anti\"`.\n\nInner is the default `how`. Prefer `df1.join(df2, on=\"key\")` over\n`df1.join(df2, on=\"key\", how=\"inner\")` — drop `how=` unless you need a\nnon-inner join type.\n\nWhen the two sides' join columns have different native names, use\n`left_on=`\u002F`right_on=` with the original names rather than aliasing one side\nto match the other — see pitfall #7.\n\n### Window Functions\n\n```python\nfrom datafusion import WindowFrame\n\n# Row number partitioned by group, ordered by value\ndf.window(\n F.row_number(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\")],\n ).alias(\"rn\")\n)\n\n# Using a Window object for reuse\nfrom datafusion.expr import Window\n\nwin = Window(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\").sort(ascending=True)],\n)\ndf.select(\n col(\"group\"),\n col(\"value\"),\n F.sum(col(\"value\")).over(win).alias(\"running_total\"),\n)\n\n# With explicit frame bounds\nwin = Window(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\").sort(ascending=True)],\n window_frame=WindowFrame(\"rows\", 0, None), # current row to unbounded following\n)\n```\n\n### Set Operations\n\n```python\ndf1.union(df2) # UNION ALL (by position)\ndf1.union(df2, distinct=True) # UNION DISTINCT\ndf1.union_by_name(df2) # match columns by name, not position\ndf1.intersect(df2) # INTERSECT ALL\ndf1.intersect(df2, distinct=True) # INTERSECT (distinct)\ndf1.except_all(df2) # EXCEPT ALL\ndf1.except_all(df2, distinct=True) # EXCEPT (distinct)\n```\n\n### Limit and Offset\n\n```python\ndf.limit(10) # first 10 rows\ndf.limit(10, offset=20) # skip 20, then take 10\n```\n\n### Deduplication\n\n```python\ndf.distinct() # remove duplicate rows\ndf.distinct_on( # keep first row per group (like DISTINCT ON in Postgres)\n [col(\"a\")], # uniqueness columns\n [col(\"a\"), col(\"b\")], # output columns\n [col(\"b\").sort(ascending=True)], # which row to keep\n)\n```\n\n## Executing and Collecting Results\n\nDataFrames are lazy until you collect.\n\n```python\ndf.show() # print formatted table to stdout\nbatches = df.collect() # list[pa.RecordBatch]\narr = df.collect_column(\"col_name\") # pa.Array | pa.ChunkedArray (single column)\ntable = df.to_arrow_table() # pa.Table\npandas_df = df.to_pandas() # pd.DataFrame\npolars_df = df.to_polars() # pl.DataFrame\npy_dict = df.to_pydict() # dict[str, list]\npy_list = df.to_pylist() # list[dict]\ncount = df.count() # int\ndf = df.cache() # materialize in memory, return DataFrame\n```\n\n### Date and Timestamp Type Conversion\n\nThe Python type returned by `to_pydict()` \u002F `to_pylist()` depends on the Arrow\ncolumn type, and the mapping is inherited from PyArrow:\n\n| Arrow type | Python type returned |\n|---|---|\n| `timestamp(s)` \u002F `(ms)` \u002F `(us)` | `datetime.datetime` |\n| `timestamp(ns)` | `pandas.Timestamp` |\n| `date32` \u002F `date64` | `datetime.date` |\n| `duration(s)` \u002F `(ms)` \u002F `(us)` | `datetime.timedelta` |\n| `duration(ns)` | `pandas.Timedelta` |\n\nThe nanosecond-precision fallback to pandas types is the main surprise:\npandas is not a hard dependency of `datafusion`, but PyArrow reaches for it\nwhen `datetime.datetime` \u002F `datetime.timedelta` would lose precision (stdlib\ntypes only go to microseconds). If you need plain stdlib types, cast to a\ncoarser unit before collecting, e.g.\n`df.select(col(\"ts\").cast(pa.timestamp(\"us\")))`.\n\n`df.to_pandas()` has its own footgun for dates: pandas has no pure-date dtype,\nso a `date32`\u002F`date64` column comes back as an `object` column of\n`datetime.date` values rather than `datetime64[ns]`. If downstream code\nexpects a datetime column, cast on the DataFusion side first:\n`col(\"ship_date\").cast(pa.timestamp(\"ns\"))`.\n\n### Streaming Results\n\nPrefer streaming over `collect()` when the result is too large to materialize\nin memory, when you want to start processing before the query finishes, or\nwhen you may break out of the loop early. `execute_stream()` pulls one\n`RecordBatch` at a time from the execution plan rather than buffering the\nwhole result up front.\n\n```python\n# Single-partition stream; batch is a datafusion.RecordBatch\nstream = df.execute_stream()\nfor batch in stream:\n process(batch.to_pyarrow()) # convert to pa.RecordBatch if needed\n\n# DataFrame is iterable directly (delegates to execute_stream)\nfor batch in df:\n process(batch.to_pyarrow())\n\n# One stream per partition, for parallel consumption\nfor stream in df.execute_stream_partitioned():\n for batch in stream:\n process(batch.to_pyarrow())\n```\n\nAsync iteration is also supported via `async for batch in df: ...` (or\n`df.execute_stream()`), which is useful when batches are interleaved with\nother I\u002FO.\n\n### Caching Intermediate Results\n\n`df.cache()` materializes a DataFrame as an in-memory table and returns a new\nDataFrame backed by it. Reach for it when the same intermediate result feeds\nmultiple downstream queries — without `cache()`, each branch re-executes the\nfull upstream plan (re-reading files, recomputing filters\u002Faggregates).\n\n```python\nbase = (\n ctx.read_parquet(\"orders.parquet\")\n .filter(col(\"status\") == \"shipped\")\n .cache() # materialize once, reuse below\n)\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\nby_customer = base.aggregate([\"customer\"], [F.sum(col(\"amount\")).alias(\"total\")])\n```\n\nSkip `cache()` for single-use DataFrames — the lazy plan is already optimal.\n\nThe cached table is owned by the DataFrame returned from `cache()` (and any\nDataFrames chained from it). To free the memory, drop every reference — let\nthem go out of scope, or `del base; del by_region; del by_customer`.\n\n### Writing Results\n\n```python\ndf.write_parquet(\"output.parquet\")\ndf.write_csv(\"output.csv\")\ndf.write_json(\"output.json\")\n```\n\nYou can also pass a directory path (e.g., `\"output\u002F\"`) to write a multi-file\npartitioned output.\n\n## Expression Building\n\n### Column References and Literals\n\n```python\ncol(\"column_name\") # reference a column\nlit(42) # integer literal\nlit(\"hello\") # string literal\nlit(3.14) # float literal\nlit(pa.scalar(value)) # PyArrow scalar (preserves Arrow type)\n```\n\n`lit()` accepts PyArrow scalars directly -- prefer this over converting Arrow\ndata to Python and back when working with values extracted from query results.\n\n### Arithmetic\n\n```python\ncol(\"price\") * col(\"quantity\") # multiplication\ncol(\"a\") + lit(1) # addition\ncol(\"a\") - col(\"b\") # subtraction\ncol(\"a\") \u002F lit(2) # division\ncol(\"a\") % lit(3) # modulo\n```\n\n### Date Arithmetic\n\n`Date32` and `Date64` columns both require `Interval` types for arithmetic,\nnot `Duration`. Use PyArrow's `month_day_nano_interval` type, which takes a\n`(months, days, nanos)` tuple:\n\n```python\nimport pyarrow as pa\n\n# Subtract 90 days from a date column\ncol(\"ship_date\") - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n\n# Subtract 3 months\ncol(\"ship_date\") - lit(pa.scalar((3, 0, 0), type=pa.month_day_nano_interval()))\n```\n\n**Important**: `lit(datetime.timedelta(days=90))` creates a `Duration(µs)`\nliteral, which is **not** compatible with `Date32`\u002F`Date64` arithmetic\n(`Duration(ms)` and `Duration(ns)` are rejected too). Always use\n`pa.month_day_nano_interval()` for date operations.\n\n**Timestamps behave differently**: `Timestamp` columns *do* accept `Duration`,\nso `col(\"ts\") - lit(datetime.timedelta(days=1))` works. The interval-only\nrule applies specifically to date columns.\n\n### Comparisons\n\n```python\ncol(\"a\") > 10\ncol(\"a\") >= 10\ncol(\"a\") \u003C 10\ncol(\"a\") \u003C= 10\ncol(\"a\") == \"x\"\ncol(\"a\") != \"x\"\ncol(\"a\") == None # same as col(\"a\").is_null()\ncol(\"a\") != None # same as col(\"a\").is_not_null()\n```\n\nComparison operators auto-wrap the right-hand Python value into a literal,\nso writing `col(\"a\") > lit(10)` is redundant. Drop the `lit()` in\ncomparisons. Reach for `lit()` only when auto-wrapping does not apply — see\npitfall #2.\n\n### Boolean Logic\n\n**Important**: Python's `and`, `or`, `not` keywords do NOT work with Expr\nobjects. You must use the bitwise operators:\n\n```python\n(col(\"a\") > 1) & (col(\"b\") \u003C 10) # AND\n(col(\"a\") > 1) | (col(\"b\") \u003C 10) # OR\n~(col(\"a\") > 1) # NOT\n```\n\nAlways wrap each comparison in parentheses when combining with `&`, `|`, `~`\nbecause Python's operator precedence for bitwise operators is different from\nlogical operators.\n\n### Null Handling\n\n```python\ncol(\"a\").is_null()\ncol(\"a\").is_not_null()\ncol(\"a\").fill_null(lit(0)) # replace NULL with a value (single expression)\nF.coalesce(col(\"a\"), col(\"b\")) # first non-null value\nF.nullif(col(\"a\"), lit(0)) # return NULL if a == 0\n```\n\nTo fill nulls across the whole DataFrame (optionally limited to a subset of\ncolumns), use the DataFrame-level method:\n\n```python\ndf.fill_null(0) # every column\ndf.fill_null(0, subset=[\"a\", \"b\"]) # only these columns\n```\n\n### CASE \u002F WHEN\n\n```python\n# Simple CASE (matching on a single expression)\nstatus_label = (\n F.case(col(\"status\"))\n .when(lit(\"A\"), lit(\"Active\"))\n .when(lit(\"I\"), lit(\"Inactive\"))\n .otherwise(lit(\"Unknown\"))\n)\n\n# Searched CASE (each branch has its own predicate)\nseverity = (\n F.when(col(\"value\") > 100, lit(\"high\"))\n .when(col(\"value\") > 50, lit(\"medium\"))\n .otherwise(lit(\"low\"))\n)\n```\n\n### Casting\n\n```python\nimport pyarrow as pa\n\ncol(\"a\").cast(pa.float64())\ncol(\"a\").cast(pa.utf8())\ncol(\"a\").cast(pa.date32())\n\ncol(\"a\").try_cast(pa.int32()) # like cast(), but yields NULL on failure instead of erroring\n```\n\nTo cast several columns at once at the DataFrame level, pass a mapping to\n`df.cast(...)`:\n\n```python\ndf.cast({\"a\": pa.float64(), \"b\": pa.int32()})\n```\n\n### Aliasing\n\n```python\n(col(\"a\") + col(\"b\")).alias(\"total\")\n```\n\n### BETWEEN and IN\n\n```python\ncol(\"a\").between(1, 10) # 1 \u003C= a \u003C= 10 (bounds auto-wrap)\nF.in_list(col(\"a\"), [lit(1), lit(2), lit(3)]) # a IN (1, 2, 3)\nF.in_list(col(\"a\"), [lit(1), lit(2)], negated=True) # a NOT IN (1, 2)\n```\n\n### Struct and Array Access\n\n```python\ncol(\"struct_col\")[\"field_name\"] # access struct field\ncol(\"array_col\")[0] # access array element (0-indexed)\ncol(\"array_col\")[1:3] # array slice (0-indexed)\n```\n\n### Lambda Functions\n\nSome array functions take a lambda function that runs once per element. Pass a\nPython `lambda` directly — its parameter names become the lambda parameters and\nits return value becomes the body:\n\n```python\nF.array_transform(col(\"a\"), lambda v: v * 2) # map: [1,2,3] -> [2,4,6]\nF.array_filter(col(\"a\"), lambda v: v > 2) # filter: [1,2,3] -> [3]\nF.array_any_match(col(\"a\"), lambda v: v > 3) # predicate: any element > 3\n```\n\nFor explicit parameter names, build the lambda by hand:\n\n```python\nF.array_transform(col(\"a\"), F.lambda_([\"v\"], F.lambda_var(\"v\") * lit(2)))\n```\n\n## SQL-to-DataFrame Reference\n\n| SQL | DataFrame API |\n|---|---|\n| `SELECT a, b` | `df.select(\"a\", \"b\")` |\n| `SELECT a, b + 1 AS c` | `df.select(col(\"a\"), (col(\"b\") + lit(1)).alias(\"c\"))` |\n| `SELECT *, a + 1 AS c` | `df.with_column(\"c\", col(\"a\") + lit(1))` |\n| `WHERE a > 10` | `df.filter(col(\"a\") > 10)` |\n| `GROUP BY a` with `SUM(b)` | `df.aggregate([\"a\"], [F.sum(col(\"b\"))])` |\n| `SUM(b) FILTER (WHERE b > 100)` | `F.sum(col(\"b\"), filter=col(\"b\") > 100)` |\n| `ORDER BY a DESC` | `df.sort(col(\"a\").sort(ascending=False))` |\n| `LIMIT 10 OFFSET 5` | `df.limit(10, offset=5)` |\n| `DISTINCT` | `df.distinct()` |\n| `a INNER JOIN b ON a.id = b.id` | `a.join(b, on=\"id\")` |\n| `a LEFT JOIN b ON a.id = b.fk` | `a.join(b, left_on=\"id\", right_on=\"fk\", how=\"left\")` |\n| `WHERE EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"semi\")` |\n| `WHERE NOT EXISTS (SELECT ...)` | `a.join(b, on=\"key\", how=\"anti\")` |\n| `UNION ALL` | `df1.union(df2)` |\n| `UNION` (distinct) | `df1.union(df2, distinct=True)` |\n| `INTERSECT ALL` | `df1.intersect(df2)` |\n| `INTERSECT` (distinct) | `df1.intersect(df2, distinct=True)` |\n| `EXCEPT ALL` | `df1.except_all(df2)` |\n| `EXCEPT` (distinct) | `df1.except_all(df2, distinct=True)` |\n| `CASE x WHEN 1 THEN 'a' END` | `F.case(col(\"x\")).when(lit(1), lit(\"a\")).end()` |\n| `CASE WHEN x > 1 THEN 'a' END` | `F.when(col(\"x\") > 1, lit(\"a\")).end()` |\n| `x IN (1, 2, 3)` | `F.in_list(col(\"x\"), [lit(1), lit(2), lit(3)])` |\n| `x BETWEEN 1 AND 10` | `col(\"x\").between(1, 10)` |\n| `CAST(x AS DOUBLE)` | `col(\"x\").cast(pa.float64())` |\n| `ROW_NUMBER() OVER (...)` | `F.row_number(partition_by=[...], order_by=[...])` |\n| `SUM(x) OVER (...)` | `F.sum(col(\"x\")).over(window)` |\n| `x IS NULL` | `col(\"x\").is_null()` |\n| `COALESCE(a, b)` | `F.coalesce(col(\"a\"), col(\"b\"))` |\n\n## Common Pitfalls\n\n1. **Boolean operators**: Use `&`, `|`, `~` -- not Python's `and`, `or`, `not`.\n Always parenthesize: `(col(\"a\") > 1) & (col(\"b\") \u003C 2)`.\n\n2. **Wrapping scalars with `lit()`**: Prefer raw Python values on the\n right-hand side of comparisons — `col(\"a\") > 10`, `col(\"name\") == \"Alice\"`\n — because the Expr comparison operators auto-wrap them. Writing\n `col(\"a\") > lit(10)` is redundant. Reserve `lit()` for places where\n auto-wrapping does *not* apply:\n - standalone scalars passed into function calls:\n `F.coalesce(col(\"a\"), lit(0))`, not `F.coalesce(col(\"a\"), 0)`\n - arithmetic between two literals with no column involved:\n `lit(1) - col(\"discount\")` is fine, but `lit(1) - lit(2)` needs both\n - values that must carry a specific Arrow type, via `lit(pa.scalar(...))`\n - `.when(...)`, `.otherwise(...)`, `F.nullif(...)`, `F.in_list(...)`\n and similar method\u002Ffunction arguments (note: `.between(...)`\n auto-wraps its bounds, so `col(\"a\").between(1, 10)` needs no `lit()`)\n\n3. **Column name quoting**: Column names are normalized to lowercase by default\n in both `select(\"...\")` and `col(\"...\")`. To reference a column with\n uppercase letters, use double quotes inside the string:\n `select('\"MyColumn\"')` or `col('\"MyColumn\"')`.\n\n4. **DataFrames are immutable**: Every method returns a **new** DataFrame. You\n must capture the return value:\n ```python\n df = df.filter(col(\"a\") > 1) # correct\n df.filter(col(\"a\") > 1) # WRONG -- result is discarded\n ```\n\n5. **Window frame defaults**: When using `order_by` in a window, the default\n frame is `RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. For a full\n partition frame, set `window_frame=WindowFrame(\"rows\", None, None)`.\n\n6. **Arithmetic on aggregates belongs in a later `select`, not inside\n `aggregate`** *(applies to datafusion-python 53 and earlier; fixed in 54)*:\n Each item in the aggregate list must be a single aggregate\n call (optionally aliased). Combining aggregates with arithmetic inside\n `aggregate(...)` fails with `Internal error: Invalid aggregate expression`.\n Alias the aggregates, then compute the combination downstream:\n ```python\n # WRONG -- arithmetic wraps two aggregates\n df.aggregate([], [(lit(100) * F.sum(col(\"a\")) \u002F F.sum(col(\"b\"))).alias(\"ratio\")])\n\n # CORRECT -- aggregate first, then combine\n (df.aggregate([], [F.sum(col(\"a\")).alias(\"num\"), F.sum(col(\"b\")).alias(\"den\")])\n .select((lit(100) * col(\"num\") \u002F col(\"den\")).alias(\"ratio\")))\n ```\n\n7. **Don't alias a join column to match the other side**: When equi-joining\n with `on=\"key\"`, renaming the join column on one side via `.alias(\"key\")`\n in a fresh projection creates a schema where one side's `key` is\n qualified (`?table?.key`) and the other is unqualified. The join then\n fails with `Schema contains qualified field name ... and unqualified\n field name ... which would be ambiguous`. Use `left_on=`\u002F`right_on=` with\n the native names, or use `join_on(...)` with an explicit equality.\n ```python\n # WRONG -- alias on one side produces ambiguous schema after join\n failed = orders.select(col(\"o_orderkey\").alias(\"l_orderkey\"))\n li.join(failed, on=\"l_orderkey\") # ambiguous l_orderkey error\n\n # CORRECT -- keep native names, use left_on\u002Fright_on\n failed = orders.select(\"o_orderkey\")\n li.join(failed, left_on=\"l_orderkey\", right_on=\"o_orderkey\")\n\n # ALSO CORRECT -- explicit predicate via join_on\n # (note: join_on keeps both key columns in the output, unlike on=\"key\")\n li.join_on(failed, col(\"l_orderkey\") == col(\"o_orderkey\"))\n ```\n When the same column name exists on both sides, `DataFrame.col(name)`\n (and `DataFrame.column(name)`) returns a column reference qualified to\n that DataFrame, which disambiguates the predicate explicitly:\n ```python\n li.join_on(failed, li.col(\"l_orderkey\") == failed.col(\"o_orderkey\"))\n ```\n\n## Idiomatic Patterns\n\n### Fluent Chaining\n\n```python\nresult = (\n ctx.read_parquet(\"data.parquet\")\n .filter(col(\"year\") >= 2020)\n .select(col(\"region\"), col(\"sales\"))\n .aggregate([\"region\"], [F.sum(col(\"sales\")).alias(\"total\")])\n .sort(col(\"total\").sort(ascending=False))\n .limit(10)\n)\nresult.show()\n```\n\n### Using Variables as CTEs\n\nInstead of SQL CTEs (`WITH ... AS`), assign intermediate DataFrames to\nvariables:\n\n```python\nbase = ctx.read_parquet(\"orders.parquet\").filter(col(\"status\") == \"shipped\")\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\ntop_regions = by_region.filter(col(\"total\") > 10000)\n```\n\n### Reusing Expressions as Variables\n\nJust like DataFrames, expressions (`Expr`) can be stored in variables and used\nanywhere an `Expr` is expected. This is useful for building up complex\nexpressions or reusing a computed value across multiple operations:\n\n```python\n# Build an expression and reuse it\ndisc_price = col(\"price\") * (lit(1) - col(\"discount\"))\ndf = df.select(\n col(\"id\"),\n disc_price.alias(\"disc_price\"),\n (disc_price * (lit(1) + col(\"tax\"))).alias(\"total\"),\n)\n\n# Use a collected scalar as an expression\nmax_val = result_df.collect_column(\"max_price\")[0] # PyArrow scalar\ncutoff = lit(max_val) - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\ndf = df.filter(col(\"ship_date\") \u003C= cutoff) # cutoff is already an Expr\n```\n\n**Important**: Do not wrap an `Expr` in `lit()`. `lit()` is for converting\nPython\u002FPyArrow values into expressions. If a value is already an `Expr`, use it\ndirectly.\n\n### Window Functions for Scalar Subqueries\n\nWhere SQL uses a correlated scalar subquery, the idiomatic DataFrame approach\nis a window function:\n\n```sql\n-- SQL scalar subquery\nSELECT *, (SELECT SUM(b) FROM t WHERE t.group = s.group) AS group_total FROM s\n```\n\n```python\n# DataFrame: window function\nwin = Window(partition_by=[col(\"group\")])\ndf = df.with_column(\"group_total\", F.sum(col(\"b\")).over(win))\n```\n\n### Semi\u002FAnti Joins for EXISTS \u002F NOT EXISTS\n\n```sql\n-- SQL: WHERE EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"semi\")\n\n-- SQL: WHERE NOT EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"anti\")\n```\n\n### Computed Columns\n\n```python\n# Add computed columns while keeping all originals\ndf = df.with_column(\"full_name\", F.concat(col(\"first\"), lit(\" \"), col(\"last\")))\ndf = df.with_column(\"discounted\", col(\"price\") * lit(0.9))\n```\n\n## Available Functions (Categorized)\n\nThe `functions` module (imported as `F`) provides 290+ functions. Key categories:\n\n**Aggregate**: `sum`, `avg`, `min`, `max`, `count`, `count_star`, `median`,\n`stddev`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar`, `approx_distinct`,\n`approx_median`, `approx_percentile_cont`, `array_agg`, `string_agg`,\n`first_value`, `last_value`, `bit_and`, `bit_or`, `bit_xor`, `bool_and`,\n`bool_or`, `grouping`, `regr_*` (9 regression functions)\n\n**Window**: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`,\n`ntile`, `lag`, `lead`, `first_value`, `last_value`, `nth_value`\n\n**String**: `length`, `lower`, `upper`, `trim`, `ltrim`, `rtrim`, `lpad`,\n`rpad`, `starts_with`, `ends_with`, `contains`, `substr`, `substring`,\n`replace`, `reverse`, `repeat`, `split_part`, `concat`, `concat_ws`,\n`initcap`, `ascii`, `chr`, `left`, `right`, `strpos`, `translate`, `overlay`,\n`levenshtein`\n\n`F.substr(str, start)` takes **only two arguments** and returns the tail of\nthe string from `start` onward — passing a third length argument raises\n`TypeError: substr() takes 2 positional arguments but 3 were given`. For the\nSQL-style 3-arg form (`SUBSTRING(str FROM start FOR length)`), use\n`F.substring(col(\"s\"), lit(start), lit(length))`. For a fixed-length prefix,\n`F.left(col(\"s\"), lit(n))` is cleanest.\n\n```python\n# WRONG — substr does not accept a length argument\nF.substr(col(\"c_phone\"), lit(1), lit(2))\n# CORRECT\nF.substring(col(\"c_phone\"), lit(1), lit(2)) # explicit length\nF.left(col(\"c_phone\"), lit(2)) # prefix shortcut\n```\n\n**Math**: `abs`, `ceil`, `floor`, `round`, `trunc`, `sqrt`, `cbrt`, `exp`,\n`ln`, `log`, `log2`, `log10`, `pow`, `signum`, `pi`, `random`, `factorial`,\n`gcd`, `lcm`, `greatest`, `least`, sin\u002Fcos\u002Ftan and inverse\u002Fhyperbolic variants\n\n**Date\u002FTime**: `now`, `today`, `current_date`, `current_time`,\n`current_timestamp`, `date_part`, `date_trunc`, `date_bin`, `extract`,\n`to_timestamp`, `to_timestamp_millis`, `to_timestamp_micros`,\n`to_timestamp_nanos`, `to_timestamp_seconds`, `to_unixtime`, `from_unixtime`,\n`make_date`, `make_time`, `to_date`, `to_time`, `to_local_time`, `date_format`\n\n**Conditional**: `case`, `when`, `coalesce`, `nullif`, `ifnull`, `nvl`, `nvl2`\n\n**Array\u002FList**: `array`, `make_array`, `array_agg`, `array_length`,\n`array_element`, `array_slice`, `array_append`, `array_prepend`,\n`array_concat`, `array_contains`, `array_has`, `array_has_all`, `array_has_any`, `array_position`,\n`array_remove`, `array_distinct`, `array_sort`, `array_reverse`, `flatten`,\n`array_to_string`, `array_intersect`, `array_union`, `array_except`,\n`generate_series`\n(Most `array_*` functions also have `list_*` aliases.)\n\n**Struct\u002FMap**: `struct`, `named_struct`, `get_field`, `make_map`, `map_keys`,\n`map_values`, `map_entries`, `map_extract`\n\n**Regex**: `regexp_like`, `regexp_match`, `regexp_replace`, `regexp_count`,\n`regexp_instr`\n\n**Hash**: `md5`, `sha224`, `sha256`, `sha384`, `sha512`, `digest`\n\n**Type**: `arrow_typeof`, `arrow_cast`, `arrow_try_cast`, `arrow_field`,\n`arrow_metadata`, `cast_to_type`, `with_metadata`\n\nNote: ``cast_to_type(value, type_ref, *, try_cast=False)`` is the single\nPython entry point for both upstream ``cast_to_type`` and ``try_cast_to_type``;\npass ``try_cast=True`` for the variant that returns NULL on failure.\n\n**Other**: `in_list`, `order_by`, `alias`, `col`, `encode`, `decode`,\n`to_hex`, `to_char`, `uuid`, `version`, `bit_length`, `octet_length`\n\n### Spark-Compatible Functions\n\nA separate `datafusion.functions.spark` namespace mirrors the\n`pyspark.sql.functions` API for callers porting code from PySpark.\n\n```python\nfrom datafusion.functions import spark\n```\n\nUse it for DataFrame work; for SQL, register the Spark UDFs first:\n\n```python\nctx = SessionContext()\nctx.enable_spark_functions() # makes Spark UDFs visible to SQL\nctx.sql(\"SELECT sha2('hello', 256)\").show()\n```\n\nCoverage spans aggregate, array, bitmap, bitwise, datetime, hash, JSON,\nmap, math, string, URL, and conditional categories. The authoritative\nlist of what is currently exposed is the `__all__` in\n`python\u002Fdatafusion\u002Ffunctions\u002Fspark.py`:\n\n```bash\npython -c \"from datafusion.functions import spark; print(sorted(spark.__all__))\"\n```\n\nWhen you need to know whether a specific pyspark function is available,\ncheck `__all__` rather than this skill — the list there moves with the\ncode; any enumeration here would drift.\n\n**Semantic divergences vs the default namespace.** Functions that exist in\nboth `functions` and `functions.spark` may behave differently:\n\n| Function | Default `functions` | `functions.spark` |\n|---|---|---|\n| `concat` | NULL inputs treated as empty | NULL inputs propagate to NULL |\n| `round` | HALF_EVEN (banker's) | HALF_UP |\n| `trunc` | Numeric truncation | Date truncation |\n\nPick the namespace whose semantics match your intent — both stay imported\nside by side; `enable_spark_functions()` only affects SQL.\n\n**Parameter names match pyspark exactly.** The spark namespace uses\npyspark parameter names (`col`, `str`, `numBits`, `partToExtract`, ...) so\nyou can paste pyspark code and keep keyword arguments working. The default\nnamespace keeps DataFusion's parameter names.\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,47,54,77,116,122,288,294,333,339,504,509,532,538,550,557,650,670,720,726,757,785,791,860,881,894,900,954,996,1016,1022,1145,1193,1229,1250,1256,1495,1501,1564,1570,1593,1599,1653,1659,1664,1751,1757,1777,1937,1970,2024,2030,2059,2168,2189,2195,2214,2276,2288,2307,2313,2344,2357,2363,2369,2416,2427,2433,2480,2486,2536,2597,2666,2706,2712,2783,2809,2815,2845,2876,2903,2909,2956,2961,2984,2990,3106,3112,3172,3185,3199,3205,3219,3225,3256,3262,3293,3299,3312,3343,3348,3362,3368,3989,3995,4626,4632,4638,4716,4722,4735,4765,4771,4790,4891,4928,4934,4939,4963,4994,5000,5061,5067,5098,5104,5124,5324,5407,5611,5669,5716,5873,6035,6092,6285,6349,6392,6442,6499,6534,6625,6631,6651,6665,6670,6700,6720,6759,6771,6794,6892,6905,6943],{"type":39,"tag":40,"props":41,"children":43},"element","h1",{"id":42},"datafusion-python-dataframe-api-guide",[44],{"type":45,"value":46},"text","DataFusion Python DataFrame API Guide",{"type":39,"tag":48,"props":49,"children":51},"h2",{"id":50},"what-is-datafusion",[52],{"type":45,"value":53},"What Is DataFusion?",{"type":39,"tag":55,"props":56,"children":57},"p",{},[58,60,66,68,75],{"type":45,"value":59},"DataFusion is an ",{"type":39,"tag":61,"props":62,"children":63},"strong",{},[64],{"type":45,"value":65},"in-process query engine",{"type":45,"value":67}," built on Apache Arrow. It is not a\ndatabase -- there is no server, no connection string, and no external\ndependencies. You create a ",{"type":39,"tag":69,"props":70,"children":72},"code",{"className":71},[],[73],{"type":45,"value":74},"SessionContext",{"type":45,"value":76},", point it at data (Parquet, CSV,\nJSON, Arrow IPC, Pandas, Polars, or raw Python dicts\u002Flists), and run queries\nusing either SQL or the DataFrame API described below.",{"type":39,"tag":55,"props":78,"children":79},{},[80,82,87,89,95,97,103,105,114],{"type":45,"value":81},"All data flows through ",{"type":39,"tag":61,"props":83,"children":84},{},[85],{"type":45,"value":86},"Apache Arrow",{"type":45,"value":88},". The canonical Python implementation is\nPyArrow (",{"type":39,"tag":69,"props":90,"children":92},{"className":91},[],[93],{"type":45,"value":94},"pyarrow.RecordBatch",{"type":45,"value":96}," \u002F ",{"type":39,"tag":69,"props":98,"children":100},{"className":99},[],[101],{"type":45,"value":102},"pyarrow.Table",{"type":45,"value":104},"), but any library that\nconforms to the ",{"type":39,"tag":106,"props":107,"children":111},"a",{"href":108,"rel":109},"https:\u002F\u002Farrow.apache.org\u002Fdocs\u002Fformat\u002FCDataInterface.html",[110],"nofollow",[112],{"type":45,"value":113},"Arrow C Data Interface",{"type":45,"value":115},"\ncan interoperate with DataFusion.",{"type":39,"tag":48,"props":117,"children":119},{"id":118},"core-abstractions",[120],{"type":45,"value":121},"Core Abstractions",{"type":39,"tag":123,"props":124,"children":125},"table",{},[126,150],{"type":39,"tag":127,"props":128,"children":129},"thead",{},[130],{"type":39,"tag":131,"props":132,"children":133},"tr",{},[134,140,145],{"type":39,"tag":135,"props":136,"children":137},"th",{},[138],{"type":45,"value":139},"Abstraction",{"type":39,"tag":135,"props":141,"children":142},{},[143],{"type":45,"value":144},"Role",{"type":39,"tag":135,"props":146,"children":147},{},[148],{"type":45,"value":149},"Key import",{"type":39,"tag":151,"props":152,"children":153},"tbody",{},[154,180,202,228,254],{"type":39,"tag":131,"props":155,"children":156},{},[157,166,171],{"type":39,"tag":158,"props":159,"children":160},"td",{},[161],{"type":39,"tag":69,"props":162,"children":164},{"className":163},[],[165],{"type":45,"value":74},{"type":39,"tag":158,"props":167,"children":168},{},[169],{"type":45,"value":170},"Entry point. Loads data, runs SQL, produces DataFrames.",{"type":39,"tag":158,"props":172,"children":173},{},[174],{"type":39,"tag":69,"props":175,"children":177},{"className":176},[],[178],{"type":45,"value":179},"from datafusion import SessionContext",{"type":39,"tag":131,"props":181,"children":182},{},[183,192,197],{"type":39,"tag":158,"props":184,"children":185},{},[186],{"type":39,"tag":69,"props":187,"children":189},{"className":188},[],[190],{"type":45,"value":191},"DataFrame",{"type":39,"tag":158,"props":193,"children":194},{},[195],{"type":45,"value":196},"Lazy query builder. Each method returns a new DataFrame.",{"type":39,"tag":158,"props":198,"children":199},{},[200],{"type":45,"value":201},"Returned by context methods",{"type":39,"tag":131,"props":203,"children":204},{},[205,214,219],{"type":39,"tag":158,"props":206,"children":207},{},[208],{"type":39,"tag":69,"props":209,"children":211},{"className":210},[],[212],{"type":45,"value":213},"Expr",{"type":39,"tag":158,"props":215,"children":216},{},[217],{"type":45,"value":218},"Expression tree node (column ref, literal, function call, ...).",{"type":39,"tag":158,"props":220,"children":221},{},[222],{"type":39,"tag":69,"props":223,"children":225},{"className":224},[],[226],{"type":45,"value":227},"from datafusion import col, lit",{"type":39,"tag":131,"props":229,"children":230},{},[231,240,245],{"type":39,"tag":158,"props":232,"children":233},{},[234],{"type":39,"tag":69,"props":235,"children":237},{"className":236},[],[238],{"type":45,"value":239},"functions",{"type":39,"tag":158,"props":241,"children":242},{},[243],{"type":45,"value":244},"290+ built-in scalar, aggregate, and window functions.",{"type":39,"tag":158,"props":246,"children":247},{},[248],{"type":39,"tag":69,"props":249,"children":251},{"className":250},[],[252],{"type":45,"value":253},"from datafusion import functions as F",{"type":39,"tag":131,"props":255,"children":256},{},[257,266,279],{"type":39,"tag":158,"props":258,"children":259},{},[260],{"type":39,"tag":69,"props":261,"children":263},{"className":262},[],[264],{"type":45,"value":265},"functions.spark",{"type":39,"tag":158,"props":267,"children":268},{},[269,271,277],{"type":45,"value":270},"PySpark-compatible function surface (parameter names match ",{"type":39,"tag":69,"props":272,"children":274},{"className":273},[],[275],{"type":45,"value":276},"pyspark.sql.functions",{"type":45,"value":278},").",{"type":39,"tag":158,"props":280,"children":281},{},[282],{"type":39,"tag":69,"props":283,"children":285},{"className":284},[],[286],{"type":45,"value":287},"from datafusion.functions import spark",{"type":39,"tag":48,"props":289,"children":291},{"id":290},"import-conventions",[292],{"type":45,"value":293},"Import Conventions",{"type":39,"tag":295,"props":296,"children":300},"pre",{"className":297,"code":298,"language":18,"meta":299,"style":299},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from datafusion import SessionContext, col, lit\nfrom datafusion import functions as F\nfrom datafusion.functions import spark # only when porting pyspark code\n","",[301],{"type":39,"tag":69,"props":302,"children":303},{"__ignoreMap":299},[304,315,324],{"type":39,"tag":305,"props":306,"children":309},"span",{"class":307,"line":308},"line",1,[310],{"type":39,"tag":305,"props":311,"children":312},{},[313],{"type":45,"value":314},"from datafusion import SessionContext, col, lit\n",{"type":39,"tag":305,"props":316,"children":318},{"class":307,"line":317},2,[319],{"type":39,"tag":305,"props":320,"children":321},{},[322],{"type":45,"value":323},"from datafusion import functions as F\n",{"type":39,"tag":305,"props":325,"children":327},{"class":307,"line":326},3,[328],{"type":39,"tag":305,"props":329,"children":330},{},[331],{"type":45,"value":332},"from datafusion.functions import spark # only when porting pyspark code\n",{"type":39,"tag":48,"props":334,"children":336},{"id":335},"data-loading",[337],{"type":45,"value":338},"Data Loading",{"type":39,"tag":295,"props":340,"children":342},{"className":297,"code":341,"language":18,"meta":299,"style":299},"ctx = SessionContext()\n\n# From files\ndf = ctx.read_parquet(\"path\u002Fto\u002Fdata.parquet\")\ndf = ctx.read_csv(\"path\u002Fto\u002Fdata.csv\")\ndf = ctx.read_json(\"path\u002Fto\u002Fdata.json\")\n\n# From Python objects\ndf = ctx.from_pydict({\"a\": [1, 2, 3], \"b\": [\"x\", \"y\", \"z\"]})\ndf = ctx.from_pylist([{\"a\": 1, \"b\": \"x\"}, {\"a\": 2, \"b\": \"y\"}])\ndf = ctx.from_pandas(pandas_df)\ndf = ctx.from_polars(polars_df)\ndf = ctx.from_arrow(arrow_table)\ndf = ctx.read_batch(record_batch) # one pa.RecordBatch, no named table\ndf = ctx.read_batches([batch1, batch2]) # several pa.RecordBatch\n\n# From SQL\ndf = ctx.sql(\"SELECT a, b FROM my_table WHERE a > 1\")\n",[343],{"type":39,"tag":69,"props":344,"children":345},{"__ignoreMap":299},[346,354,363,371,380,389,398,406,415,424,433,442,451,460,469,478,486,495],{"type":39,"tag":305,"props":347,"children":348},{"class":307,"line":308},[349],{"type":39,"tag":305,"props":350,"children":351},{},[352],{"type":45,"value":353},"ctx = SessionContext()\n",{"type":39,"tag":305,"props":355,"children":356},{"class":307,"line":317},[357],{"type":39,"tag":305,"props":358,"children":360},{"emptyLinePlaceholder":359},true,[361],{"type":45,"value":362},"\n",{"type":39,"tag":305,"props":364,"children":365},{"class":307,"line":326},[366],{"type":39,"tag":305,"props":367,"children":368},{},[369],{"type":45,"value":370},"# From files\n",{"type":39,"tag":305,"props":372,"children":374},{"class":307,"line":373},4,[375],{"type":39,"tag":305,"props":376,"children":377},{},[378],{"type":45,"value":379},"df = ctx.read_parquet(\"path\u002Fto\u002Fdata.parquet\")\n",{"type":39,"tag":305,"props":381,"children":383},{"class":307,"line":382},5,[384],{"type":39,"tag":305,"props":385,"children":386},{},[387],{"type":45,"value":388},"df = ctx.read_csv(\"path\u002Fto\u002Fdata.csv\")\n",{"type":39,"tag":305,"props":390,"children":392},{"class":307,"line":391},6,[393],{"type":39,"tag":305,"props":394,"children":395},{},[396],{"type":45,"value":397},"df = ctx.read_json(\"path\u002Fto\u002Fdata.json\")\n",{"type":39,"tag":305,"props":399,"children":401},{"class":307,"line":400},7,[402],{"type":39,"tag":305,"props":403,"children":404},{"emptyLinePlaceholder":359},[405],{"type":45,"value":362},{"type":39,"tag":305,"props":407,"children":409},{"class":307,"line":408},8,[410],{"type":39,"tag":305,"props":411,"children":412},{},[413],{"type":45,"value":414},"# From Python objects\n",{"type":39,"tag":305,"props":416,"children":418},{"class":307,"line":417},9,[419],{"type":39,"tag":305,"props":420,"children":421},{},[422],{"type":45,"value":423},"df = ctx.from_pydict({\"a\": [1, 2, 3], \"b\": [\"x\", \"y\", \"z\"]})\n",{"type":39,"tag":305,"props":425,"children":427},{"class":307,"line":426},10,[428],{"type":39,"tag":305,"props":429,"children":430},{},[431],{"type":45,"value":432},"df = ctx.from_pylist([{\"a\": 1, \"b\": \"x\"}, {\"a\": 2, \"b\": \"y\"}])\n",{"type":39,"tag":305,"props":434,"children":436},{"class":307,"line":435},11,[437],{"type":39,"tag":305,"props":438,"children":439},{},[440],{"type":45,"value":441},"df = ctx.from_pandas(pandas_df)\n",{"type":39,"tag":305,"props":443,"children":445},{"class":307,"line":444},12,[446],{"type":39,"tag":305,"props":447,"children":448},{},[449],{"type":45,"value":450},"df = ctx.from_polars(polars_df)\n",{"type":39,"tag":305,"props":452,"children":454},{"class":307,"line":453},13,[455],{"type":39,"tag":305,"props":456,"children":457},{},[458],{"type":45,"value":459},"df = ctx.from_arrow(arrow_table)\n",{"type":39,"tag":305,"props":461,"children":463},{"class":307,"line":462},14,[464],{"type":39,"tag":305,"props":465,"children":466},{},[467],{"type":45,"value":468},"df = ctx.read_batch(record_batch) # one pa.RecordBatch, no named table\n",{"type":39,"tag":305,"props":470,"children":472},{"class":307,"line":471},15,[473],{"type":39,"tag":305,"props":474,"children":475},{},[476],{"type":45,"value":477},"df = ctx.read_batches([batch1, batch2]) # several pa.RecordBatch\n",{"type":39,"tag":305,"props":479,"children":481},{"class":307,"line":480},16,[482],{"type":39,"tag":305,"props":483,"children":484},{"emptyLinePlaceholder":359},[485],{"type":45,"value":362},{"type":39,"tag":305,"props":487,"children":489},{"class":307,"line":488},17,[490],{"type":39,"tag":305,"props":491,"children":492},{},[493],{"type":45,"value":494},"# From SQL\n",{"type":39,"tag":305,"props":496,"children":498},{"class":307,"line":497},18,[499],{"type":39,"tag":305,"props":500,"children":501},{},[502],{"type":45,"value":503},"df = ctx.sql(\"SELECT a, b FROM my_table WHERE a > 1\")\n",{"type":39,"tag":55,"props":505,"children":506},{},[507],{"type":45,"value":508},"To make a DataFrame queryable by name in SQL, register it first:",{"type":39,"tag":295,"props":510,"children":512},{"className":297,"code":511,"language":18,"meta":299,"style":299},"ctx.register_parquet(\"my_table\", \"path\u002Fto\u002Fdata.parquet\")\nctx.register_csv(\"my_table\", \"path\u002Fto\u002Fdata.csv\")\n",[513],{"type":39,"tag":69,"props":514,"children":515},{"__ignoreMap":299},[516,524],{"type":39,"tag":305,"props":517,"children":518},{"class":307,"line":308},[519],{"type":39,"tag":305,"props":520,"children":521},{},[522],{"type":45,"value":523},"ctx.register_parquet(\"my_table\", \"path\u002Fto\u002Fdata.parquet\")\n",{"type":39,"tag":305,"props":525,"children":526},{"class":307,"line":317},[527],{"type":39,"tag":305,"props":528,"children":529},{},[530],{"type":45,"value":531},"ctx.register_csv(\"my_table\", \"path\u002Fto\u002Fdata.csv\")\n",{"type":39,"tag":48,"props":533,"children":535},{"id":534},"dataframe-operations-quick-reference",[536],{"type":45,"value":537},"DataFrame Operations Quick Reference",{"type":39,"tag":55,"props":539,"children":540},{},[541,543,548],{"type":45,"value":542},"Every method returns a ",{"type":39,"tag":61,"props":544,"children":545},{},[546],{"type":45,"value":547},"new",{"type":45,"value":549}," DataFrame (immutable\u002Flazy). Chain them fluently.",{"type":39,"tag":551,"props":552,"children":554},"h3",{"id":553},"projection",[555],{"type":45,"value":556},"Projection",{"type":39,"tag":295,"props":558,"children":560},{"className":297,"code":559,"language":18,"meta":299,"style":299},"df.select(\"a\", \"b\") # preferred: plain names as strings\ndf.select(col(\"a\"), (col(\"b\") + 1).alias(\"b_plus_1\")) # use col()\u002FExpr only when you need an expression\n\ndf.with_column(\"new_col\", col(\"a\") + lit(10)) # add one column\ndf.with_columns(\n col(\"a\").alias(\"x\"),\n y=col(\"b\") + lit(1), # named keyword form\n)\n\ndf.drop(\"unwanted_col\")\ndf.with_column_renamed(\"old_name\", \"new_name\")\n",[561],{"type":39,"tag":69,"props":562,"children":563},{"__ignoreMap":299},[564,572,580,587,595,603,611,619,627,634,642],{"type":39,"tag":305,"props":565,"children":566},{"class":307,"line":308},[567],{"type":39,"tag":305,"props":568,"children":569},{},[570],{"type":45,"value":571},"df.select(\"a\", \"b\") # preferred: plain names as strings\n",{"type":39,"tag":305,"props":573,"children":574},{"class":307,"line":317},[575],{"type":39,"tag":305,"props":576,"children":577},{},[578],{"type":45,"value":579},"df.select(col(\"a\"), (col(\"b\") + 1).alias(\"b_plus_1\")) # use col()\u002FExpr only when you need an expression\n",{"type":39,"tag":305,"props":581,"children":582},{"class":307,"line":326},[583],{"type":39,"tag":305,"props":584,"children":585},{"emptyLinePlaceholder":359},[586],{"type":45,"value":362},{"type":39,"tag":305,"props":588,"children":589},{"class":307,"line":373},[590],{"type":39,"tag":305,"props":591,"children":592},{},[593],{"type":45,"value":594},"df.with_column(\"new_col\", col(\"a\") + lit(10)) # add one column\n",{"type":39,"tag":305,"props":596,"children":597},{"class":307,"line":382},[598],{"type":39,"tag":305,"props":599,"children":600},{},[601],{"type":45,"value":602},"df.with_columns(\n",{"type":39,"tag":305,"props":604,"children":605},{"class":307,"line":391},[606],{"type":39,"tag":305,"props":607,"children":608},{},[609],{"type":45,"value":610}," col(\"a\").alias(\"x\"),\n",{"type":39,"tag":305,"props":612,"children":613},{"class":307,"line":400},[614],{"type":39,"tag":305,"props":615,"children":616},{},[617],{"type":45,"value":618}," y=col(\"b\") + lit(1), # named keyword form\n",{"type":39,"tag":305,"props":620,"children":621},{"class":307,"line":408},[622],{"type":39,"tag":305,"props":623,"children":624},{},[625],{"type":45,"value":626},")\n",{"type":39,"tag":305,"props":628,"children":629},{"class":307,"line":417},[630],{"type":39,"tag":305,"props":631,"children":632},{"emptyLinePlaceholder":359},[633],{"type":45,"value":362},{"type":39,"tag":305,"props":635,"children":636},{"class":307,"line":426},[637],{"type":39,"tag":305,"props":638,"children":639},{},[640],{"type":45,"value":641},"df.drop(\"unwanted_col\")\n",{"type":39,"tag":305,"props":643,"children":644},{"class":307,"line":435},[645],{"type":39,"tag":305,"props":646,"children":647},{},[648],{"type":45,"value":649},"df.with_column_renamed(\"old_name\", \"new_name\")\n",{"type":39,"tag":55,"props":651,"children":652},{},[653,655,661,663,668],{"type":45,"value":654},"When a column is referenced by name alone, pass the name as a string rather\nthan wrapping it in ",{"type":39,"tag":69,"props":656,"children":658},{"className":657},[],[659],{"type":45,"value":660},"col()",{"type":45,"value":662},". Reach for ",{"type":39,"tag":69,"props":664,"children":666},{"className":665},[],[667],{"type":45,"value":660},{"type":45,"value":669}," only when the projection needs\narithmetic, aliasing, casting, or another expression operation.",{"type":39,"tag":55,"props":671,"children":672},{},[673,678,680,686,688,694,696,702,704,710,712,718],{"type":39,"tag":61,"props":674,"children":675},{},[676],{"type":45,"value":677},"Case sensitivity",{"type":45,"value":679},": both ",{"type":39,"tag":69,"props":681,"children":683},{"className":682},[],[684],{"type":45,"value":685},"select(\"Name\")",{"type":45,"value":687}," and ",{"type":39,"tag":69,"props":689,"children":691},{"className":690},[],[692],{"type":45,"value":693},"col(\"Name\")",{"type":45,"value":695}," lowercase the\nidentifier. For a column whose real name has uppercase letters, embed double\nquotes inside the string: ",{"type":39,"tag":69,"props":697,"children":699},{"className":698},[],[700],{"type":45,"value":701},"select('\"MyCol\"')",{"type":45,"value":703}," or ",{"type":39,"tag":69,"props":705,"children":707},{"className":706},[],[708],{"type":45,"value":709},"col('\"MyCol\"')",{"type":45,"value":711},". Without the\ninner quotes the lookup will fail with ",{"type":39,"tag":69,"props":713,"children":715},{"className":714},[],[716],{"type":45,"value":717},"No field named mycol",{"type":45,"value":719},".",{"type":39,"tag":551,"props":721,"children":723},{"id":722},"filtering",[724],{"type":45,"value":725},"Filtering",{"type":39,"tag":295,"props":727,"children":729},{"className":297,"code":728,"language":18,"meta":299,"style":299},"df.filter(col(\"a\") > 10)\ndf.filter(col(\"a\") > 10, col(\"b\") == \"x\") # multiple = AND\ndf.filter(\"a > 10\") # SQL expression string\n",[730],{"type":39,"tag":69,"props":731,"children":732},{"__ignoreMap":299},[733,741,749],{"type":39,"tag":305,"props":734,"children":735},{"class":307,"line":308},[736],{"type":39,"tag":305,"props":737,"children":738},{},[739],{"type":45,"value":740},"df.filter(col(\"a\") > 10)\n",{"type":39,"tag":305,"props":742,"children":743},{"class":307,"line":317},[744],{"type":39,"tag":305,"props":745,"children":746},{},[747],{"type":45,"value":748},"df.filter(col(\"a\") > 10, col(\"b\") == \"x\") # multiple = AND\n",{"type":39,"tag":305,"props":750,"children":751},{"class":307,"line":326},[752],{"type":39,"tag":305,"props":753,"children":754},{},[755],{"type":45,"value":756},"df.filter(\"a > 10\") # SQL expression string\n",{"type":39,"tag":55,"props":758,"children":759},{},[760,762,767,769,775,777,783],{"type":45,"value":761},"Raw Python values on the right-hand side of a comparison are auto-wrapped\ninto literals by the ",{"type":39,"tag":69,"props":763,"children":765},{"className":764},[],[766],{"type":45,"value":213},{"type":45,"value":768}," operators, so prefer ",{"type":39,"tag":69,"props":770,"children":772},{"className":771},[],[773],{"type":45,"value":774},"col(\"a\") > 10",{"type":45,"value":776}," over\n",{"type":39,"tag":69,"props":778,"children":780},{"className":779},[],[781],{"type":45,"value":782},"col(\"a\") > lit(10)",{"type":45,"value":784},". See the Comparisons section and pitfall #2 for the\nfull rule.",{"type":39,"tag":551,"props":786,"children":788},{"id":787},"aggregation",[789],{"type":45,"value":790},"Aggregation",{"type":39,"tag":295,"props":792,"children":794},{"className":297,"code":793,"language":18,"meta":299,"style":299},"# GROUP BY a, compute sum(b) and count(*)\ndf.aggregate([\"a\"], [F.sum(col(\"b\")), F.count(col(\"a\"))])\n\n# HAVING equivalent: use the filter keyword on the aggregate function\ndf.aggregate(\n [\"region\"],\n [F.sum(col(\"sales\"), filter=col(\"sales\") > 1000).alias(\"large_sales\")],\n)\n",[795],{"type":39,"tag":69,"props":796,"children":797},{"__ignoreMap":299},[798,806,814,821,829,837,845,853],{"type":39,"tag":305,"props":799,"children":800},{"class":307,"line":308},[801],{"type":39,"tag":305,"props":802,"children":803},{},[804],{"type":45,"value":805},"# GROUP BY a, compute sum(b) and count(*)\n",{"type":39,"tag":305,"props":807,"children":808},{"class":307,"line":317},[809],{"type":39,"tag":305,"props":810,"children":811},{},[812],{"type":45,"value":813},"df.aggregate([\"a\"], [F.sum(col(\"b\")), F.count(col(\"a\"))])\n",{"type":39,"tag":305,"props":815,"children":816},{"class":307,"line":326},[817],{"type":39,"tag":305,"props":818,"children":819},{"emptyLinePlaceholder":359},[820],{"type":45,"value":362},{"type":39,"tag":305,"props":822,"children":823},{"class":307,"line":373},[824],{"type":39,"tag":305,"props":825,"children":826},{},[827],{"type":45,"value":828},"# HAVING equivalent: use the filter keyword on the aggregate function\n",{"type":39,"tag":305,"props":830,"children":831},{"class":307,"line":382},[832],{"type":39,"tag":305,"props":833,"children":834},{},[835],{"type":45,"value":836},"df.aggregate(\n",{"type":39,"tag":305,"props":838,"children":839},{"class":307,"line":391},[840],{"type":39,"tag":305,"props":841,"children":842},{},[843],{"type":45,"value":844}," [\"region\"],\n",{"type":39,"tag":305,"props":846,"children":847},{"class":307,"line":400},[848],{"type":39,"tag":305,"props":849,"children":850},{},[851],{"type":45,"value":852}," [F.sum(col(\"sales\"), filter=col(\"sales\") > 1000).alias(\"large_sales\")],\n",{"type":39,"tag":305,"props":854,"children":855},{"class":307,"line":408},[856],{"type":39,"tag":305,"props":857,"children":858},{},[859],{"type":45,"value":626},{"type":39,"tag":55,"props":861,"children":862},{},[863,865,871,873,879],{"type":45,"value":864},"As with ",{"type":39,"tag":69,"props":866,"children":868},{"className":867},[],[869],{"type":45,"value":870},"select()",{"type":45,"value":872},", group keys can be passed as plain name strings. Reach for\n",{"type":39,"tag":69,"props":874,"children":876},{"className":875},[],[877],{"type":45,"value":878},"col(...)",{"type":45,"value":880}," only when the grouping expression needs arithmetic, aliasing,\ncasting, or another expression operation.",{"type":39,"tag":55,"props":882,"children":883},{},[884,886,892],{"type":45,"value":885},"Most aggregate functions accept an optional ",{"type":39,"tag":69,"props":887,"children":889},{"className":888},[],[890],{"type":45,"value":891},"filter",{"type":45,"value":893}," keyword argument. When\nprovided, only rows where the filter expression is true contribute to the\naggregate.",{"type":39,"tag":551,"props":895,"children":897},{"id":896},"sorting",[898],{"type":45,"value":899},"Sorting",{"type":39,"tag":295,"props":901,"children":903},{"className":297,"code":902,"language":18,"meta":299,"style":299},"df.sort(\"a\") # ascending (plain name, preferred)\ndf.sort(col(\"a\")) # ascending via col()\ndf.sort(col(\"a\").sort(ascending=False)) # descending\ndf.sort(col(\"a\").sort(nulls_first=False)) # override null placement\n\ndf.sort_by(\"a\", \"b\") # ascending-only shortcut\n",[904],{"type":39,"tag":69,"props":905,"children":906},{"__ignoreMap":299},[907,915,923,931,939,946],{"type":39,"tag":305,"props":908,"children":909},{"class":307,"line":308},[910],{"type":39,"tag":305,"props":911,"children":912},{},[913],{"type":45,"value":914},"df.sort(\"a\") # ascending (plain name, preferred)\n",{"type":39,"tag":305,"props":916,"children":917},{"class":307,"line":317},[918],{"type":39,"tag":305,"props":919,"children":920},{},[921],{"type":45,"value":922},"df.sort(col(\"a\")) # ascending via col()\n",{"type":39,"tag":305,"props":924,"children":925},{"class":307,"line":326},[926],{"type":39,"tag":305,"props":927,"children":928},{},[929],{"type":45,"value":930},"df.sort(col(\"a\").sort(ascending=False)) # descending\n",{"type":39,"tag":305,"props":932,"children":933},{"class":307,"line":373},[934],{"type":39,"tag":305,"props":935,"children":936},{},[937],{"type":45,"value":938},"df.sort(col(\"a\").sort(nulls_first=False)) # override null placement\n",{"type":39,"tag":305,"props":940,"children":941},{"class":307,"line":382},[942],{"type":39,"tag":305,"props":943,"children":944},{"emptyLinePlaceholder":359},[945],{"type":45,"value":362},{"type":39,"tag":305,"props":947,"children":948},{"class":307,"line":391},[949],{"type":39,"tag":305,"props":950,"children":951},{},[952],{"type":45,"value":953},"df.sort_by(\"a\", \"b\") # ascending-only shortcut\n",{"type":39,"tag":55,"props":955,"children":956},{},[957,958,963,964,970,972,978,980,986,988,994],{"type":45,"value":864},{"type":39,"tag":69,"props":959,"children":961},{"className":960},[],[962],{"type":45,"value":870},{"type":45,"value":687},{"type":39,"tag":69,"props":965,"children":967},{"className":966},[],[968],{"type":45,"value":969},"aggregate()",{"type":45,"value":971},", bare column references can be passed as\nplain name strings. A plain expression passed to ",{"type":39,"tag":69,"props":973,"children":975},{"className":974},[],[976],{"type":45,"value":977},"sort()",{"type":45,"value":979}," is already treated\nas ascending, so reach for ",{"type":39,"tag":69,"props":981,"children":983},{"className":982},[],[984],{"type":45,"value":985},"col(...).sort(...)",{"type":45,"value":987}," only when you need to override\na default (descending order or null placement). Writing\n",{"type":39,"tag":69,"props":989,"children":991},{"className":990},[],[992],{"type":45,"value":993},"col(\"a\").sort(ascending=True)",{"type":45,"value":995}," is redundant.",{"type":39,"tag":55,"props":997,"children":998},{},[999,1001,1007,1009,1015],{"type":45,"value":1000},"For ascending-only sorts with no null-placement override, ",{"type":39,"tag":69,"props":1002,"children":1004},{"className":1003},[],[1005],{"type":45,"value":1006},"df.sort_by(...)",{"type":45,"value":1008}," is\na shorter alias for ",{"type":39,"tag":69,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":45,"value":1014},"df.sort(...)",{"type":45,"value":719},{"type":39,"tag":551,"props":1017,"children":1019},{"id":1018},"joining",[1020],{"type":45,"value":1021},"Joining",{"type":39,"tag":295,"props":1023,"children":1025},{"className":297,"code":1024,"language":18,"meta":299,"style":299},"# Equi-join on shared column name\ndf1.join(df2, on=\"key\")\ndf1.join(df2, on=\"key\", how=\"left\")\n\n# Different column names\ndf1.join(df2, left_on=\"id\", right_on=\"fk_id\", how=\"inner\")\n\n# Expression-based join (supports inequality predicates)\ndf1.join_on(df2, col(\"a\") == col(\"b\"), how=\"inner\")\n\n# Semi join: keep rows from left where a match exists in right (like EXISTS)\ndf1.join(df2, on=\"key\", how=\"semi\")\n\n# Anti join: keep rows from left where NO match exists in right (like NOT EXISTS)\ndf1.join(df2, on=\"key\", how=\"anti\")\n",[1026],{"type":39,"tag":69,"props":1027,"children":1028},{"__ignoreMap":299},[1029,1037,1045,1053,1060,1068,1076,1083,1091,1099,1106,1114,1122,1129,1137],{"type":39,"tag":305,"props":1030,"children":1031},{"class":307,"line":308},[1032],{"type":39,"tag":305,"props":1033,"children":1034},{},[1035],{"type":45,"value":1036},"# Equi-join on shared column name\n",{"type":39,"tag":305,"props":1038,"children":1039},{"class":307,"line":317},[1040],{"type":39,"tag":305,"props":1041,"children":1042},{},[1043],{"type":45,"value":1044},"df1.join(df2, on=\"key\")\n",{"type":39,"tag":305,"props":1046,"children":1047},{"class":307,"line":326},[1048],{"type":39,"tag":305,"props":1049,"children":1050},{},[1051],{"type":45,"value":1052},"df1.join(df2, on=\"key\", how=\"left\")\n",{"type":39,"tag":305,"props":1054,"children":1055},{"class":307,"line":373},[1056],{"type":39,"tag":305,"props":1057,"children":1058},{"emptyLinePlaceholder":359},[1059],{"type":45,"value":362},{"type":39,"tag":305,"props":1061,"children":1062},{"class":307,"line":382},[1063],{"type":39,"tag":305,"props":1064,"children":1065},{},[1066],{"type":45,"value":1067},"# Different column names\n",{"type":39,"tag":305,"props":1069,"children":1070},{"class":307,"line":391},[1071],{"type":39,"tag":305,"props":1072,"children":1073},{},[1074],{"type":45,"value":1075},"df1.join(df2, left_on=\"id\", right_on=\"fk_id\", how=\"inner\")\n",{"type":39,"tag":305,"props":1077,"children":1078},{"class":307,"line":400},[1079],{"type":39,"tag":305,"props":1080,"children":1081},{"emptyLinePlaceholder":359},[1082],{"type":45,"value":362},{"type":39,"tag":305,"props":1084,"children":1085},{"class":307,"line":408},[1086],{"type":39,"tag":305,"props":1087,"children":1088},{},[1089],{"type":45,"value":1090},"# Expression-based join (supports inequality predicates)\n",{"type":39,"tag":305,"props":1092,"children":1093},{"class":307,"line":417},[1094],{"type":39,"tag":305,"props":1095,"children":1096},{},[1097],{"type":45,"value":1098},"df1.join_on(df2, col(\"a\") == col(\"b\"), how=\"inner\")\n",{"type":39,"tag":305,"props":1100,"children":1101},{"class":307,"line":426},[1102],{"type":39,"tag":305,"props":1103,"children":1104},{"emptyLinePlaceholder":359},[1105],{"type":45,"value":362},{"type":39,"tag":305,"props":1107,"children":1108},{"class":307,"line":435},[1109],{"type":39,"tag":305,"props":1110,"children":1111},{},[1112],{"type":45,"value":1113},"# Semi join: keep rows from left where a match exists in right (like EXISTS)\n",{"type":39,"tag":305,"props":1115,"children":1116},{"class":307,"line":444},[1117],{"type":39,"tag":305,"props":1118,"children":1119},{},[1120],{"type":45,"value":1121},"df1.join(df2, on=\"key\", how=\"semi\")\n",{"type":39,"tag":305,"props":1123,"children":1124},{"class":307,"line":453},[1125],{"type":39,"tag":305,"props":1126,"children":1127},{"emptyLinePlaceholder":359},[1128],{"type":45,"value":362},{"type":39,"tag":305,"props":1130,"children":1131},{"class":307,"line":462},[1132],{"type":39,"tag":305,"props":1133,"children":1134},{},[1135],{"type":45,"value":1136},"# Anti join: keep rows from left where NO match exists in right (like NOT EXISTS)\n",{"type":39,"tag":305,"props":1138,"children":1139},{"class":307,"line":471},[1140],{"type":39,"tag":305,"props":1141,"children":1142},{},[1143],{"type":45,"value":1144},"df1.join(df2, on=\"key\", how=\"anti\")\n",{"type":39,"tag":55,"props":1146,"children":1147},{},[1148,1150,1156,1158,1164,1165,1171,1172,1178,1179,1185,1186,1192],{"type":45,"value":1149},"Join types: ",{"type":39,"tag":69,"props":1151,"children":1153},{"className":1152},[],[1154],{"type":45,"value":1155},"\"inner\"",{"type":45,"value":1157},", ",{"type":39,"tag":69,"props":1159,"children":1161},{"className":1160},[],[1162],{"type":45,"value":1163},"\"left\"",{"type":45,"value":1157},{"type":39,"tag":69,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":45,"value":1170},"\"right\"",{"type":45,"value":1157},{"type":39,"tag":69,"props":1173,"children":1175},{"className":1174},[],[1176],{"type":45,"value":1177},"\"full\"",{"type":45,"value":1157},{"type":39,"tag":69,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":45,"value":1184},"\"semi\"",{"type":45,"value":1157},{"type":39,"tag":69,"props":1187,"children":1189},{"className":1188},[],[1190],{"type":45,"value":1191},"\"anti\"",{"type":45,"value":719},{"type":39,"tag":55,"props":1194,"children":1195},{},[1196,1198,1204,1206,1212,1213,1219,1221,1227],{"type":45,"value":1197},"Inner is the default ",{"type":39,"tag":69,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":45,"value":1203},"how",{"type":45,"value":1205},". Prefer ",{"type":39,"tag":69,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":45,"value":1211},"df1.join(df2, on=\"key\")",{"type":45,"value":776},{"type":39,"tag":69,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":45,"value":1218},"df1.join(df2, on=\"key\", how=\"inner\")",{"type":45,"value":1220}," — drop ",{"type":39,"tag":69,"props":1222,"children":1224},{"className":1223},[],[1225],{"type":45,"value":1226},"how=",{"type":45,"value":1228}," unless you need a\nnon-inner join type.",{"type":39,"tag":55,"props":1230,"children":1231},{},[1232,1234,1240,1242,1248],{"type":45,"value":1233},"When the two sides' join columns have different native names, use\n",{"type":39,"tag":69,"props":1235,"children":1237},{"className":1236},[],[1238],{"type":45,"value":1239},"left_on=",{"type":45,"value":1241},"\u002F",{"type":39,"tag":69,"props":1243,"children":1245},{"className":1244},[],[1246],{"type":45,"value":1247},"right_on=",{"type":45,"value":1249}," with the original names rather than aliasing one side\nto match the other — see pitfall #7.",{"type":39,"tag":551,"props":1251,"children":1253},{"id":1252},"window-functions",[1254],{"type":45,"value":1255},"Window Functions",{"type":39,"tag":295,"props":1257,"children":1259},{"className":297,"code":1258,"language":18,"meta":299,"style":299},"from datafusion import WindowFrame\n\n# Row number partitioned by group, ordered by value\ndf.window(\n F.row_number(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\")],\n ).alias(\"rn\")\n)\n\n# Using a Window object for reuse\nfrom datafusion.expr import Window\n\nwin = Window(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\").sort(ascending=True)],\n)\ndf.select(\n col(\"group\"),\n col(\"value\"),\n F.sum(col(\"value\")).over(win).alias(\"running_total\"),\n)\n\n# With explicit frame bounds\nwin = Window(\n partition_by=[col(\"group\")],\n order_by=[col(\"value\").sort(ascending=True)],\n window_frame=WindowFrame(\"rows\", 0, None), # current row to unbounded following\n)\n",[1260],{"type":39,"tag":69,"props":1261,"children":1262},{"__ignoreMap":299},[1263,1271,1278,1286,1294,1302,1310,1318,1326,1333,1340,1348,1356,1363,1371,1379,1387,1394,1402,1411,1420,1429,1437,1445,1454,1462,1470,1478,1487],{"type":39,"tag":305,"props":1264,"children":1265},{"class":307,"line":308},[1266],{"type":39,"tag":305,"props":1267,"children":1268},{},[1269],{"type":45,"value":1270},"from datafusion import WindowFrame\n",{"type":39,"tag":305,"props":1272,"children":1273},{"class":307,"line":317},[1274],{"type":39,"tag":305,"props":1275,"children":1276},{"emptyLinePlaceholder":359},[1277],{"type":45,"value":362},{"type":39,"tag":305,"props":1279,"children":1280},{"class":307,"line":326},[1281],{"type":39,"tag":305,"props":1282,"children":1283},{},[1284],{"type":45,"value":1285},"# Row number partitioned by group, ordered by value\n",{"type":39,"tag":305,"props":1287,"children":1288},{"class":307,"line":373},[1289],{"type":39,"tag":305,"props":1290,"children":1291},{},[1292],{"type":45,"value":1293},"df.window(\n",{"type":39,"tag":305,"props":1295,"children":1296},{"class":307,"line":382},[1297],{"type":39,"tag":305,"props":1298,"children":1299},{},[1300],{"type":45,"value":1301}," F.row_number(\n",{"type":39,"tag":305,"props":1303,"children":1304},{"class":307,"line":391},[1305],{"type":39,"tag":305,"props":1306,"children":1307},{},[1308],{"type":45,"value":1309}," partition_by=[col(\"group\")],\n",{"type":39,"tag":305,"props":1311,"children":1312},{"class":307,"line":400},[1313],{"type":39,"tag":305,"props":1314,"children":1315},{},[1316],{"type":45,"value":1317}," order_by=[col(\"value\")],\n",{"type":39,"tag":305,"props":1319,"children":1320},{"class":307,"line":408},[1321],{"type":39,"tag":305,"props":1322,"children":1323},{},[1324],{"type":45,"value":1325}," ).alias(\"rn\")\n",{"type":39,"tag":305,"props":1327,"children":1328},{"class":307,"line":417},[1329],{"type":39,"tag":305,"props":1330,"children":1331},{},[1332],{"type":45,"value":626},{"type":39,"tag":305,"props":1334,"children":1335},{"class":307,"line":426},[1336],{"type":39,"tag":305,"props":1337,"children":1338},{"emptyLinePlaceholder":359},[1339],{"type":45,"value":362},{"type":39,"tag":305,"props":1341,"children":1342},{"class":307,"line":435},[1343],{"type":39,"tag":305,"props":1344,"children":1345},{},[1346],{"type":45,"value":1347},"# Using a Window object for reuse\n",{"type":39,"tag":305,"props":1349,"children":1350},{"class":307,"line":444},[1351],{"type":39,"tag":305,"props":1352,"children":1353},{},[1354],{"type":45,"value":1355},"from datafusion.expr import Window\n",{"type":39,"tag":305,"props":1357,"children":1358},{"class":307,"line":453},[1359],{"type":39,"tag":305,"props":1360,"children":1361},{"emptyLinePlaceholder":359},[1362],{"type":45,"value":362},{"type":39,"tag":305,"props":1364,"children":1365},{"class":307,"line":462},[1366],{"type":39,"tag":305,"props":1367,"children":1368},{},[1369],{"type":45,"value":1370},"win = Window(\n",{"type":39,"tag":305,"props":1372,"children":1373},{"class":307,"line":471},[1374],{"type":39,"tag":305,"props":1375,"children":1376},{},[1377],{"type":45,"value":1378}," partition_by=[col(\"group\")],\n",{"type":39,"tag":305,"props":1380,"children":1381},{"class":307,"line":480},[1382],{"type":39,"tag":305,"props":1383,"children":1384},{},[1385],{"type":45,"value":1386}," order_by=[col(\"value\").sort(ascending=True)],\n",{"type":39,"tag":305,"props":1388,"children":1389},{"class":307,"line":488},[1390],{"type":39,"tag":305,"props":1391,"children":1392},{},[1393],{"type":45,"value":626},{"type":39,"tag":305,"props":1395,"children":1396},{"class":307,"line":497},[1397],{"type":39,"tag":305,"props":1398,"children":1399},{},[1400],{"type":45,"value":1401},"df.select(\n",{"type":39,"tag":305,"props":1403,"children":1405},{"class":307,"line":1404},19,[1406],{"type":39,"tag":305,"props":1407,"children":1408},{},[1409],{"type":45,"value":1410}," col(\"group\"),\n",{"type":39,"tag":305,"props":1412,"children":1414},{"class":307,"line":1413},20,[1415],{"type":39,"tag":305,"props":1416,"children":1417},{},[1418],{"type":45,"value":1419}," col(\"value\"),\n",{"type":39,"tag":305,"props":1421,"children":1423},{"class":307,"line":1422},21,[1424],{"type":39,"tag":305,"props":1425,"children":1426},{},[1427],{"type":45,"value":1428}," F.sum(col(\"value\")).over(win).alias(\"running_total\"),\n",{"type":39,"tag":305,"props":1430,"children":1432},{"class":307,"line":1431},22,[1433],{"type":39,"tag":305,"props":1434,"children":1435},{},[1436],{"type":45,"value":626},{"type":39,"tag":305,"props":1438,"children":1440},{"class":307,"line":1439},23,[1441],{"type":39,"tag":305,"props":1442,"children":1443},{"emptyLinePlaceholder":359},[1444],{"type":45,"value":362},{"type":39,"tag":305,"props":1446,"children":1448},{"class":307,"line":1447},24,[1449],{"type":39,"tag":305,"props":1450,"children":1451},{},[1452],{"type":45,"value":1453},"# With explicit frame bounds\n",{"type":39,"tag":305,"props":1455,"children":1457},{"class":307,"line":1456},25,[1458],{"type":39,"tag":305,"props":1459,"children":1460},{},[1461],{"type":45,"value":1370},{"type":39,"tag":305,"props":1463,"children":1465},{"class":307,"line":1464},26,[1466],{"type":39,"tag":305,"props":1467,"children":1468},{},[1469],{"type":45,"value":1378},{"type":39,"tag":305,"props":1471,"children":1473},{"class":307,"line":1472},27,[1474],{"type":39,"tag":305,"props":1475,"children":1476},{},[1477],{"type":45,"value":1386},{"type":39,"tag":305,"props":1479,"children":1481},{"class":307,"line":1480},28,[1482],{"type":39,"tag":305,"props":1483,"children":1484},{},[1485],{"type":45,"value":1486}," window_frame=WindowFrame(\"rows\", 0, None), # current row to unbounded following\n",{"type":39,"tag":305,"props":1488,"children":1490},{"class":307,"line":1489},29,[1491],{"type":39,"tag":305,"props":1492,"children":1493},{},[1494],{"type":45,"value":626},{"type":39,"tag":551,"props":1496,"children":1498},{"id":1497},"set-operations",[1499],{"type":45,"value":1500},"Set Operations",{"type":39,"tag":295,"props":1502,"children":1504},{"className":297,"code":1503,"language":18,"meta":299,"style":299},"df1.union(df2) # UNION ALL (by position)\ndf1.union(df2, distinct=True) # UNION DISTINCT\ndf1.union_by_name(df2) # match columns by name, not position\ndf1.intersect(df2) # INTERSECT ALL\ndf1.intersect(df2, distinct=True) # INTERSECT (distinct)\ndf1.except_all(df2) # EXCEPT ALL\ndf1.except_all(df2, distinct=True) # EXCEPT (distinct)\n",[1505],{"type":39,"tag":69,"props":1506,"children":1507},{"__ignoreMap":299},[1508,1516,1524,1532,1540,1548,1556],{"type":39,"tag":305,"props":1509,"children":1510},{"class":307,"line":308},[1511],{"type":39,"tag":305,"props":1512,"children":1513},{},[1514],{"type":45,"value":1515},"df1.union(df2) # UNION ALL (by position)\n",{"type":39,"tag":305,"props":1517,"children":1518},{"class":307,"line":317},[1519],{"type":39,"tag":305,"props":1520,"children":1521},{},[1522],{"type":45,"value":1523},"df1.union(df2, distinct=True) # UNION DISTINCT\n",{"type":39,"tag":305,"props":1525,"children":1526},{"class":307,"line":326},[1527],{"type":39,"tag":305,"props":1528,"children":1529},{},[1530],{"type":45,"value":1531},"df1.union_by_name(df2) # match columns by name, not position\n",{"type":39,"tag":305,"props":1533,"children":1534},{"class":307,"line":373},[1535],{"type":39,"tag":305,"props":1536,"children":1537},{},[1538],{"type":45,"value":1539},"df1.intersect(df2) # INTERSECT ALL\n",{"type":39,"tag":305,"props":1541,"children":1542},{"class":307,"line":382},[1543],{"type":39,"tag":305,"props":1544,"children":1545},{},[1546],{"type":45,"value":1547},"df1.intersect(df2, distinct=True) # INTERSECT (distinct)\n",{"type":39,"tag":305,"props":1549,"children":1550},{"class":307,"line":391},[1551],{"type":39,"tag":305,"props":1552,"children":1553},{},[1554],{"type":45,"value":1555},"df1.except_all(df2) # EXCEPT ALL\n",{"type":39,"tag":305,"props":1557,"children":1558},{"class":307,"line":400},[1559],{"type":39,"tag":305,"props":1560,"children":1561},{},[1562],{"type":45,"value":1563},"df1.except_all(df2, distinct=True) # EXCEPT (distinct)\n",{"type":39,"tag":551,"props":1565,"children":1567},{"id":1566},"limit-and-offset",[1568],{"type":45,"value":1569},"Limit and Offset",{"type":39,"tag":295,"props":1571,"children":1573},{"className":297,"code":1572,"language":18,"meta":299,"style":299},"df.limit(10) # first 10 rows\ndf.limit(10, offset=20) # skip 20, then take 10\n",[1574],{"type":39,"tag":69,"props":1575,"children":1576},{"__ignoreMap":299},[1577,1585],{"type":39,"tag":305,"props":1578,"children":1579},{"class":307,"line":308},[1580],{"type":39,"tag":305,"props":1581,"children":1582},{},[1583],{"type":45,"value":1584},"df.limit(10) # first 10 rows\n",{"type":39,"tag":305,"props":1586,"children":1587},{"class":307,"line":317},[1588],{"type":39,"tag":305,"props":1589,"children":1590},{},[1591],{"type":45,"value":1592},"df.limit(10, offset=20) # skip 20, then take 10\n",{"type":39,"tag":551,"props":1594,"children":1596},{"id":1595},"deduplication",[1597],{"type":45,"value":1598},"Deduplication",{"type":39,"tag":295,"props":1600,"children":1602},{"className":297,"code":1601,"language":18,"meta":299,"style":299},"df.distinct() # remove duplicate rows\ndf.distinct_on( # keep first row per group (like DISTINCT ON in Postgres)\n [col(\"a\")], # uniqueness columns\n [col(\"a\"), col(\"b\")], # output columns\n [col(\"b\").sort(ascending=True)], # which row to keep\n)\n",[1603],{"type":39,"tag":69,"props":1604,"children":1605},{"__ignoreMap":299},[1606,1614,1622,1630,1638,1646],{"type":39,"tag":305,"props":1607,"children":1608},{"class":307,"line":308},[1609],{"type":39,"tag":305,"props":1610,"children":1611},{},[1612],{"type":45,"value":1613},"df.distinct() # remove duplicate rows\n",{"type":39,"tag":305,"props":1615,"children":1616},{"class":307,"line":317},[1617],{"type":39,"tag":305,"props":1618,"children":1619},{},[1620],{"type":45,"value":1621},"df.distinct_on( # keep first row per group (like DISTINCT ON in Postgres)\n",{"type":39,"tag":305,"props":1623,"children":1624},{"class":307,"line":326},[1625],{"type":39,"tag":305,"props":1626,"children":1627},{},[1628],{"type":45,"value":1629}," [col(\"a\")], # uniqueness columns\n",{"type":39,"tag":305,"props":1631,"children":1632},{"class":307,"line":373},[1633],{"type":39,"tag":305,"props":1634,"children":1635},{},[1636],{"type":45,"value":1637}," [col(\"a\"), col(\"b\")], # output columns\n",{"type":39,"tag":305,"props":1639,"children":1640},{"class":307,"line":382},[1641],{"type":39,"tag":305,"props":1642,"children":1643},{},[1644],{"type":45,"value":1645}," [col(\"b\").sort(ascending=True)], # which row to keep\n",{"type":39,"tag":305,"props":1647,"children":1648},{"class":307,"line":391},[1649],{"type":39,"tag":305,"props":1650,"children":1651},{},[1652],{"type":45,"value":626},{"type":39,"tag":48,"props":1654,"children":1656},{"id":1655},"executing-and-collecting-results",[1657],{"type":45,"value":1658},"Executing and Collecting Results",{"type":39,"tag":55,"props":1660,"children":1661},{},[1662],{"type":45,"value":1663},"DataFrames are lazy until you collect.",{"type":39,"tag":295,"props":1665,"children":1667},{"className":297,"code":1666,"language":18,"meta":299,"style":299},"df.show() # print formatted table to stdout\nbatches = df.collect() # list[pa.RecordBatch]\narr = df.collect_column(\"col_name\") # pa.Array | pa.ChunkedArray (single column)\ntable = df.to_arrow_table() # pa.Table\npandas_df = df.to_pandas() # pd.DataFrame\npolars_df = df.to_polars() # pl.DataFrame\npy_dict = df.to_pydict() # dict[str, list]\npy_list = df.to_pylist() # list[dict]\ncount = df.count() # int\ndf = df.cache() # materialize in memory, return DataFrame\n",[1668],{"type":39,"tag":69,"props":1669,"children":1670},{"__ignoreMap":299},[1671,1679,1687,1695,1703,1711,1719,1727,1735,1743],{"type":39,"tag":305,"props":1672,"children":1673},{"class":307,"line":308},[1674],{"type":39,"tag":305,"props":1675,"children":1676},{},[1677],{"type":45,"value":1678},"df.show() # print formatted table to stdout\n",{"type":39,"tag":305,"props":1680,"children":1681},{"class":307,"line":317},[1682],{"type":39,"tag":305,"props":1683,"children":1684},{},[1685],{"type":45,"value":1686},"batches = df.collect() # list[pa.RecordBatch]\n",{"type":39,"tag":305,"props":1688,"children":1689},{"class":307,"line":326},[1690],{"type":39,"tag":305,"props":1691,"children":1692},{},[1693],{"type":45,"value":1694},"arr = df.collect_column(\"col_name\") # pa.Array | pa.ChunkedArray (single column)\n",{"type":39,"tag":305,"props":1696,"children":1697},{"class":307,"line":373},[1698],{"type":39,"tag":305,"props":1699,"children":1700},{},[1701],{"type":45,"value":1702},"table = df.to_arrow_table() # pa.Table\n",{"type":39,"tag":305,"props":1704,"children":1705},{"class":307,"line":382},[1706],{"type":39,"tag":305,"props":1707,"children":1708},{},[1709],{"type":45,"value":1710},"pandas_df = df.to_pandas() # pd.DataFrame\n",{"type":39,"tag":305,"props":1712,"children":1713},{"class":307,"line":391},[1714],{"type":39,"tag":305,"props":1715,"children":1716},{},[1717],{"type":45,"value":1718},"polars_df = df.to_polars() # pl.DataFrame\n",{"type":39,"tag":305,"props":1720,"children":1721},{"class":307,"line":400},[1722],{"type":39,"tag":305,"props":1723,"children":1724},{},[1725],{"type":45,"value":1726},"py_dict = df.to_pydict() # dict[str, list]\n",{"type":39,"tag":305,"props":1728,"children":1729},{"class":307,"line":408},[1730],{"type":39,"tag":305,"props":1731,"children":1732},{},[1733],{"type":45,"value":1734},"py_list = df.to_pylist() # list[dict]\n",{"type":39,"tag":305,"props":1736,"children":1737},{"class":307,"line":417},[1738],{"type":39,"tag":305,"props":1739,"children":1740},{},[1741],{"type":45,"value":1742},"count = df.count() # int\n",{"type":39,"tag":305,"props":1744,"children":1745},{"class":307,"line":426},[1746],{"type":39,"tag":305,"props":1747,"children":1748},{},[1749],{"type":45,"value":1750},"df = df.cache() # materialize in memory, return DataFrame\n",{"type":39,"tag":551,"props":1752,"children":1754},{"id":1753},"date-and-timestamp-type-conversion",[1755],{"type":45,"value":1756},"Date and Timestamp Type Conversion",{"type":39,"tag":55,"props":1758,"children":1759},{},[1760,1762,1768,1769,1775],{"type":45,"value":1761},"The Python type returned by ",{"type":39,"tag":69,"props":1763,"children":1765},{"className":1764},[],[1766],{"type":45,"value":1767},"to_pydict()",{"type":45,"value":96},{"type":39,"tag":69,"props":1770,"children":1772},{"className":1771},[],[1773],{"type":45,"value":1774},"to_pylist()",{"type":45,"value":1776}," depends on the Arrow\ncolumn type, and the mapping is inherited from PyArrow:",{"type":39,"tag":123,"props":1778,"children":1779},{},[1780,1796],{"type":39,"tag":127,"props":1781,"children":1782},{},[1783],{"type":39,"tag":131,"props":1784,"children":1785},{},[1786,1791],{"type":39,"tag":135,"props":1787,"children":1788},{},[1789],{"type":45,"value":1790},"Arrow type",{"type":39,"tag":135,"props":1792,"children":1793},{},[1794],{"type":45,"value":1795},"Python type returned",{"type":39,"tag":151,"props":1797,"children":1798},{},[1799,1834,1855,1883,1916],{"type":39,"tag":131,"props":1800,"children":1801},{},[1802,1825],{"type":39,"tag":158,"props":1803,"children":1804},{},[1805,1811,1812,1818,1819],{"type":39,"tag":69,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":45,"value":1810},"timestamp(s)",{"type":45,"value":96},{"type":39,"tag":69,"props":1813,"children":1815},{"className":1814},[],[1816],{"type":45,"value":1817},"(ms)",{"type":45,"value":96},{"type":39,"tag":69,"props":1820,"children":1822},{"className":1821},[],[1823],{"type":45,"value":1824},"(us)",{"type":39,"tag":158,"props":1826,"children":1827},{},[1828],{"type":39,"tag":69,"props":1829,"children":1831},{"className":1830},[],[1832],{"type":45,"value":1833},"datetime.datetime",{"type":39,"tag":131,"props":1835,"children":1836},{},[1837,1846],{"type":39,"tag":158,"props":1838,"children":1839},{},[1840],{"type":39,"tag":69,"props":1841,"children":1843},{"className":1842},[],[1844],{"type":45,"value":1845},"timestamp(ns)",{"type":39,"tag":158,"props":1847,"children":1848},{},[1849],{"type":39,"tag":69,"props":1850,"children":1852},{"className":1851},[],[1853],{"type":45,"value":1854},"pandas.Timestamp",{"type":39,"tag":131,"props":1856,"children":1857},{},[1858,1874],{"type":39,"tag":158,"props":1859,"children":1860},{},[1861,1867,1868],{"type":39,"tag":69,"props":1862,"children":1864},{"className":1863},[],[1865],{"type":45,"value":1866},"date32",{"type":45,"value":96},{"type":39,"tag":69,"props":1869,"children":1871},{"className":1870},[],[1872],{"type":45,"value":1873},"date64",{"type":39,"tag":158,"props":1875,"children":1876},{},[1877],{"type":39,"tag":69,"props":1878,"children":1880},{"className":1879},[],[1881],{"type":45,"value":1882},"datetime.date",{"type":39,"tag":131,"props":1884,"children":1885},{},[1886,1907],{"type":39,"tag":158,"props":1887,"children":1888},{},[1889,1895,1896,1901,1902],{"type":39,"tag":69,"props":1890,"children":1892},{"className":1891},[],[1893],{"type":45,"value":1894},"duration(s)",{"type":45,"value":96},{"type":39,"tag":69,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":45,"value":1817},{"type":45,"value":96},{"type":39,"tag":69,"props":1903,"children":1905},{"className":1904},[],[1906],{"type":45,"value":1824},{"type":39,"tag":158,"props":1908,"children":1909},{},[1910],{"type":39,"tag":69,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":45,"value":1915},"datetime.timedelta",{"type":39,"tag":131,"props":1917,"children":1918},{},[1919,1928],{"type":39,"tag":158,"props":1920,"children":1921},{},[1922],{"type":39,"tag":69,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":45,"value":1927},"duration(ns)",{"type":39,"tag":158,"props":1929,"children":1930},{},[1931],{"type":39,"tag":69,"props":1932,"children":1934},{"className":1933},[],[1935],{"type":45,"value":1936},"pandas.Timedelta",{"type":39,"tag":55,"props":1938,"children":1939},{},[1940,1942,1948,1950,1955,1956,1961,1963,1969],{"type":45,"value":1941},"The nanosecond-precision fallback to pandas types is the main surprise:\npandas is not a hard dependency of ",{"type":39,"tag":69,"props":1943,"children":1945},{"className":1944},[],[1946],{"type":45,"value":1947},"datafusion",{"type":45,"value":1949},", but PyArrow reaches for it\nwhen ",{"type":39,"tag":69,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":45,"value":1833},{"type":45,"value":96},{"type":39,"tag":69,"props":1957,"children":1959},{"className":1958},[],[1960],{"type":45,"value":1915},{"type":45,"value":1962}," would lose precision (stdlib\ntypes only go to microseconds). If you need plain stdlib types, cast to a\ncoarser unit before collecting, e.g.\n",{"type":39,"tag":69,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":45,"value":1968},"df.select(col(\"ts\").cast(pa.timestamp(\"us\")))",{"type":45,"value":719},{"type":39,"tag":55,"props":1971,"children":1972},{},[1973,1979,1981,1986,1987,1992,1994,2000,2002,2007,2009,2015,2017,2023],{"type":39,"tag":69,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":45,"value":1978},"df.to_pandas()",{"type":45,"value":1980}," has its own footgun for dates: pandas has no pure-date dtype,\nso a ",{"type":39,"tag":69,"props":1982,"children":1984},{"className":1983},[],[1985],{"type":45,"value":1866},{"type":45,"value":1241},{"type":39,"tag":69,"props":1988,"children":1990},{"className":1989},[],[1991],{"type":45,"value":1873},{"type":45,"value":1993}," column comes back as an ",{"type":39,"tag":69,"props":1995,"children":1997},{"className":1996},[],[1998],{"type":45,"value":1999},"object",{"type":45,"value":2001}," column of\n",{"type":39,"tag":69,"props":2003,"children":2005},{"className":2004},[],[2006],{"type":45,"value":1882},{"type":45,"value":2008}," values rather than ",{"type":39,"tag":69,"props":2010,"children":2012},{"className":2011},[],[2013],{"type":45,"value":2014},"datetime64[ns]",{"type":45,"value":2016},". If downstream code\nexpects a datetime column, cast on the DataFusion side first:\n",{"type":39,"tag":69,"props":2018,"children":2020},{"className":2019},[],[2021],{"type":45,"value":2022},"col(\"ship_date\").cast(pa.timestamp(\"ns\"))",{"type":45,"value":719},{"type":39,"tag":551,"props":2025,"children":2027},{"id":2026},"streaming-results",[2028],{"type":45,"value":2029},"Streaming Results",{"type":39,"tag":55,"props":2031,"children":2032},{},[2033,2035,2041,2043,2049,2051,2057],{"type":45,"value":2034},"Prefer streaming over ",{"type":39,"tag":69,"props":2036,"children":2038},{"className":2037},[],[2039],{"type":45,"value":2040},"collect()",{"type":45,"value":2042}," when the result is too large to materialize\nin memory, when you want to start processing before the query finishes, or\nwhen you may break out of the loop early. ",{"type":39,"tag":69,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":45,"value":2048},"execute_stream()",{"type":45,"value":2050}," pulls one\n",{"type":39,"tag":69,"props":2052,"children":2054},{"className":2053},[],[2055],{"type":45,"value":2056},"RecordBatch",{"type":45,"value":2058}," at a time from the execution plan rather than buffering the\nwhole result up front.",{"type":39,"tag":295,"props":2060,"children":2062},{"className":297,"code":2061,"language":18,"meta":299,"style":299},"# Single-partition stream; batch is a datafusion.RecordBatch\nstream = df.execute_stream()\nfor batch in stream:\n process(batch.to_pyarrow()) # convert to pa.RecordBatch if needed\n\n# DataFrame is iterable directly (delegates to execute_stream)\nfor batch in df:\n process(batch.to_pyarrow())\n\n# One stream per partition, for parallel consumption\nfor stream in df.execute_stream_partitioned():\n for batch in stream:\n process(batch.to_pyarrow())\n",[2063],{"type":39,"tag":69,"props":2064,"children":2065},{"__ignoreMap":299},[2066,2074,2082,2090,2098,2105,2113,2121,2129,2136,2144,2152,2160],{"type":39,"tag":305,"props":2067,"children":2068},{"class":307,"line":308},[2069],{"type":39,"tag":305,"props":2070,"children":2071},{},[2072],{"type":45,"value":2073},"# Single-partition stream; batch is a datafusion.RecordBatch\n",{"type":39,"tag":305,"props":2075,"children":2076},{"class":307,"line":317},[2077],{"type":39,"tag":305,"props":2078,"children":2079},{},[2080],{"type":45,"value":2081},"stream = df.execute_stream()\n",{"type":39,"tag":305,"props":2083,"children":2084},{"class":307,"line":326},[2085],{"type":39,"tag":305,"props":2086,"children":2087},{},[2088],{"type":45,"value":2089},"for batch in stream:\n",{"type":39,"tag":305,"props":2091,"children":2092},{"class":307,"line":373},[2093],{"type":39,"tag":305,"props":2094,"children":2095},{},[2096],{"type":45,"value":2097}," process(batch.to_pyarrow()) # convert to pa.RecordBatch if needed\n",{"type":39,"tag":305,"props":2099,"children":2100},{"class":307,"line":382},[2101],{"type":39,"tag":305,"props":2102,"children":2103},{"emptyLinePlaceholder":359},[2104],{"type":45,"value":362},{"type":39,"tag":305,"props":2106,"children":2107},{"class":307,"line":391},[2108],{"type":39,"tag":305,"props":2109,"children":2110},{},[2111],{"type":45,"value":2112},"# DataFrame is iterable directly (delegates to execute_stream)\n",{"type":39,"tag":305,"props":2114,"children":2115},{"class":307,"line":400},[2116],{"type":39,"tag":305,"props":2117,"children":2118},{},[2119],{"type":45,"value":2120},"for batch in df:\n",{"type":39,"tag":305,"props":2122,"children":2123},{"class":307,"line":408},[2124],{"type":39,"tag":305,"props":2125,"children":2126},{},[2127],{"type":45,"value":2128}," process(batch.to_pyarrow())\n",{"type":39,"tag":305,"props":2130,"children":2131},{"class":307,"line":417},[2132],{"type":39,"tag":305,"props":2133,"children":2134},{"emptyLinePlaceholder":359},[2135],{"type":45,"value":362},{"type":39,"tag":305,"props":2137,"children":2138},{"class":307,"line":426},[2139],{"type":39,"tag":305,"props":2140,"children":2141},{},[2142],{"type":45,"value":2143},"# One stream per partition, for parallel consumption\n",{"type":39,"tag":305,"props":2145,"children":2146},{"class":307,"line":435},[2147],{"type":39,"tag":305,"props":2148,"children":2149},{},[2150],{"type":45,"value":2151},"for stream in df.execute_stream_partitioned():\n",{"type":39,"tag":305,"props":2153,"children":2154},{"class":307,"line":444},[2155],{"type":39,"tag":305,"props":2156,"children":2157},{},[2158],{"type":45,"value":2159}," for batch in stream:\n",{"type":39,"tag":305,"props":2161,"children":2162},{"class":307,"line":453},[2163],{"type":39,"tag":305,"props":2164,"children":2165},{},[2166],{"type":45,"value":2167}," process(batch.to_pyarrow())\n",{"type":39,"tag":55,"props":2169,"children":2170},{},[2171,2173,2179,2181,2187],{"type":45,"value":2172},"Async iteration is also supported via ",{"type":39,"tag":69,"props":2174,"children":2176},{"className":2175},[],[2177],{"type":45,"value":2178},"async for batch in df: ...",{"type":45,"value":2180}," (or\n",{"type":39,"tag":69,"props":2182,"children":2184},{"className":2183},[],[2185],{"type":45,"value":2186},"df.execute_stream()",{"type":45,"value":2188},"), which is useful when batches are interleaved with\nother I\u002FO.",{"type":39,"tag":551,"props":2190,"children":2192},{"id":2191},"caching-intermediate-results",[2193],{"type":45,"value":2194},"Caching Intermediate Results",{"type":39,"tag":55,"props":2196,"children":2197},{},[2198,2204,2206,2212],{"type":39,"tag":69,"props":2199,"children":2201},{"className":2200},[],[2202],{"type":45,"value":2203},"df.cache()",{"type":45,"value":2205}," materializes a DataFrame as an in-memory table and returns a new\nDataFrame backed by it. Reach for it when the same intermediate result feeds\nmultiple downstream queries — without ",{"type":39,"tag":69,"props":2207,"children":2209},{"className":2208},[],[2210],{"type":45,"value":2211},"cache()",{"type":45,"value":2213},", each branch re-executes the\nfull upstream plan (re-reading files, recomputing filters\u002Faggregates).",{"type":39,"tag":295,"props":2215,"children":2217},{"className":297,"code":2216,"language":18,"meta":299,"style":299},"base = (\n ctx.read_parquet(\"orders.parquet\")\n .filter(col(\"status\") == \"shipped\")\n .cache() # materialize once, reuse below\n)\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\nby_customer = base.aggregate([\"customer\"], [F.sum(col(\"amount\")).alias(\"total\")])\n",[2218],{"type":39,"tag":69,"props":2219,"children":2220},{"__ignoreMap":299},[2221,2229,2237,2245,2253,2260,2268],{"type":39,"tag":305,"props":2222,"children":2223},{"class":307,"line":308},[2224],{"type":39,"tag":305,"props":2225,"children":2226},{},[2227],{"type":45,"value":2228},"base = (\n",{"type":39,"tag":305,"props":2230,"children":2231},{"class":307,"line":317},[2232],{"type":39,"tag":305,"props":2233,"children":2234},{},[2235],{"type":45,"value":2236}," ctx.read_parquet(\"orders.parquet\")\n",{"type":39,"tag":305,"props":2238,"children":2239},{"class":307,"line":326},[2240],{"type":39,"tag":305,"props":2241,"children":2242},{},[2243],{"type":45,"value":2244}," .filter(col(\"status\") == \"shipped\")\n",{"type":39,"tag":305,"props":2246,"children":2247},{"class":307,"line":373},[2248],{"type":39,"tag":305,"props":2249,"children":2250},{},[2251],{"type":45,"value":2252}," .cache() # materialize once, reuse below\n",{"type":39,"tag":305,"props":2254,"children":2255},{"class":307,"line":382},[2256],{"type":39,"tag":305,"props":2257,"children":2258},{},[2259],{"type":45,"value":626},{"type":39,"tag":305,"props":2261,"children":2262},{"class":307,"line":391},[2263],{"type":39,"tag":305,"props":2264,"children":2265},{},[2266],{"type":45,"value":2267},"by_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\n",{"type":39,"tag":305,"props":2269,"children":2270},{"class":307,"line":400},[2271],{"type":39,"tag":305,"props":2272,"children":2273},{},[2274],{"type":45,"value":2275},"by_customer = base.aggregate([\"customer\"], [F.sum(col(\"amount\")).alias(\"total\")])\n",{"type":39,"tag":55,"props":2277,"children":2278},{},[2279,2281,2286],{"type":45,"value":2280},"Skip ",{"type":39,"tag":69,"props":2282,"children":2284},{"className":2283},[],[2285],{"type":45,"value":2211},{"type":45,"value":2287}," for single-use DataFrames — the lazy plan is already optimal.",{"type":39,"tag":55,"props":2289,"children":2290},{},[2291,2293,2298,2300,2306],{"type":45,"value":2292},"The cached table is owned by the DataFrame returned from ",{"type":39,"tag":69,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":45,"value":2211},{"type":45,"value":2299}," (and any\nDataFrames chained from it). To free the memory, drop every reference — let\nthem go out of scope, or ",{"type":39,"tag":69,"props":2301,"children":2303},{"className":2302},[],[2304],{"type":45,"value":2305},"del base; del by_region; del by_customer",{"type":45,"value":719},{"type":39,"tag":551,"props":2308,"children":2310},{"id":2309},"writing-results",[2311],{"type":45,"value":2312},"Writing Results",{"type":39,"tag":295,"props":2314,"children":2316},{"className":297,"code":2315,"language":18,"meta":299,"style":299},"df.write_parquet(\"output.parquet\")\ndf.write_csv(\"output.csv\")\ndf.write_json(\"output.json\")\n",[2317],{"type":39,"tag":69,"props":2318,"children":2319},{"__ignoreMap":299},[2320,2328,2336],{"type":39,"tag":305,"props":2321,"children":2322},{"class":307,"line":308},[2323],{"type":39,"tag":305,"props":2324,"children":2325},{},[2326],{"type":45,"value":2327},"df.write_parquet(\"output.parquet\")\n",{"type":39,"tag":305,"props":2329,"children":2330},{"class":307,"line":317},[2331],{"type":39,"tag":305,"props":2332,"children":2333},{},[2334],{"type":45,"value":2335},"df.write_csv(\"output.csv\")\n",{"type":39,"tag":305,"props":2337,"children":2338},{"class":307,"line":326},[2339],{"type":39,"tag":305,"props":2340,"children":2341},{},[2342],{"type":45,"value":2343},"df.write_json(\"output.json\")\n",{"type":39,"tag":55,"props":2345,"children":2346},{},[2347,2349,2355],{"type":45,"value":2348},"You can also pass a directory path (e.g., ",{"type":39,"tag":69,"props":2350,"children":2352},{"className":2351},[],[2353],{"type":45,"value":2354},"\"output\u002F\"",{"type":45,"value":2356},") to write a multi-file\npartitioned output.",{"type":39,"tag":48,"props":2358,"children":2360},{"id":2359},"expression-building",[2361],{"type":45,"value":2362},"Expression Building",{"type":39,"tag":551,"props":2364,"children":2366},{"id":2365},"column-references-and-literals",[2367],{"type":45,"value":2368},"Column References and Literals",{"type":39,"tag":295,"props":2370,"children":2372},{"className":297,"code":2371,"language":18,"meta":299,"style":299},"col(\"column_name\") # reference a column\nlit(42) # integer literal\nlit(\"hello\") # string literal\nlit(3.14) # float literal\nlit(pa.scalar(value)) # PyArrow scalar (preserves Arrow type)\n",[2373],{"type":39,"tag":69,"props":2374,"children":2375},{"__ignoreMap":299},[2376,2384,2392,2400,2408],{"type":39,"tag":305,"props":2377,"children":2378},{"class":307,"line":308},[2379],{"type":39,"tag":305,"props":2380,"children":2381},{},[2382],{"type":45,"value":2383},"col(\"column_name\") # reference a column\n",{"type":39,"tag":305,"props":2385,"children":2386},{"class":307,"line":317},[2387],{"type":39,"tag":305,"props":2388,"children":2389},{},[2390],{"type":45,"value":2391},"lit(42) # integer literal\n",{"type":39,"tag":305,"props":2393,"children":2394},{"class":307,"line":326},[2395],{"type":39,"tag":305,"props":2396,"children":2397},{},[2398],{"type":45,"value":2399},"lit(\"hello\") # string literal\n",{"type":39,"tag":305,"props":2401,"children":2402},{"class":307,"line":373},[2403],{"type":39,"tag":305,"props":2404,"children":2405},{},[2406],{"type":45,"value":2407},"lit(3.14) # float literal\n",{"type":39,"tag":305,"props":2409,"children":2410},{"class":307,"line":382},[2411],{"type":39,"tag":305,"props":2412,"children":2413},{},[2414],{"type":45,"value":2415},"lit(pa.scalar(value)) # PyArrow scalar (preserves Arrow type)\n",{"type":39,"tag":55,"props":2417,"children":2418},{},[2419,2425],{"type":39,"tag":69,"props":2420,"children":2422},{"className":2421},[],[2423],{"type":45,"value":2424},"lit()",{"type":45,"value":2426}," accepts PyArrow scalars directly -- prefer this over converting Arrow\ndata to Python and back when working with values extracted from query results.",{"type":39,"tag":551,"props":2428,"children":2430},{"id":2429},"arithmetic",[2431],{"type":45,"value":2432},"Arithmetic",{"type":39,"tag":295,"props":2434,"children":2436},{"className":297,"code":2435,"language":18,"meta":299,"style":299},"col(\"price\") * col(\"quantity\") # multiplication\ncol(\"a\") + lit(1) # addition\ncol(\"a\") - col(\"b\") # subtraction\ncol(\"a\") \u002F lit(2) # division\ncol(\"a\") % lit(3) # modulo\n",[2437],{"type":39,"tag":69,"props":2438,"children":2439},{"__ignoreMap":299},[2440,2448,2456,2464,2472],{"type":39,"tag":305,"props":2441,"children":2442},{"class":307,"line":308},[2443],{"type":39,"tag":305,"props":2444,"children":2445},{},[2446],{"type":45,"value":2447},"col(\"price\") * col(\"quantity\") # multiplication\n",{"type":39,"tag":305,"props":2449,"children":2450},{"class":307,"line":317},[2451],{"type":39,"tag":305,"props":2452,"children":2453},{},[2454],{"type":45,"value":2455},"col(\"a\") + lit(1) # addition\n",{"type":39,"tag":305,"props":2457,"children":2458},{"class":307,"line":326},[2459],{"type":39,"tag":305,"props":2460,"children":2461},{},[2462],{"type":45,"value":2463},"col(\"a\") - col(\"b\") # subtraction\n",{"type":39,"tag":305,"props":2465,"children":2466},{"class":307,"line":373},[2467],{"type":39,"tag":305,"props":2468,"children":2469},{},[2470],{"type":45,"value":2471},"col(\"a\") \u002F lit(2) # division\n",{"type":39,"tag":305,"props":2473,"children":2474},{"class":307,"line":382},[2475],{"type":39,"tag":305,"props":2476,"children":2477},{},[2478],{"type":45,"value":2479},"col(\"a\") % lit(3) # modulo\n",{"type":39,"tag":551,"props":2481,"children":2483},{"id":2482},"date-arithmetic",[2484],{"type":45,"value":2485},"Date Arithmetic",{"type":39,"tag":55,"props":2487,"children":2488},{},[2489,2495,2496,2502,2504,2510,2512,2518,2520,2526,2528,2534],{"type":39,"tag":69,"props":2490,"children":2492},{"className":2491},[],[2493],{"type":45,"value":2494},"Date32",{"type":45,"value":687},{"type":39,"tag":69,"props":2497,"children":2499},{"className":2498},[],[2500],{"type":45,"value":2501},"Date64",{"type":45,"value":2503}," columns both require ",{"type":39,"tag":69,"props":2505,"children":2507},{"className":2506},[],[2508],{"type":45,"value":2509},"Interval",{"type":45,"value":2511}," types for arithmetic,\nnot ",{"type":39,"tag":69,"props":2513,"children":2515},{"className":2514},[],[2516],{"type":45,"value":2517},"Duration",{"type":45,"value":2519},". Use PyArrow's ",{"type":39,"tag":69,"props":2521,"children":2523},{"className":2522},[],[2524],{"type":45,"value":2525},"month_day_nano_interval",{"type":45,"value":2527}," type, which takes a\n",{"type":39,"tag":69,"props":2529,"children":2531},{"className":2530},[],[2532],{"type":45,"value":2533},"(months, days, nanos)",{"type":45,"value":2535}," tuple:",{"type":39,"tag":295,"props":2537,"children":2539},{"className":297,"code":2538,"language":18,"meta":299,"style":299},"import pyarrow as pa\n\n# Subtract 90 days from a date column\ncol(\"ship_date\") - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n\n# Subtract 3 months\ncol(\"ship_date\") - lit(pa.scalar((3, 0, 0), type=pa.month_day_nano_interval()))\n",[2540],{"type":39,"tag":69,"props":2541,"children":2542},{"__ignoreMap":299},[2543,2551,2558,2566,2574,2581,2589],{"type":39,"tag":305,"props":2544,"children":2545},{"class":307,"line":308},[2546],{"type":39,"tag":305,"props":2547,"children":2548},{},[2549],{"type":45,"value":2550},"import pyarrow as pa\n",{"type":39,"tag":305,"props":2552,"children":2553},{"class":307,"line":317},[2554],{"type":39,"tag":305,"props":2555,"children":2556},{"emptyLinePlaceholder":359},[2557],{"type":45,"value":362},{"type":39,"tag":305,"props":2559,"children":2560},{"class":307,"line":326},[2561],{"type":39,"tag":305,"props":2562,"children":2563},{},[2564],{"type":45,"value":2565},"# Subtract 90 days from a date column\n",{"type":39,"tag":305,"props":2567,"children":2568},{"class":307,"line":373},[2569],{"type":39,"tag":305,"props":2570,"children":2571},{},[2572],{"type":45,"value":2573},"col(\"ship_date\") - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n",{"type":39,"tag":305,"props":2575,"children":2576},{"class":307,"line":382},[2577],{"type":39,"tag":305,"props":2578,"children":2579},{"emptyLinePlaceholder":359},[2580],{"type":45,"value":362},{"type":39,"tag":305,"props":2582,"children":2583},{"class":307,"line":391},[2584],{"type":39,"tag":305,"props":2585,"children":2586},{},[2587],{"type":45,"value":2588},"# Subtract 3 months\n",{"type":39,"tag":305,"props":2590,"children":2591},{"class":307,"line":400},[2592],{"type":39,"tag":305,"props":2593,"children":2594},{},[2595],{"type":45,"value":2596},"col(\"ship_date\") - lit(pa.scalar((3, 0, 0), type=pa.month_day_nano_interval()))\n",{"type":39,"tag":55,"props":2598,"children":2599},{},[2600,2605,2607,2613,2615,2621,2623,2628,2630,2635,2636,2641,2643,2649,2650,2656,2658,2664],{"type":39,"tag":61,"props":2601,"children":2602},{},[2603],{"type":45,"value":2604},"Important",{"type":45,"value":2606},": ",{"type":39,"tag":69,"props":2608,"children":2610},{"className":2609},[],[2611],{"type":45,"value":2612},"lit(datetime.timedelta(days=90))",{"type":45,"value":2614}," creates a ",{"type":39,"tag":69,"props":2616,"children":2618},{"className":2617},[],[2619],{"type":45,"value":2620},"Duration(µs)",{"type":45,"value":2622},"\nliteral, which is ",{"type":39,"tag":61,"props":2624,"children":2625},{},[2626],{"type":45,"value":2627},"not",{"type":45,"value":2629}," compatible with ",{"type":39,"tag":69,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":45,"value":2494},{"type":45,"value":1241},{"type":39,"tag":69,"props":2637,"children":2639},{"className":2638},[],[2640],{"type":45,"value":2501},{"type":45,"value":2642}," arithmetic\n(",{"type":39,"tag":69,"props":2644,"children":2646},{"className":2645},[],[2647],{"type":45,"value":2648},"Duration(ms)",{"type":45,"value":687},{"type":39,"tag":69,"props":2651,"children":2653},{"className":2652},[],[2654],{"type":45,"value":2655},"Duration(ns)",{"type":45,"value":2657}," are rejected too). Always use\n",{"type":39,"tag":69,"props":2659,"children":2661},{"className":2660},[],[2662],{"type":45,"value":2663},"pa.month_day_nano_interval()",{"type":45,"value":2665}," for date operations.",{"type":39,"tag":55,"props":2667,"children":2668},{},[2669,2674,2675,2681,2683,2689,2691,2696,2698,2704],{"type":39,"tag":61,"props":2670,"children":2671},{},[2672],{"type":45,"value":2673},"Timestamps behave differently",{"type":45,"value":2606},{"type":39,"tag":69,"props":2676,"children":2678},{"className":2677},[],[2679],{"type":45,"value":2680},"Timestamp",{"type":45,"value":2682}," columns ",{"type":39,"tag":2684,"props":2685,"children":2686},"em",{},[2687],{"type":45,"value":2688},"do",{"type":45,"value":2690}," accept ",{"type":39,"tag":69,"props":2692,"children":2694},{"className":2693},[],[2695],{"type":45,"value":2517},{"type":45,"value":2697},",\nso ",{"type":39,"tag":69,"props":2699,"children":2701},{"className":2700},[],[2702],{"type":45,"value":2703},"col(\"ts\") - lit(datetime.timedelta(days=1))",{"type":45,"value":2705}," works. The interval-only\nrule applies specifically to date columns.",{"type":39,"tag":551,"props":2707,"children":2709},{"id":2708},"comparisons",[2710],{"type":45,"value":2711},"Comparisons",{"type":39,"tag":295,"props":2713,"children":2715},{"className":297,"code":2714,"language":18,"meta":299,"style":299},"col(\"a\") > 10\ncol(\"a\") >= 10\ncol(\"a\") \u003C 10\ncol(\"a\") \u003C= 10\ncol(\"a\") == \"x\"\ncol(\"a\") != \"x\"\ncol(\"a\") == None # same as col(\"a\").is_null()\ncol(\"a\") != None # same as col(\"a\").is_not_null()\n",[2716],{"type":39,"tag":69,"props":2717,"children":2718},{"__ignoreMap":299},[2719,2727,2735,2743,2751,2759,2767,2775],{"type":39,"tag":305,"props":2720,"children":2721},{"class":307,"line":308},[2722],{"type":39,"tag":305,"props":2723,"children":2724},{},[2725],{"type":45,"value":2726},"col(\"a\") > 10\n",{"type":39,"tag":305,"props":2728,"children":2729},{"class":307,"line":317},[2730],{"type":39,"tag":305,"props":2731,"children":2732},{},[2733],{"type":45,"value":2734},"col(\"a\") >= 10\n",{"type":39,"tag":305,"props":2736,"children":2737},{"class":307,"line":326},[2738],{"type":39,"tag":305,"props":2739,"children":2740},{},[2741],{"type":45,"value":2742},"col(\"a\") \u003C 10\n",{"type":39,"tag":305,"props":2744,"children":2745},{"class":307,"line":373},[2746],{"type":39,"tag":305,"props":2747,"children":2748},{},[2749],{"type":45,"value":2750},"col(\"a\") \u003C= 10\n",{"type":39,"tag":305,"props":2752,"children":2753},{"class":307,"line":382},[2754],{"type":39,"tag":305,"props":2755,"children":2756},{},[2757],{"type":45,"value":2758},"col(\"a\") == \"x\"\n",{"type":39,"tag":305,"props":2760,"children":2761},{"class":307,"line":391},[2762],{"type":39,"tag":305,"props":2763,"children":2764},{},[2765],{"type":45,"value":2766},"col(\"a\") != \"x\"\n",{"type":39,"tag":305,"props":2768,"children":2769},{"class":307,"line":400},[2770],{"type":39,"tag":305,"props":2771,"children":2772},{},[2773],{"type":45,"value":2774},"col(\"a\") == None # same as col(\"a\").is_null()\n",{"type":39,"tag":305,"props":2776,"children":2777},{"class":307,"line":408},[2778],{"type":39,"tag":305,"props":2779,"children":2780},{},[2781],{"type":45,"value":2782},"col(\"a\") != None # same as col(\"a\").is_not_null()\n",{"type":39,"tag":55,"props":2784,"children":2785},{},[2786,2788,2793,2795,2800,2802,2807],{"type":45,"value":2787},"Comparison operators auto-wrap the right-hand Python value into a literal,\nso writing ",{"type":39,"tag":69,"props":2789,"children":2791},{"className":2790},[],[2792],{"type":45,"value":782},{"type":45,"value":2794}," is redundant. Drop the ",{"type":39,"tag":69,"props":2796,"children":2798},{"className":2797},[],[2799],{"type":45,"value":2424},{"type":45,"value":2801}," in\ncomparisons. Reach for ",{"type":39,"tag":69,"props":2803,"children":2805},{"className":2804},[],[2806],{"type":45,"value":2424},{"type":45,"value":2808}," only when auto-wrapping does not apply — see\npitfall #2.",{"type":39,"tag":551,"props":2810,"children":2812},{"id":2811},"boolean-logic",[2813],{"type":45,"value":2814},"Boolean Logic",{"type":39,"tag":55,"props":2816,"children":2817},{},[2818,2822,2824,2830,2831,2837,2838,2843],{"type":39,"tag":61,"props":2819,"children":2820},{},[2821],{"type":45,"value":2604},{"type":45,"value":2823},": Python's ",{"type":39,"tag":69,"props":2825,"children":2827},{"className":2826},[],[2828],{"type":45,"value":2829},"and",{"type":45,"value":1157},{"type":39,"tag":69,"props":2832,"children":2834},{"className":2833},[],[2835],{"type":45,"value":2836},"or",{"type":45,"value":1157},{"type":39,"tag":69,"props":2839,"children":2841},{"className":2840},[],[2842],{"type":45,"value":2627},{"type":45,"value":2844}," keywords do NOT work with Expr\nobjects. You must use the bitwise operators:",{"type":39,"tag":295,"props":2846,"children":2848},{"className":297,"code":2847,"language":18,"meta":299,"style":299},"(col(\"a\") > 1) & (col(\"b\") \u003C 10) # AND\n(col(\"a\") > 1) | (col(\"b\") \u003C 10) # OR\n~(col(\"a\") > 1) # NOT\n",[2849],{"type":39,"tag":69,"props":2850,"children":2851},{"__ignoreMap":299},[2852,2860,2868],{"type":39,"tag":305,"props":2853,"children":2854},{"class":307,"line":308},[2855],{"type":39,"tag":305,"props":2856,"children":2857},{},[2858],{"type":45,"value":2859},"(col(\"a\") > 1) & (col(\"b\") \u003C 10) # AND\n",{"type":39,"tag":305,"props":2861,"children":2862},{"class":307,"line":317},[2863],{"type":39,"tag":305,"props":2864,"children":2865},{},[2866],{"type":45,"value":2867},"(col(\"a\") > 1) | (col(\"b\") \u003C 10) # OR\n",{"type":39,"tag":305,"props":2869,"children":2870},{"class":307,"line":326},[2871],{"type":39,"tag":305,"props":2872,"children":2873},{},[2874],{"type":45,"value":2875},"~(col(\"a\") > 1) # NOT\n",{"type":39,"tag":55,"props":2877,"children":2878},{},[2879,2881,2887,2888,2894,2895,2901],{"type":45,"value":2880},"Always wrap each comparison in parentheses when combining with ",{"type":39,"tag":69,"props":2882,"children":2884},{"className":2883},[],[2885],{"type":45,"value":2886},"&",{"type":45,"value":1157},{"type":39,"tag":69,"props":2889,"children":2891},{"className":2890},[],[2892],{"type":45,"value":2893},"|",{"type":45,"value":1157},{"type":39,"tag":69,"props":2896,"children":2898},{"className":2897},[],[2899],{"type":45,"value":2900},"~",{"type":45,"value":2902},"\nbecause Python's operator precedence for bitwise operators is different from\nlogical operators.",{"type":39,"tag":551,"props":2904,"children":2906},{"id":2905},"null-handling",[2907],{"type":45,"value":2908},"Null Handling",{"type":39,"tag":295,"props":2910,"children":2912},{"className":297,"code":2911,"language":18,"meta":299,"style":299},"col(\"a\").is_null()\ncol(\"a\").is_not_null()\ncol(\"a\").fill_null(lit(0)) # replace NULL with a value (single expression)\nF.coalesce(col(\"a\"), col(\"b\")) # first non-null value\nF.nullif(col(\"a\"), lit(0)) # return NULL if a == 0\n",[2913],{"type":39,"tag":69,"props":2914,"children":2915},{"__ignoreMap":299},[2916,2924,2932,2940,2948],{"type":39,"tag":305,"props":2917,"children":2918},{"class":307,"line":308},[2919],{"type":39,"tag":305,"props":2920,"children":2921},{},[2922],{"type":45,"value":2923},"col(\"a\").is_null()\n",{"type":39,"tag":305,"props":2925,"children":2926},{"class":307,"line":317},[2927],{"type":39,"tag":305,"props":2928,"children":2929},{},[2930],{"type":45,"value":2931},"col(\"a\").is_not_null()\n",{"type":39,"tag":305,"props":2933,"children":2934},{"class":307,"line":326},[2935],{"type":39,"tag":305,"props":2936,"children":2937},{},[2938],{"type":45,"value":2939},"col(\"a\").fill_null(lit(0)) # replace NULL with a value (single expression)\n",{"type":39,"tag":305,"props":2941,"children":2942},{"class":307,"line":373},[2943],{"type":39,"tag":305,"props":2944,"children":2945},{},[2946],{"type":45,"value":2947},"F.coalesce(col(\"a\"), col(\"b\")) # first non-null value\n",{"type":39,"tag":305,"props":2949,"children":2950},{"class":307,"line":382},[2951],{"type":39,"tag":305,"props":2952,"children":2953},{},[2954],{"type":45,"value":2955},"F.nullif(col(\"a\"), lit(0)) # return NULL if a == 0\n",{"type":39,"tag":55,"props":2957,"children":2958},{},[2959],{"type":45,"value":2960},"To fill nulls across the whole DataFrame (optionally limited to a subset of\ncolumns), use the DataFrame-level method:",{"type":39,"tag":295,"props":2962,"children":2964},{"className":297,"code":2963,"language":18,"meta":299,"style":299},"df.fill_null(0) # every column\ndf.fill_null(0, subset=[\"a\", \"b\"]) # only these columns\n",[2965],{"type":39,"tag":69,"props":2966,"children":2967},{"__ignoreMap":299},[2968,2976],{"type":39,"tag":305,"props":2969,"children":2970},{"class":307,"line":308},[2971],{"type":39,"tag":305,"props":2972,"children":2973},{},[2974],{"type":45,"value":2975},"df.fill_null(0) # every column\n",{"type":39,"tag":305,"props":2977,"children":2978},{"class":307,"line":317},[2979],{"type":39,"tag":305,"props":2980,"children":2981},{},[2982],{"type":45,"value":2983},"df.fill_null(0, subset=[\"a\", \"b\"]) # only these columns\n",{"type":39,"tag":551,"props":2985,"children":2987},{"id":2986},"case-when",[2988],{"type":45,"value":2989},"CASE \u002F WHEN",{"type":39,"tag":295,"props":2991,"children":2993},{"className":297,"code":2992,"language":18,"meta":299,"style":299},"# Simple CASE (matching on a single expression)\nstatus_label = (\n F.case(col(\"status\"))\n .when(lit(\"A\"), lit(\"Active\"))\n .when(lit(\"I\"), lit(\"Inactive\"))\n .otherwise(lit(\"Unknown\"))\n)\n\n# Searched CASE (each branch has its own predicate)\nseverity = (\n F.when(col(\"value\") > 100, lit(\"high\"))\n .when(col(\"value\") > 50, lit(\"medium\"))\n .otherwise(lit(\"low\"))\n)\n",[2994],{"type":39,"tag":69,"props":2995,"children":2996},{"__ignoreMap":299},[2997,3005,3013,3021,3029,3037,3045,3052,3059,3067,3075,3083,3091,3099],{"type":39,"tag":305,"props":2998,"children":2999},{"class":307,"line":308},[3000],{"type":39,"tag":305,"props":3001,"children":3002},{},[3003],{"type":45,"value":3004},"# Simple CASE (matching on a single expression)\n",{"type":39,"tag":305,"props":3006,"children":3007},{"class":307,"line":317},[3008],{"type":39,"tag":305,"props":3009,"children":3010},{},[3011],{"type":45,"value":3012},"status_label = (\n",{"type":39,"tag":305,"props":3014,"children":3015},{"class":307,"line":326},[3016],{"type":39,"tag":305,"props":3017,"children":3018},{},[3019],{"type":45,"value":3020}," F.case(col(\"status\"))\n",{"type":39,"tag":305,"props":3022,"children":3023},{"class":307,"line":373},[3024],{"type":39,"tag":305,"props":3025,"children":3026},{},[3027],{"type":45,"value":3028}," .when(lit(\"A\"), lit(\"Active\"))\n",{"type":39,"tag":305,"props":3030,"children":3031},{"class":307,"line":382},[3032],{"type":39,"tag":305,"props":3033,"children":3034},{},[3035],{"type":45,"value":3036}," .when(lit(\"I\"), lit(\"Inactive\"))\n",{"type":39,"tag":305,"props":3038,"children":3039},{"class":307,"line":391},[3040],{"type":39,"tag":305,"props":3041,"children":3042},{},[3043],{"type":45,"value":3044}," .otherwise(lit(\"Unknown\"))\n",{"type":39,"tag":305,"props":3046,"children":3047},{"class":307,"line":400},[3048],{"type":39,"tag":305,"props":3049,"children":3050},{},[3051],{"type":45,"value":626},{"type":39,"tag":305,"props":3053,"children":3054},{"class":307,"line":408},[3055],{"type":39,"tag":305,"props":3056,"children":3057},{"emptyLinePlaceholder":359},[3058],{"type":45,"value":362},{"type":39,"tag":305,"props":3060,"children":3061},{"class":307,"line":417},[3062],{"type":39,"tag":305,"props":3063,"children":3064},{},[3065],{"type":45,"value":3066},"# Searched CASE (each branch has its own predicate)\n",{"type":39,"tag":305,"props":3068,"children":3069},{"class":307,"line":426},[3070],{"type":39,"tag":305,"props":3071,"children":3072},{},[3073],{"type":45,"value":3074},"severity = (\n",{"type":39,"tag":305,"props":3076,"children":3077},{"class":307,"line":435},[3078],{"type":39,"tag":305,"props":3079,"children":3080},{},[3081],{"type":45,"value":3082}," F.when(col(\"value\") > 100, lit(\"high\"))\n",{"type":39,"tag":305,"props":3084,"children":3085},{"class":307,"line":444},[3086],{"type":39,"tag":305,"props":3087,"children":3088},{},[3089],{"type":45,"value":3090}," .when(col(\"value\") > 50, lit(\"medium\"))\n",{"type":39,"tag":305,"props":3092,"children":3093},{"class":307,"line":453},[3094],{"type":39,"tag":305,"props":3095,"children":3096},{},[3097],{"type":45,"value":3098}," .otherwise(lit(\"low\"))\n",{"type":39,"tag":305,"props":3100,"children":3101},{"class":307,"line":462},[3102],{"type":39,"tag":305,"props":3103,"children":3104},{},[3105],{"type":45,"value":626},{"type":39,"tag":551,"props":3107,"children":3109},{"id":3108},"casting",[3110],{"type":45,"value":3111},"Casting",{"type":39,"tag":295,"props":3113,"children":3115},{"className":297,"code":3114,"language":18,"meta":299,"style":299},"import pyarrow as pa\n\ncol(\"a\").cast(pa.float64())\ncol(\"a\").cast(pa.utf8())\ncol(\"a\").cast(pa.date32())\n\ncol(\"a\").try_cast(pa.int32()) # like cast(), but yields NULL on failure instead of erroring\n",[3116],{"type":39,"tag":69,"props":3117,"children":3118},{"__ignoreMap":299},[3119,3126,3133,3141,3149,3157,3164],{"type":39,"tag":305,"props":3120,"children":3121},{"class":307,"line":308},[3122],{"type":39,"tag":305,"props":3123,"children":3124},{},[3125],{"type":45,"value":2550},{"type":39,"tag":305,"props":3127,"children":3128},{"class":307,"line":317},[3129],{"type":39,"tag":305,"props":3130,"children":3131},{"emptyLinePlaceholder":359},[3132],{"type":45,"value":362},{"type":39,"tag":305,"props":3134,"children":3135},{"class":307,"line":326},[3136],{"type":39,"tag":305,"props":3137,"children":3138},{},[3139],{"type":45,"value":3140},"col(\"a\").cast(pa.float64())\n",{"type":39,"tag":305,"props":3142,"children":3143},{"class":307,"line":373},[3144],{"type":39,"tag":305,"props":3145,"children":3146},{},[3147],{"type":45,"value":3148},"col(\"a\").cast(pa.utf8())\n",{"type":39,"tag":305,"props":3150,"children":3151},{"class":307,"line":382},[3152],{"type":39,"tag":305,"props":3153,"children":3154},{},[3155],{"type":45,"value":3156},"col(\"a\").cast(pa.date32())\n",{"type":39,"tag":305,"props":3158,"children":3159},{"class":307,"line":391},[3160],{"type":39,"tag":305,"props":3161,"children":3162},{"emptyLinePlaceholder":359},[3163],{"type":45,"value":362},{"type":39,"tag":305,"props":3165,"children":3166},{"class":307,"line":400},[3167],{"type":39,"tag":305,"props":3168,"children":3169},{},[3170],{"type":45,"value":3171},"col(\"a\").try_cast(pa.int32()) # like cast(), but yields NULL on failure instead of erroring\n",{"type":39,"tag":55,"props":3173,"children":3174},{},[3175,3177,3183],{"type":45,"value":3176},"To cast several columns at once at the DataFrame level, pass a mapping to\n",{"type":39,"tag":69,"props":3178,"children":3180},{"className":3179},[],[3181],{"type":45,"value":3182},"df.cast(...)",{"type":45,"value":3184},":",{"type":39,"tag":295,"props":3186,"children":3188},{"className":297,"code":3187,"language":18,"meta":299,"style":299},"df.cast({\"a\": pa.float64(), \"b\": pa.int32()})\n",[3189],{"type":39,"tag":69,"props":3190,"children":3191},{"__ignoreMap":299},[3192],{"type":39,"tag":305,"props":3193,"children":3194},{"class":307,"line":308},[3195],{"type":39,"tag":305,"props":3196,"children":3197},{},[3198],{"type":45,"value":3187},{"type":39,"tag":551,"props":3200,"children":3202},{"id":3201},"aliasing",[3203],{"type":45,"value":3204},"Aliasing",{"type":39,"tag":295,"props":3206,"children":3208},{"className":297,"code":3207,"language":18,"meta":299,"style":299},"(col(\"a\") + col(\"b\")).alias(\"total\")\n",[3209],{"type":39,"tag":69,"props":3210,"children":3211},{"__ignoreMap":299},[3212],{"type":39,"tag":305,"props":3213,"children":3214},{"class":307,"line":308},[3215],{"type":39,"tag":305,"props":3216,"children":3217},{},[3218],{"type":45,"value":3207},{"type":39,"tag":551,"props":3220,"children":3222},{"id":3221},"between-and-in",[3223],{"type":45,"value":3224},"BETWEEN and IN",{"type":39,"tag":295,"props":3226,"children":3228},{"className":297,"code":3227,"language":18,"meta":299,"style":299},"col(\"a\").between(1, 10) # 1 \u003C= a \u003C= 10 (bounds auto-wrap)\nF.in_list(col(\"a\"), [lit(1), lit(2), lit(3)]) # a IN (1, 2, 3)\nF.in_list(col(\"a\"), [lit(1), lit(2)], negated=True) # a NOT IN (1, 2)\n",[3229],{"type":39,"tag":69,"props":3230,"children":3231},{"__ignoreMap":299},[3232,3240,3248],{"type":39,"tag":305,"props":3233,"children":3234},{"class":307,"line":308},[3235],{"type":39,"tag":305,"props":3236,"children":3237},{},[3238],{"type":45,"value":3239},"col(\"a\").between(1, 10) # 1 \u003C= a \u003C= 10 (bounds auto-wrap)\n",{"type":39,"tag":305,"props":3241,"children":3242},{"class":307,"line":317},[3243],{"type":39,"tag":305,"props":3244,"children":3245},{},[3246],{"type":45,"value":3247},"F.in_list(col(\"a\"), [lit(1), lit(2), lit(3)]) # a IN (1, 2, 3)\n",{"type":39,"tag":305,"props":3249,"children":3250},{"class":307,"line":326},[3251],{"type":39,"tag":305,"props":3252,"children":3253},{},[3254],{"type":45,"value":3255},"F.in_list(col(\"a\"), [lit(1), lit(2)], negated=True) # a NOT IN (1, 2)\n",{"type":39,"tag":551,"props":3257,"children":3259},{"id":3258},"struct-and-array-access",[3260],{"type":45,"value":3261},"Struct and Array Access",{"type":39,"tag":295,"props":3263,"children":3265},{"className":297,"code":3264,"language":18,"meta":299,"style":299},"col(\"struct_col\")[\"field_name\"] # access struct field\ncol(\"array_col\")[0] # access array element (0-indexed)\ncol(\"array_col\")[1:3] # array slice (0-indexed)\n",[3266],{"type":39,"tag":69,"props":3267,"children":3268},{"__ignoreMap":299},[3269,3277,3285],{"type":39,"tag":305,"props":3270,"children":3271},{"class":307,"line":308},[3272],{"type":39,"tag":305,"props":3273,"children":3274},{},[3275],{"type":45,"value":3276},"col(\"struct_col\")[\"field_name\"] # access struct field\n",{"type":39,"tag":305,"props":3278,"children":3279},{"class":307,"line":317},[3280],{"type":39,"tag":305,"props":3281,"children":3282},{},[3283],{"type":45,"value":3284},"col(\"array_col\")[0] # access array element (0-indexed)\n",{"type":39,"tag":305,"props":3286,"children":3287},{"class":307,"line":326},[3288],{"type":39,"tag":305,"props":3289,"children":3290},{},[3291],{"type":45,"value":3292},"col(\"array_col\")[1:3] # array slice (0-indexed)\n",{"type":39,"tag":551,"props":3294,"children":3296},{"id":3295},"lambda-functions",[3297],{"type":45,"value":3298},"Lambda Functions",{"type":39,"tag":55,"props":3300,"children":3301},{},[3302,3304,3310],{"type":45,"value":3303},"Some array functions take a lambda function that runs once per element. Pass a\nPython ",{"type":39,"tag":69,"props":3305,"children":3307},{"className":3306},[],[3308],{"type":45,"value":3309},"lambda",{"type":45,"value":3311}," directly — its parameter names become the lambda parameters and\nits return value becomes the body:",{"type":39,"tag":295,"props":3313,"children":3315},{"className":297,"code":3314,"language":18,"meta":299,"style":299},"F.array_transform(col(\"a\"), lambda v: v * 2) # map: [1,2,3] -> [2,4,6]\nF.array_filter(col(\"a\"), lambda v: v > 2) # filter: [1,2,3] -> [3]\nF.array_any_match(col(\"a\"), lambda v: v > 3) # predicate: any element > 3\n",[3316],{"type":39,"tag":69,"props":3317,"children":3318},{"__ignoreMap":299},[3319,3327,3335],{"type":39,"tag":305,"props":3320,"children":3321},{"class":307,"line":308},[3322],{"type":39,"tag":305,"props":3323,"children":3324},{},[3325],{"type":45,"value":3326},"F.array_transform(col(\"a\"), lambda v: v * 2) # map: [1,2,3] -> [2,4,6]\n",{"type":39,"tag":305,"props":3328,"children":3329},{"class":307,"line":317},[3330],{"type":39,"tag":305,"props":3331,"children":3332},{},[3333],{"type":45,"value":3334},"F.array_filter(col(\"a\"), lambda v: v > 2) # filter: [1,2,3] -> [3]\n",{"type":39,"tag":305,"props":3336,"children":3337},{"class":307,"line":326},[3338],{"type":39,"tag":305,"props":3339,"children":3340},{},[3341],{"type":45,"value":3342},"F.array_any_match(col(\"a\"), lambda v: v > 3) # predicate: any element > 3\n",{"type":39,"tag":55,"props":3344,"children":3345},{},[3346],{"type":45,"value":3347},"For explicit parameter names, build the lambda by hand:",{"type":39,"tag":295,"props":3349,"children":3351},{"className":297,"code":3350,"language":18,"meta":299,"style":299},"F.array_transform(col(\"a\"), F.lambda_([\"v\"], F.lambda_var(\"v\") * lit(2)))\n",[3352],{"type":39,"tag":69,"props":3353,"children":3354},{"__ignoreMap":299},[3355],{"type":39,"tag":305,"props":3356,"children":3357},{"class":307,"line":308},[3358],{"type":39,"tag":305,"props":3359,"children":3360},{},[3361],{"type":45,"value":3350},{"type":39,"tag":48,"props":3363,"children":3365},{"id":3364},"sql-to-dataframe-reference",[3366],{"type":45,"value":3367},"SQL-to-DataFrame Reference",{"type":39,"tag":123,"props":3369,"children":3370},{},[3371,3386],{"type":39,"tag":127,"props":3372,"children":3373},{},[3374],{"type":39,"tag":131,"props":3375,"children":3376},{},[3377,3381],{"type":39,"tag":135,"props":3378,"children":3379},{},[3380],{"type":45,"value":20},{"type":39,"tag":135,"props":3382,"children":3383},{},[3384],{"type":45,"value":3385},"DataFrame API",{"type":39,"tag":151,"props":3387,"children":3388},{},[3389,3410,3431,3452,3473,3502,3523,3544,3565,3586,3607,3628,3649,3670,3691,3714,3735,3757,3778,3800,3821,3842,3863,3884,3905,3926,3947,3968],{"type":39,"tag":131,"props":3390,"children":3391},{},[3392,3401],{"type":39,"tag":158,"props":3393,"children":3394},{},[3395],{"type":39,"tag":69,"props":3396,"children":3398},{"className":3397},[],[3399],{"type":45,"value":3400},"SELECT a, b",{"type":39,"tag":158,"props":3402,"children":3403},{},[3404],{"type":39,"tag":69,"props":3405,"children":3407},{"className":3406},[],[3408],{"type":45,"value":3409},"df.select(\"a\", \"b\")",{"type":39,"tag":131,"props":3411,"children":3412},{},[3413,3422],{"type":39,"tag":158,"props":3414,"children":3415},{},[3416],{"type":39,"tag":69,"props":3417,"children":3419},{"className":3418},[],[3420],{"type":45,"value":3421},"SELECT a, b + 1 AS c",{"type":39,"tag":158,"props":3423,"children":3424},{},[3425],{"type":39,"tag":69,"props":3426,"children":3428},{"className":3427},[],[3429],{"type":45,"value":3430},"df.select(col(\"a\"), (col(\"b\") + lit(1)).alias(\"c\"))",{"type":39,"tag":131,"props":3432,"children":3433},{},[3434,3443],{"type":39,"tag":158,"props":3435,"children":3436},{},[3437],{"type":39,"tag":69,"props":3438,"children":3440},{"className":3439},[],[3441],{"type":45,"value":3442},"SELECT *, a + 1 AS c",{"type":39,"tag":158,"props":3444,"children":3445},{},[3446],{"type":39,"tag":69,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":45,"value":3451},"df.with_column(\"c\", col(\"a\") + lit(1))",{"type":39,"tag":131,"props":3453,"children":3454},{},[3455,3464],{"type":39,"tag":158,"props":3456,"children":3457},{},[3458],{"type":39,"tag":69,"props":3459,"children":3461},{"className":3460},[],[3462],{"type":45,"value":3463},"WHERE a > 10",{"type":39,"tag":158,"props":3465,"children":3466},{},[3467],{"type":39,"tag":69,"props":3468,"children":3470},{"className":3469},[],[3471],{"type":45,"value":3472},"df.filter(col(\"a\") > 10)",{"type":39,"tag":131,"props":3474,"children":3475},{},[3476,3493],{"type":39,"tag":158,"props":3477,"children":3478},{},[3479,3485,3487],{"type":39,"tag":69,"props":3480,"children":3482},{"className":3481},[],[3483],{"type":45,"value":3484},"GROUP BY a",{"type":45,"value":3486}," with ",{"type":39,"tag":69,"props":3488,"children":3490},{"className":3489},[],[3491],{"type":45,"value":3492},"SUM(b)",{"type":39,"tag":158,"props":3494,"children":3495},{},[3496],{"type":39,"tag":69,"props":3497,"children":3499},{"className":3498},[],[3500],{"type":45,"value":3501},"df.aggregate([\"a\"], [F.sum(col(\"b\"))])",{"type":39,"tag":131,"props":3503,"children":3504},{},[3505,3514],{"type":39,"tag":158,"props":3506,"children":3507},{},[3508],{"type":39,"tag":69,"props":3509,"children":3511},{"className":3510},[],[3512],{"type":45,"value":3513},"SUM(b) FILTER (WHERE b > 100)",{"type":39,"tag":158,"props":3515,"children":3516},{},[3517],{"type":39,"tag":69,"props":3518,"children":3520},{"className":3519},[],[3521],{"type":45,"value":3522},"F.sum(col(\"b\"), filter=col(\"b\") > 100)",{"type":39,"tag":131,"props":3524,"children":3525},{},[3526,3535],{"type":39,"tag":158,"props":3527,"children":3528},{},[3529],{"type":39,"tag":69,"props":3530,"children":3532},{"className":3531},[],[3533],{"type":45,"value":3534},"ORDER BY a DESC",{"type":39,"tag":158,"props":3536,"children":3537},{},[3538],{"type":39,"tag":69,"props":3539,"children":3541},{"className":3540},[],[3542],{"type":45,"value":3543},"df.sort(col(\"a\").sort(ascending=False))",{"type":39,"tag":131,"props":3545,"children":3546},{},[3547,3556],{"type":39,"tag":158,"props":3548,"children":3549},{},[3550],{"type":39,"tag":69,"props":3551,"children":3553},{"className":3552},[],[3554],{"type":45,"value":3555},"LIMIT 10 OFFSET 5",{"type":39,"tag":158,"props":3557,"children":3558},{},[3559],{"type":39,"tag":69,"props":3560,"children":3562},{"className":3561},[],[3563],{"type":45,"value":3564},"df.limit(10, offset=5)",{"type":39,"tag":131,"props":3566,"children":3567},{},[3568,3577],{"type":39,"tag":158,"props":3569,"children":3570},{},[3571],{"type":39,"tag":69,"props":3572,"children":3574},{"className":3573},[],[3575],{"type":45,"value":3576},"DISTINCT",{"type":39,"tag":158,"props":3578,"children":3579},{},[3580],{"type":39,"tag":69,"props":3581,"children":3583},{"className":3582},[],[3584],{"type":45,"value":3585},"df.distinct()",{"type":39,"tag":131,"props":3587,"children":3588},{},[3589,3598],{"type":39,"tag":158,"props":3590,"children":3591},{},[3592],{"type":39,"tag":69,"props":3593,"children":3595},{"className":3594},[],[3596],{"type":45,"value":3597},"a INNER JOIN b ON a.id = b.id",{"type":39,"tag":158,"props":3599,"children":3600},{},[3601],{"type":39,"tag":69,"props":3602,"children":3604},{"className":3603},[],[3605],{"type":45,"value":3606},"a.join(b, on=\"id\")",{"type":39,"tag":131,"props":3608,"children":3609},{},[3610,3619],{"type":39,"tag":158,"props":3611,"children":3612},{},[3613],{"type":39,"tag":69,"props":3614,"children":3616},{"className":3615},[],[3617],{"type":45,"value":3618},"a LEFT JOIN b ON a.id = b.fk",{"type":39,"tag":158,"props":3620,"children":3621},{},[3622],{"type":39,"tag":69,"props":3623,"children":3625},{"className":3624},[],[3626],{"type":45,"value":3627},"a.join(b, left_on=\"id\", right_on=\"fk\", how=\"left\")",{"type":39,"tag":131,"props":3629,"children":3630},{},[3631,3640],{"type":39,"tag":158,"props":3632,"children":3633},{},[3634],{"type":39,"tag":69,"props":3635,"children":3637},{"className":3636},[],[3638],{"type":45,"value":3639},"WHERE EXISTS (SELECT ...)",{"type":39,"tag":158,"props":3641,"children":3642},{},[3643],{"type":39,"tag":69,"props":3644,"children":3646},{"className":3645},[],[3647],{"type":45,"value":3648},"a.join(b, on=\"key\", how=\"semi\")",{"type":39,"tag":131,"props":3650,"children":3651},{},[3652,3661],{"type":39,"tag":158,"props":3653,"children":3654},{},[3655],{"type":39,"tag":69,"props":3656,"children":3658},{"className":3657},[],[3659],{"type":45,"value":3660},"WHERE NOT EXISTS (SELECT ...)",{"type":39,"tag":158,"props":3662,"children":3663},{},[3664],{"type":39,"tag":69,"props":3665,"children":3667},{"className":3666},[],[3668],{"type":45,"value":3669},"a.join(b, on=\"key\", how=\"anti\")",{"type":39,"tag":131,"props":3671,"children":3672},{},[3673,3682],{"type":39,"tag":158,"props":3674,"children":3675},{},[3676],{"type":39,"tag":69,"props":3677,"children":3679},{"className":3678},[],[3680],{"type":45,"value":3681},"UNION ALL",{"type":39,"tag":158,"props":3683,"children":3684},{},[3685],{"type":39,"tag":69,"props":3686,"children":3688},{"className":3687},[],[3689],{"type":45,"value":3690},"df1.union(df2)",{"type":39,"tag":131,"props":3692,"children":3693},{},[3694,3705],{"type":39,"tag":158,"props":3695,"children":3696},{},[3697,3703],{"type":39,"tag":69,"props":3698,"children":3700},{"className":3699},[],[3701],{"type":45,"value":3702},"UNION",{"type":45,"value":3704}," (distinct)",{"type":39,"tag":158,"props":3706,"children":3707},{},[3708],{"type":39,"tag":69,"props":3709,"children":3711},{"className":3710},[],[3712],{"type":45,"value":3713},"df1.union(df2, distinct=True)",{"type":39,"tag":131,"props":3715,"children":3716},{},[3717,3726],{"type":39,"tag":158,"props":3718,"children":3719},{},[3720],{"type":39,"tag":69,"props":3721,"children":3723},{"className":3722},[],[3724],{"type":45,"value":3725},"INTERSECT ALL",{"type":39,"tag":158,"props":3727,"children":3728},{},[3729],{"type":39,"tag":69,"props":3730,"children":3732},{"className":3731},[],[3733],{"type":45,"value":3734},"df1.intersect(df2)",{"type":39,"tag":131,"props":3736,"children":3737},{},[3738,3748],{"type":39,"tag":158,"props":3739,"children":3740},{},[3741,3747],{"type":39,"tag":69,"props":3742,"children":3744},{"className":3743},[],[3745],{"type":45,"value":3746},"INTERSECT",{"type":45,"value":3704},{"type":39,"tag":158,"props":3749,"children":3750},{},[3751],{"type":39,"tag":69,"props":3752,"children":3754},{"className":3753},[],[3755],{"type":45,"value":3756},"df1.intersect(df2, distinct=True)",{"type":39,"tag":131,"props":3758,"children":3759},{},[3760,3769],{"type":39,"tag":158,"props":3761,"children":3762},{},[3763],{"type":39,"tag":69,"props":3764,"children":3766},{"className":3765},[],[3767],{"type":45,"value":3768},"EXCEPT ALL",{"type":39,"tag":158,"props":3770,"children":3771},{},[3772],{"type":39,"tag":69,"props":3773,"children":3775},{"className":3774},[],[3776],{"type":45,"value":3777},"df1.except_all(df2)",{"type":39,"tag":131,"props":3779,"children":3780},{},[3781,3791],{"type":39,"tag":158,"props":3782,"children":3783},{},[3784,3790],{"type":39,"tag":69,"props":3785,"children":3787},{"className":3786},[],[3788],{"type":45,"value":3789},"EXCEPT",{"type":45,"value":3704},{"type":39,"tag":158,"props":3792,"children":3793},{},[3794],{"type":39,"tag":69,"props":3795,"children":3797},{"className":3796},[],[3798],{"type":45,"value":3799},"df1.except_all(df2, distinct=True)",{"type":39,"tag":131,"props":3801,"children":3802},{},[3803,3812],{"type":39,"tag":158,"props":3804,"children":3805},{},[3806],{"type":39,"tag":69,"props":3807,"children":3809},{"className":3808},[],[3810],{"type":45,"value":3811},"CASE x WHEN 1 THEN 'a' END",{"type":39,"tag":158,"props":3813,"children":3814},{},[3815],{"type":39,"tag":69,"props":3816,"children":3818},{"className":3817},[],[3819],{"type":45,"value":3820},"F.case(col(\"x\")).when(lit(1), lit(\"a\")).end()",{"type":39,"tag":131,"props":3822,"children":3823},{},[3824,3833],{"type":39,"tag":158,"props":3825,"children":3826},{},[3827],{"type":39,"tag":69,"props":3828,"children":3830},{"className":3829},[],[3831],{"type":45,"value":3832},"CASE WHEN x > 1 THEN 'a' END",{"type":39,"tag":158,"props":3834,"children":3835},{},[3836],{"type":39,"tag":69,"props":3837,"children":3839},{"className":3838},[],[3840],{"type":45,"value":3841},"F.when(col(\"x\") > 1, lit(\"a\")).end()",{"type":39,"tag":131,"props":3843,"children":3844},{},[3845,3854],{"type":39,"tag":158,"props":3846,"children":3847},{},[3848],{"type":39,"tag":69,"props":3849,"children":3851},{"className":3850},[],[3852],{"type":45,"value":3853},"x IN (1, 2, 3)",{"type":39,"tag":158,"props":3855,"children":3856},{},[3857],{"type":39,"tag":69,"props":3858,"children":3860},{"className":3859},[],[3861],{"type":45,"value":3862},"F.in_list(col(\"x\"), [lit(1), lit(2), lit(3)])",{"type":39,"tag":131,"props":3864,"children":3865},{},[3866,3875],{"type":39,"tag":158,"props":3867,"children":3868},{},[3869],{"type":39,"tag":69,"props":3870,"children":3872},{"className":3871},[],[3873],{"type":45,"value":3874},"x BETWEEN 1 AND 10",{"type":39,"tag":158,"props":3876,"children":3877},{},[3878],{"type":39,"tag":69,"props":3879,"children":3881},{"className":3880},[],[3882],{"type":45,"value":3883},"col(\"x\").between(1, 10)",{"type":39,"tag":131,"props":3885,"children":3886},{},[3887,3896],{"type":39,"tag":158,"props":3888,"children":3889},{},[3890],{"type":39,"tag":69,"props":3891,"children":3893},{"className":3892},[],[3894],{"type":45,"value":3895},"CAST(x AS DOUBLE)",{"type":39,"tag":158,"props":3897,"children":3898},{},[3899],{"type":39,"tag":69,"props":3900,"children":3902},{"className":3901},[],[3903],{"type":45,"value":3904},"col(\"x\").cast(pa.float64())",{"type":39,"tag":131,"props":3906,"children":3907},{},[3908,3917],{"type":39,"tag":158,"props":3909,"children":3910},{},[3911],{"type":39,"tag":69,"props":3912,"children":3914},{"className":3913},[],[3915],{"type":45,"value":3916},"ROW_NUMBER() OVER (...)",{"type":39,"tag":158,"props":3918,"children":3919},{},[3920],{"type":39,"tag":69,"props":3921,"children":3923},{"className":3922},[],[3924],{"type":45,"value":3925},"F.row_number(partition_by=[...], order_by=[...])",{"type":39,"tag":131,"props":3927,"children":3928},{},[3929,3938],{"type":39,"tag":158,"props":3930,"children":3931},{},[3932],{"type":39,"tag":69,"props":3933,"children":3935},{"className":3934},[],[3936],{"type":45,"value":3937},"SUM(x) OVER (...)",{"type":39,"tag":158,"props":3939,"children":3940},{},[3941],{"type":39,"tag":69,"props":3942,"children":3944},{"className":3943},[],[3945],{"type":45,"value":3946},"F.sum(col(\"x\")).over(window)",{"type":39,"tag":131,"props":3948,"children":3949},{},[3950,3959],{"type":39,"tag":158,"props":3951,"children":3952},{},[3953],{"type":39,"tag":69,"props":3954,"children":3956},{"className":3955},[],[3957],{"type":45,"value":3958},"x IS NULL",{"type":39,"tag":158,"props":3960,"children":3961},{},[3962],{"type":39,"tag":69,"props":3963,"children":3965},{"className":3964},[],[3966],{"type":45,"value":3967},"col(\"x\").is_null()",{"type":39,"tag":131,"props":3969,"children":3970},{},[3971,3980],{"type":39,"tag":158,"props":3972,"children":3973},{},[3974],{"type":39,"tag":69,"props":3975,"children":3977},{"className":3976},[],[3978],{"type":45,"value":3979},"COALESCE(a, b)",{"type":39,"tag":158,"props":3981,"children":3982},{},[3983],{"type":39,"tag":69,"props":3984,"children":3986},{"className":3985},[],[3987],{"type":45,"value":3988},"F.coalesce(col(\"a\"), col(\"b\"))",{"type":39,"tag":48,"props":3990,"children":3992},{"id":3991},"common-pitfalls",[3993],{"type":45,"value":3994},"Common Pitfalls",{"type":39,"tag":3996,"props":3997,"children":3998},"ol",{},[3999,4055,4214,4253,4292,4325,4426],{"type":39,"tag":4000,"props":4001,"children":4002},"li",{},[4003,4008,4010,4015,4016,4021,4022,4027,4029,4034,4035,4040,4041,4046,4048,4054],{"type":39,"tag":61,"props":4004,"children":4005},{},[4006],{"type":45,"value":4007},"Boolean operators",{"type":45,"value":4009},": Use ",{"type":39,"tag":69,"props":4011,"children":4013},{"className":4012},[],[4014],{"type":45,"value":2886},{"type":45,"value":1157},{"type":39,"tag":69,"props":4017,"children":4019},{"className":4018},[],[4020],{"type":45,"value":2893},{"type":45,"value":1157},{"type":39,"tag":69,"props":4023,"children":4025},{"className":4024},[],[4026],{"type":45,"value":2900},{"type":45,"value":4028}," -- not Python's ",{"type":39,"tag":69,"props":4030,"children":4032},{"className":4031},[],[4033],{"type":45,"value":2829},{"type":45,"value":1157},{"type":39,"tag":69,"props":4036,"children":4038},{"className":4037},[],[4039],{"type":45,"value":2836},{"type":45,"value":1157},{"type":39,"tag":69,"props":4042,"children":4044},{"className":4043},[],[4045],{"type":45,"value":2627},{"type":45,"value":4047},".\nAlways parenthesize: ",{"type":39,"tag":69,"props":4049,"children":4051},{"className":4050},[],[4052],{"type":45,"value":4053},"(col(\"a\") > 1) & (col(\"b\") \u003C 2)",{"type":45,"value":719},{"type":39,"tag":4000,"props":4056,"children":4057},{},[4058,4068,4070,4075,4076,4082,4084,4089,4091,4096,4098,4102,4104],{"type":39,"tag":61,"props":4059,"children":4060},{},[4061,4063],{"type":45,"value":4062},"Wrapping scalars with ",{"type":39,"tag":69,"props":4064,"children":4066},{"className":4065},[],[4067],{"type":45,"value":2424},{"type":45,"value":4069},": Prefer raw Python values on the\nright-hand side of comparisons — ",{"type":39,"tag":69,"props":4071,"children":4073},{"className":4072},[],[4074],{"type":45,"value":774},{"type":45,"value":1157},{"type":39,"tag":69,"props":4077,"children":4079},{"className":4078},[],[4080],{"type":45,"value":4081},"col(\"name\") == \"Alice\"",{"type":45,"value":4083},"\n— because the Expr comparison operators auto-wrap them. Writing\n",{"type":39,"tag":69,"props":4085,"children":4087},{"className":4086},[],[4088],{"type":45,"value":782},{"type":45,"value":4090}," is redundant. Reserve ",{"type":39,"tag":69,"props":4092,"children":4094},{"className":4093},[],[4095],{"type":45,"value":2424},{"type":45,"value":4097}," for places where\nauto-wrapping does ",{"type":39,"tag":2684,"props":4099,"children":4100},{},[4101],{"type":45,"value":2627},{"type":45,"value":4103}," apply:",{"type":39,"tag":4105,"props":4106,"children":4107},"ul",{},[4108,4127,4148,4159],{"type":39,"tag":4000,"props":4109,"children":4110},{},[4111,4113,4119,4121],{"type":45,"value":4112},"standalone scalars passed into function calls:\n",{"type":39,"tag":69,"props":4114,"children":4116},{"className":4115},[],[4117],{"type":45,"value":4118},"F.coalesce(col(\"a\"), lit(0))",{"type":45,"value":4120},", not ",{"type":39,"tag":69,"props":4122,"children":4124},{"className":4123},[],[4125],{"type":45,"value":4126},"F.coalesce(col(\"a\"), 0)",{"type":39,"tag":4000,"props":4128,"children":4129},{},[4130,4132,4138,4140,4146],{"type":45,"value":4131},"arithmetic between two literals with no column involved:\n",{"type":39,"tag":69,"props":4133,"children":4135},{"className":4134},[],[4136],{"type":45,"value":4137},"lit(1) - col(\"discount\")",{"type":45,"value":4139}," is fine, but ",{"type":39,"tag":69,"props":4141,"children":4143},{"className":4142},[],[4144],{"type":45,"value":4145},"lit(1) - lit(2)",{"type":45,"value":4147}," needs both",{"type":39,"tag":4000,"props":4149,"children":4150},{},[4151,4153],{"type":45,"value":4152},"values that must carry a specific Arrow type, via ",{"type":39,"tag":69,"props":4154,"children":4156},{"className":4155},[],[4157],{"type":45,"value":4158},"lit(pa.scalar(...))",{"type":39,"tag":4000,"props":4160,"children":4161},{},[4162,4168,4169,4175,4176,4182,4183,4189,4191,4197,4199,4205,4207,4212],{"type":39,"tag":69,"props":4163,"children":4165},{"className":4164},[],[4166],{"type":45,"value":4167},".when(...)",{"type":45,"value":1157},{"type":39,"tag":69,"props":4170,"children":4172},{"className":4171},[],[4173],{"type":45,"value":4174},".otherwise(...)",{"type":45,"value":1157},{"type":39,"tag":69,"props":4177,"children":4179},{"className":4178},[],[4180],{"type":45,"value":4181},"F.nullif(...)",{"type":45,"value":1157},{"type":39,"tag":69,"props":4184,"children":4186},{"className":4185},[],[4187],{"type":45,"value":4188},"F.in_list(...)",{"type":45,"value":4190},"\nand similar method\u002Ffunction arguments (note: ",{"type":39,"tag":69,"props":4192,"children":4194},{"className":4193},[],[4195],{"type":45,"value":4196},".between(...)",{"type":45,"value":4198},"\nauto-wraps its bounds, so ",{"type":39,"tag":69,"props":4200,"children":4202},{"className":4201},[],[4203],{"type":45,"value":4204},"col(\"a\").between(1, 10)",{"type":45,"value":4206}," needs no ",{"type":39,"tag":69,"props":4208,"children":4210},{"className":4209},[],[4211],{"type":45,"value":2424},{"type":45,"value":4213},")",{"type":39,"tag":4000,"props":4215,"children":4216},{},[4217,4222,4224,4230,4231,4237,4239,4245,4246,4252],{"type":39,"tag":61,"props":4218,"children":4219},{},[4220],{"type":45,"value":4221},"Column name quoting",{"type":45,"value":4223},": Column names are normalized to lowercase by default\nin both ",{"type":39,"tag":69,"props":4225,"children":4227},{"className":4226},[],[4228],{"type":45,"value":4229},"select(\"...\")",{"type":45,"value":687},{"type":39,"tag":69,"props":4232,"children":4234},{"className":4233},[],[4235],{"type":45,"value":4236},"col(\"...\")",{"type":45,"value":4238},". To reference a column with\nuppercase letters, use double quotes inside the string:\n",{"type":39,"tag":69,"props":4240,"children":4242},{"className":4241},[],[4243],{"type":45,"value":4244},"select('\"MyColumn\"')",{"type":45,"value":703},{"type":39,"tag":69,"props":4247,"children":4249},{"className":4248},[],[4250],{"type":45,"value":4251},"col('\"MyColumn\"')",{"type":45,"value":719},{"type":39,"tag":4000,"props":4254,"children":4255},{},[4256,4261,4263,4267,4269],{"type":39,"tag":61,"props":4257,"children":4258},{},[4259],{"type":45,"value":4260},"DataFrames are immutable",{"type":45,"value":4262},": Every method returns a ",{"type":39,"tag":61,"props":4264,"children":4265},{},[4266],{"type":45,"value":547},{"type":45,"value":4268}," DataFrame. You\nmust capture the return value:",{"type":39,"tag":295,"props":4270,"children":4272},{"className":297,"code":4271,"language":18,"meta":299,"style":299},"df = df.filter(col(\"a\") > 1) # correct\ndf.filter(col(\"a\") > 1) # WRONG -- result is discarded\n",[4273],{"type":39,"tag":69,"props":4274,"children":4275},{"__ignoreMap":299},[4276,4284],{"type":39,"tag":305,"props":4277,"children":4278},{"class":307,"line":308},[4279],{"type":39,"tag":305,"props":4280,"children":4281},{},[4282],{"type":45,"value":4283},"df = df.filter(col(\"a\") > 1) # correct\n",{"type":39,"tag":305,"props":4285,"children":4286},{"class":307,"line":317},[4287],{"type":39,"tag":305,"props":4288,"children":4289},{},[4290],{"type":45,"value":4291},"df.filter(col(\"a\") > 1) # WRONG -- result is discarded\n",{"type":39,"tag":4000,"props":4293,"children":4294},{},[4295,4300,4302,4308,4310,4316,4318,4324],{"type":39,"tag":61,"props":4296,"children":4297},{},[4298],{"type":45,"value":4299},"Window frame defaults",{"type":45,"value":4301},": When using ",{"type":39,"tag":69,"props":4303,"children":4305},{"className":4304},[],[4306],{"type":45,"value":4307},"order_by",{"type":45,"value":4309}," in a window, the default\nframe is ",{"type":39,"tag":69,"props":4311,"children":4313},{"className":4312},[],[4314],{"type":45,"value":4315},"RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW",{"type":45,"value":4317},". For a full\npartition frame, set ",{"type":39,"tag":69,"props":4319,"children":4321},{"className":4320},[],[4322],{"type":45,"value":4323},"window_frame=WindowFrame(\"rows\", None, None)",{"type":45,"value":719},{"type":39,"tag":4000,"props":4326,"children":4327},{},[4328,4347,4349,4354,4356,4362,4364,4370,4372],{"type":39,"tag":61,"props":4329,"children":4330},{},[4331,4333,4339,4341],{"type":45,"value":4332},"Arithmetic on aggregates belongs in a later ",{"type":39,"tag":69,"props":4334,"children":4336},{"className":4335},[],[4337],{"type":45,"value":4338},"select",{"type":45,"value":4340},", not inside\n",{"type":39,"tag":69,"props":4342,"children":4344},{"className":4343},[],[4345],{"type":45,"value":4346},"aggregate",{"type":45,"value":4348}," ",{"type":39,"tag":2684,"props":4350,"children":4351},{},[4352],{"type":45,"value":4353},"(applies to datafusion-python 53 and earlier; fixed in 54)",{"type":45,"value":4355},":\nEach item in the aggregate list must be a single aggregate\ncall (optionally aliased). Combining aggregates with arithmetic inside\n",{"type":39,"tag":69,"props":4357,"children":4359},{"className":4358},[],[4360],{"type":45,"value":4361},"aggregate(...)",{"type":45,"value":4363}," fails with ",{"type":39,"tag":69,"props":4365,"children":4367},{"className":4366},[],[4368],{"type":45,"value":4369},"Internal error: Invalid aggregate expression",{"type":45,"value":4371},".\nAlias the aggregates, then compute the combination downstream:",{"type":39,"tag":295,"props":4373,"children":4375},{"className":297,"code":4374,"language":18,"meta":299,"style":299},"# WRONG -- arithmetic wraps two aggregates\ndf.aggregate([], [(lit(100) * F.sum(col(\"a\")) \u002F F.sum(col(\"b\"))).alias(\"ratio\")])\n\n# CORRECT -- aggregate first, then combine\n(df.aggregate([], [F.sum(col(\"a\")).alias(\"num\"), F.sum(col(\"b\")).alias(\"den\")])\n .select((lit(100) * col(\"num\") \u002F col(\"den\")).alias(\"ratio\")))\n",[4376],{"type":39,"tag":69,"props":4377,"children":4378},{"__ignoreMap":299},[4379,4387,4395,4402,4410,4418],{"type":39,"tag":305,"props":4380,"children":4381},{"class":307,"line":308},[4382],{"type":39,"tag":305,"props":4383,"children":4384},{},[4385],{"type":45,"value":4386},"# WRONG -- arithmetic wraps two aggregates\n",{"type":39,"tag":305,"props":4388,"children":4389},{"class":307,"line":317},[4390],{"type":39,"tag":305,"props":4391,"children":4392},{},[4393],{"type":45,"value":4394},"df.aggregate([], [(lit(100) * F.sum(col(\"a\")) \u002F F.sum(col(\"b\"))).alias(\"ratio\")])\n",{"type":39,"tag":305,"props":4396,"children":4397},{"class":307,"line":326},[4398],{"type":39,"tag":305,"props":4399,"children":4400},{"emptyLinePlaceholder":359},[4401],{"type":45,"value":362},{"type":39,"tag":305,"props":4403,"children":4404},{"class":307,"line":373},[4405],{"type":39,"tag":305,"props":4406,"children":4407},{},[4408],{"type":45,"value":4409},"# CORRECT -- aggregate first, then combine\n",{"type":39,"tag":305,"props":4411,"children":4412},{"class":307,"line":382},[4413],{"type":39,"tag":305,"props":4414,"children":4415},{},[4416],{"type":45,"value":4417},"(df.aggregate([], [F.sum(col(\"a\")).alias(\"num\"), F.sum(col(\"b\")).alias(\"den\")])\n",{"type":39,"tag":305,"props":4419,"children":4420},{"class":307,"line":391},[4421],{"type":39,"tag":305,"props":4422,"children":4423},{},[4424],{"type":45,"value":4425}," .select((lit(100) * col(\"num\") \u002F col(\"den\")).alias(\"ratio\")))\n",{"type":39,"tag":4000,"props":4427,"children":4428},{},[4429,4434,4436,4442,4444,4450,4452,4458,4460,4466,4468,4474,4476,4481,4482,4487,4489,4495,4497,4590,4594,4596,4602,4604,4610,4612],{"type":39,"tag":61,"props":4430,"children":4431},{},[4432],{"type":45,"value":4433},"Don't alias a join column to match the other side",{"type":45,"value":4435},": When equi-joining\nwith ",{"type":39,"tag":69,"props":4437,"children":4439},{"className":4438},[],[4440],{"type":45,"value":4441},"on=\"key\"",{"type":45,"value":4443},", renaming the join column on one side via ",{"type":39,"tag":69,"props":4445,"children":4447},{"className":4446},[],[4448],{"type":45,"value":4449},".alias(\"key\")",{"type":45,"value":4451},"\nin a fresh projection creates a schema where one side's ",{"type":39,"tag":69,"props":4453,"children":4455},{"className":4454},[],[4456],{"type":45,"value":4457},"key",{"type":45,"value":4459}," is\nqualified (",{"type":39,"tag":69,"props":4461,"children":4463},{"className":4462},[],[4464],{"type":45,"value":4465},"?table?.key",{"type":45,"value":4467},") and the other is unqualified. The join then\nfails with ",{"type":39,"tag":69,"props":4469,"children":4471},{"className":4470},[],[4472],{"type":45,"value":4473},"Schema contains qualified field name ... and unqualified field name ... which would be ambiguous",{"type":45,"value":4475},". Use ",{"type":39,"tag":69,"props":4477,"children":4479},{"className":4478},[],[4480],{"type":45,"value":1239},{"type":45,"value":1241},{"type":39,"tag":69,"props":4483,"children":4485},{"className":4484},[],[4486],{"type":45,"value":1247},{"type":45,"value":4488}," with\nthe native names, or use ",{"type":39,"tag":69,"props":4490,"children":4492},{"className":4491},[],[4493],{"type":45,"value":4494},"join_on(...)",{"type":45,"value":4496}," with an explicit equality.",{"type":39,"tag":295,"props":4498,"children":4500},{"className":297,"code":4499,"language":18,"meta":299,"style":299},"# WRONG -- alias on one side produces ambiguous schema after join\nfailed = orders.select(col(\"o_orderkey\").alias(\"l_orderkey\"))\nli.join(failed, on=\"l_orderkey\") # ambiguous l_orderkey error\n\n# CORRECT -- keep native names, use left_on\u002Fright_on\nfailed = orders.select(\"o_orderkey\")\nli.join(failed, left_on=\"l_orderkey\", right_on=\"o_orderkey\")\n\n# ALSO CORRECT -- explicit predicate via join_on\n# (note: join_on keeps both key columns in the output, unlike on=\"key\")\nli.join_on(failed, col(\"l_orderkey\") == col(\"o_orderkey\"))\n",[4501],{"type":39,"tag":69,"props":4502,"children":4503},{"__ignoreMap":299},[4504,4512,4520,4528,4535,4543,4551,4559,4566,4574,4582],{"type":39,"tag":305,"props":4505,"children":4506},{"class":307,"line":308},[4507],{"type":39,"tag":305,"props":4508,"children":4509},{},[4510],{"type":45,"value":4511},"# WRONG -- alias on one side produces ambiguous schema after join\n",{"type":39,"tag":305,"props":4513,"children":4514},{"class":307,"line":317},[4515],{"type":39,"tag":305,"props":4516,"children":4517},{},[4518],{"type":45,"value":4519},"failed = orders.select(col(\"o_orderkey\").alias(\"l_orderkey\"))\n",{"type":39,"tag":305,"props":4521,"children":4522},{"class":307,"line":326},[4523],{"type":39,"tag":305,"props":4524,"children":4525},{},[4526],{"type":45,"value":4527},"li.join(failed, on=\"l_orderkey\") # ambiguous l_orderkey error\n",{"type":39,"tag":305,"props":4529,"children":4530},{"class":307,"line":373},[4531],{"type":39,"tag":305,"props":4532,"children":4533},{"emptyLinePlaceholder":359},[4534],{"type":45,"value":362},{"type":39,"tag":305,"props":4536,"children":4537},{"class":307,"line":382},[4538],{"type":39,"tag":305,"props":4539,"children":4540},{},[4541],{"type":45,"value":4542},"# CORRECT -- keep native names, use left_on\u002Fright_on\n",{"type":39,"tag":305,"props":4544,"children":4545},{"class":307,"line":391},[4546],{"type":39,"tag":305,"props":4547,"children":4548},{},[4549],{"type":45,"value":4550},"failed = orders.select(\"o_orderkey\")\n",{"type":39,"tag":305,"props":4552,"children":4553},{"class":307,"line":400},[4554],{"type":39,"tag":305,"props":4555,"children":4556},{},[4557],{"type":45,"value":4558},"li.join(failed, left_on=\"l_orderkey\", right_on=\"o_orderkey\")\n",{"type":39,"tag":305,"props":4560,"children":4561},{"class":307,"line":408},[4562],{"type":39,"tag":305,"props":4563,"children":4564},{"emptyLinePlaceholder":359},[4565],{"type":45,"value":362},{"type":39,"tag":305,"props":4567,"children":4568},{"class":307,"line":417},[4569],{"type":39,"tag":305,"props":4570,"children":4571},{},[4572],{"type":45,"value":4573},"# ALSO CORRECT -- explicit predicate via join_on\n",{"type":39,"tag":305,"props":4575,"children":4576},{"class":307,"line":426},[4577],{"type":39,"tag":305,"props":4578,"children":4579},{},[4580],{"type":45,"value":4581},"# (note: join_on keeps both key columns in the output, unlike on=\"key\")\n",{"type":39,"tag":305,"props":4583,"children":4584},{"class":307,"line":435},[4585],{"type":39,"tag":305,"props":4586,"children":4587},{},[4588],{"type":45,"value":4589},"li.join_on(failed, col(\"l_orderkey\") == col(\"o_orderkey\"))\n",{"type":39,"tag":4591,"props":4592,"children":4593},"br",{},[],{"type":45,"value":4595},"When the same column name exists on both sides, ",{"type":39,"tag":69,"props":4597,"children":4599},{"className":4598},[],[4600],{"type":45,"value":4601},"DataFrame.col(name)",{"type":45,"value":4603},"\n(and ",{"type":39,"tag":69,"props":4605,"children":4607},{"className":4606},[],[4608],{"type":45,"value":4609},"DataFrame.column(name)",{"type":45,"value":4611},") returns a column reference qualified to\nthat DataFrame, which disambiguates the predicate explicitly:",{"type":39,"tag":295,"props":4613,"children":4615},{"className":297,"code":4614,"language":18,"meta":299,"style":299},"li.join_on(failed, li.col(\"l_orderkey\") == failed.col(\"o_orderkey\"))\n",[4616],{"type":39,"tag":69,"props":4617,"children":4618},{"__ignoreMap":299},[4619],{"type":39,"tag":305,"props":4620,"children":4621},{"class":307,"line":308},[4622],{"type":39,"tag":305,"props":4623,"children":4624},{},[4625],{"type":45,"value":4614},{"type":39,"tag":48,"props":4627,"children":4629},{"id":4628},"idiomatic-patterns",[4630],{"type":45,"value":4631},"Idiomatic Patterns",{"type":39,"tag":551,"props":4633,"children":4635},{"id":4634},"fluent-chaining",[4636],{"type":45,"value":4637},"Fluent Chaining",{"type":39,"tag":295,"props":4639,"children":4641},{"className":297,"code":4640,"language":18,"meta":299,"style":299},"result = (\n ctx.read_parquet(\"data.parquet\")\n .filter(col(\"year\") >= 2020)\n .select(col(\"region\"), col(\"sales\"))\n .aggregate([\"region\"], [F.sum(col(\"sales\")).alias(\"total\")])\n .sort(col(\"total\").sort(ascending=False))\n .limit(10)\n)\nresult.show()\n",[4642],{"type":39,"tag":69,"props":4643,"children":4644},{"__ignoreMap":299},[4645,4653,4661,4669,4677,4685,4693,4701,4708],{"type":39,"tag":305,"props":4646,"children":4647},{"class":307,"line":308},[4648],{"type":39,"tag":305,"props":4649,"children":4650},{},[4651],{"type":45,"value":4652},"result = (\n",{"type":39,"tag":305,"props":4654,"children":4655},{"class":307,"line":317},[4656],{"type":39,"tag":305,"props":4657,"children":4658},{},[4659],{"type":45,"value":4660}," ctx.read_parquet(\"data.parquet\")\n",{"type":39,"tag":305,"props":4662,"children":4663},{"class":307,"line":326},[4664],{"type":39,"tag":305,"props":4665,"children":4666},{},[4667],{"type":45,"value":4668}," .filter(col(\"year\") >= 2020)\n",{"type":39,"tag":305,"props":4670,"children":4671},{"class":307,"line":373},[4672],{"type":39,"tag":305,"props":4673,"children":4674},{},[4675],{"type":45,"value":4676}," .select(col(\"region\"), col(\"sales\"))\n",{"type":39,"tag":305,"props":4678,"children":4679},{"class":307,"line":382},[4680],{"type":39,"tag":305,"props":4681,"children":4682},{},[4683],{"type":45,"value":4684}," .aggregate([\"region\"], [F.sum(col(\"sales\")).alias(\"total\")])\n",{"type":39,"tag":305,"props":4686,"children":4687},{"class":307,"line":391},[4688],{"type":39,"tag":305,"props":4689,"children":4690},{},[4691],{"type":45,"value":4692}," .sort(col(\"total\").sort(ascending=False))\n",{"type":39,"tag":305,"props":4694,"children":4695},{"class":307,"line":400},[4696],{"type":39,"tag":305,"props":4697,"children":4698},{},[4699],{"type":45,"value":4700}," .limit(10)\n",{"type":39,"tag":305,"props":4702,"children":4703},{"class":307,"line":408},[4704],{"type":39,"tag":305,"props":4705,"children":4706},{},[4707],{"type":45,"value":626},{"type":39,"tag":305,"props":4709,"children":4710},{"class":307,"line":417},[4711],{"type":39,"tag":305,"props":4712,"children":4713},{},[4714],{"type":45,"value":4715},"result.show()\n",{"type":39,"tag":551,"props":4717,"children":4719},{"id":4718},"using-variables-as-ctes",[4720],{"type":45,"value":4721},"Using Variables as CTEs",{"type":39,"tag":55,"props":4723,"children":4724},{},[4725,4727,4733],{"type":45,"value":4726},"Instead of SQL CTEs (",{"type":39,"tag":69,"props":4728,"children":4730},{"className":4729},[],[4731],{"type":45,"value":4732},"WITH ... AS",{"type":45,"value":4734},"), assign intermediate DataFrames to\nvariables:",{"type":39,"tag":295,"props":4736,"children":4738},{"className":297,"code":4737,"language":18,"meta":299,"style":299},"base = ctx.read_parquet(\"orders.parquet\").filter(col(\"status\") == \"shipped\")\nby_region = base.aggregate([\"region\"], [F.sum(col(\"amount\")).alias(\"total\")])\ntop_regions = by_region.filter(col(\"total\") > 10000)\n",[4739],{"type":39,"tag":69,"props":4740,"children":4741},{"__ignoreMap":299},[4742,4750,4757],{"type":39,"tag":305,"props":4743,"children":4744},{"class":307,"line":308},[4745],{"type":39,"tag":305,"props":4746,"children":4747},{},[4748],{"type":45,"value":4749},"base = ctx.read_parquet(\"orders.parquet\").filter(col(\"status\") == \"shipped\")\n",{"type":39,"tag":305,"props":4751,"children":4752},{"class":307,"line":317},[4753],{"type":39,"tag":305,"props":4754,"children":4755},{},[4756],{"type":45,"value":2267},{"type":39,"tag":305,"props":4758,"children":4759},{"class":307,"line":326},[4760],{"type":39,"tag":305,"props":4761,"children":4762},{},[4763],{"type":45,"value":4764},"top_regions = by_region.filter(col(\"total\") > 10000)\n",{"type":39,"tag":551,"props":4766,"children":4768},{"id":4767},"reusing-expressions-as-variables",[4769],{"type":45,"value":4770},"Reusing Expressions as Variables",{"type":39,"tag":55,"props":4772,"children":4773},{},[4774,4776,4781,4783,4788],{"type":45,"value":4775},"Just like DataFrames, expressions (",{"type":39,"tag":69,"props":4777,"children":4779},{"className":4778},[],[4780],{"type":45,"value":213},{"type":45,"value":4782},") can be stored in variables and used\nanywhere an ",{"type":39,"tag":69,"props":4784,"children":4786},{"className":4785},[],[4787],{"type":45,"value":213},{"type":45,"value":4789}," is expected. This is useful for building up complex\nexpressions or reusing a computed value across multiple operations:",{"type":39,"tag":295,"props":4791,"children":4793},{"className":297,"code":4792,"language":18,"meta":299,"style":299},"# Build an expression and reuse it\ndisc_price = col(\"price\") * (lit(1) - col(\"discount\"))\ndf = df.select(\n col(\"id\"),\n disc_price.alias(\"disc_price\"),\n (disc_price * (lit(1) + col(\"tax\"))).alias(\"total\"),\n)\n\n# Use a collected scalar as an expression\nmax_val = result_df.collect_column(\"max_price\")[0] # PyArrow scalar\ncutoff = lit(max_val) - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\ndf = df.filter(col(\"ship_date\") \u003C= cutoff) # cutoff is already an Expr\n",[4794],{"type":39,"tag":69,"props":4795,"children":4796},{"__ignoreMap":299},[4797,4805,4813,4821,4829,4837,4845,4852,4859,4867,4875,4883],{"type":39,"tag":305,"props":4798,"children":4799},{"class":307,"line":308},[4800],{"type":39,"tag":305,"props":4801,"children":4802},{},[4803],{"type":45,"value":4804},"# Build an expression and reuse it\n",{"type":39,"tag":305,"props":4806,"children":4807},{"class":307,"line":317},[4808],{"type":39,"tag":305,"props":4809,"children":4810},{},[4811],{"type":45,"value":4812},"disc_price = col(\"price\") * (lit(1) - col(\"discount\"))\n",{"type":39,"tag":305,"props":4814,"children":4815},{"class":307,"line":326},[4816],{"type":39,"tag":305,"props":4817,"children":4818},{},[4819],{"type":45,"value":4820},"df = df.select(\n",{"type":39,"tag":305,"props":4822,"children":4823},{"class":307,"line":373},[4824],{"type":39,"tag":305,"props":4825,"children":4826},{},[4827],{"type":45,"value":4828}," col(\"id\"),\n",{"type":39,"tag":305,"props":4830,"children":4831},{"class":307,"line":382},[4832],{"type":39,"tag":305,"props":4833,"children":4834},{},[4835],{"type":45,"value":4836}," disc_price.alias(\"disc_price\"),\n",{"type":39,"tag":305,"props":4838,"children":4839},{"class":307,"line":391},[4840],{"type":39,"tag":305,"props":4841,"children":4842},{},[4843],{"type":45,"value":4844}," (disc_price * (lit(1) + col(\"tax\"))).alias(\"total\"),\n",{"type":39,"tag":305,"props":4846,"children":4847},{"class":307,"line":400},[4848],{"type":39,"tag":305,"props":4849,"children":4850},{},[4851],{"type":45,"value":626},{"type":39,"tag":305,"props":4853,"children":4854},{"class":307,"line":408},[4855],{"type":39,"tag":305,"props":4856,"children":4857},{"emptyLinePlaceholder":359},[4858],{"type":45,"value":362},{"type":39,"tag":305,"props":4860,"children":4861},{"class":307,"line":417},[4862],{"type":39,"tag":305,"props":4863,"children":4864},{},[4865],{"type":45,"value":4866},"# Use a collected scalar as an expression\n",{"type":39,"tag":305,"props":4868,"children":4869},{"class":307,"line":426},[4870],{"type":39,"tag":305,"props":4871,"children":4872},{},[4873],{"type":45,"value":4874},"max_val = result_df.collect_column(\"max_price\")[0] # PyArrow scalar\n",{"type":39,"tag":305,"props":4876,"children":4877},{"class":307,"line":435},[4878],{"type":39,"tag":305,"props":4879,"children":4880},{},[4881],{"type":45,"value":4882},"cutoff = lit(max_val) - lit(pa.scalar((0, 90, 0), type=pa.month_day_nano_interval()))\n",{"type":39,"tag":305,"props":4884,"children":4885},{"class":307,"line":444},[4886],{"type":39,"tag":305,"props":4887,"children":4888},{},[4889],{"type":45,"value":4890},"df = df.filter(col(\"ship_date\") \u003C= cutoff) # cutoff is already an Expr\n",{"type":39,"tag":55,"props":4892,"children":4893},{},[4894,4898,4900,4905,4907,4912,4914,4919,4921,4926],{"type":39,"tag":61,"props":4895,"children":4896},{},[4897],{"type":45,"value":2604},{"type":45,"value":4899},": Do not wrap an ",{"type":39,"tag":69,"props":4901,"children":4903},{"className":4902},[],[4904],{"type":45,"value":213},{"type":45,"value":4906}," in ",{"type":39,"tag":69,"props":4908,"children":4910},{"className":4909},[],[4911],{"type":45,"value":2424},{"type":45,"value":4913},". ",{"type":39,"tag":69,"props":4915,"children":4917},{"className":4916},[],[4918],{"type":45,"value":2424},{"type":45,"value":4920}," is for converting\nPython\u002FPyArrow values into expressions. If a value is already an ",{"type":39,"tag":69,"props":4922,"children":4924},{"className":4923},[],[4925],{"type":45,"value":213},{"type":45,"value":4927},", use it\ndirectly.",{"type":39,"tag":551,"props":4929,"children":4931},{"id":4930},"window-functions-for-scalar-subqueries",[4932],{"type":45,"value":4933},"Window Functions for Scalar Subqueries",{"type":39,"tag":55,"props":4935,"children":4936},{},[4937],{"type":45,"value":4938},"Where SQL uses a correlated scalar subquery, the idiomatic DataFrame approach\nis a window function:",{"type":39,"tag":295,"props":4940,"children":4943},{"className":4941,"code":4942,"language":21,"meta":299,"style":299},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","-- SQL scalar subquery\nSELECT *, (SELECT SUM(b) FROM t WHERE t.group = s.group) AS group_total FROM s\n",[4944],{"type":39,"tag":69,"props":4945,"children":4946},{"__ignoreMap":299},[4947,4955],{"type":39,"tag":305,"props":4948,"children":4949},{"class":307,"line":308},[4950],{"type":39,"tag":305,"props":4951,"children":4952},{},[4953],{"type":45,"value":4954},"-- SQL scalar subquery\n",{"type":39,"tag":305,"props":4956,"children":4957},{"class":307,"line":317},[4958],{"type":39,"tag":305,"props":4959,"children":4960},{},[4961],{"type":45,"value":4962},"SELECT *, (SELECT SUM(b) FROM t WHERE t.group = s.group) AS group_total FROM s\n",{"type":39,"tag":295,"props":4964,"children":4966},{"className":297,"code":4965,"language":18,"meta":299,"style":299},"# DataFrame: window function\nwin = Window(partition_by=[col(\"group\")])\ndf = df.with_column(\"group_total\", F.sum(col(\"b\")).over(win))\n",[4967],{"type":39,"tag":69,"props":4968,"children":4969},{"__ignoreMap":299},[4970,4978,4986],{"type":39,"tag":305,"props":4971,"children":4972},{"class":307,"line":308},[4973],{"type":39,"tag":305,"props":4974,"children":4975},{},[4976],{"type":45,"value":4977},"# DataFrame: window function\n",{"type":39,"tag":305,"props":4979,"children":4980},{"class":307,"line":317},[4981],{"type":39,"tag":305,"props":4982,"children":4983},{},[4984],{"type":45,"value":4985},"win = Window(partition_by=[col(\"group\")])\n",{"type":39,"tag":305,"props":4987,"children":4988},{"class":307,"line":326},[4989],{"type":39,"tag":305,"props":4990,"children":4991},{},[4992],{"type":45,"value":4993},"df = df.with_column(\"group_total\", F.sum(col(\"b\")).over(win))\n",{"type":39,"tag":551,"props":4995,"children":4997},{"id":4996},"semianti-joins-for-exists-not-exists",[4998],{"type":45,"value":4999},"Semi\u002FAnti Joins for EXISTS \u002F NOT EXISTS",{"type":39,"tag":295,"props":5001,"children":5003},{"className":4941,"code":5002,"language":21,"meta":299,"style":299},"-- SQL: WHERE EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"semi\")\n\n-- SQL: WHERE NOT EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n-- DataFrame:\nresult = main.join(other, on=\"key\", how=\"anti\")\n",[5004],{"type":39,"tag":69,"props":5005,"children":5006},{"__ignoreMap":299},[5007,5015,5023,5031,5038,5046,5053],{"type":39,"tag":305,"props":5008,"children":5009},{"class":307,"line":308},[5010],{"type":39,"tag":305,"props":5011,"children":5012},{},[5013],{"type":45,"value":5014},"-- SQL: WHERE EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n",{"type":39,"tag":305,"props":5016,"children":5017},{"class":307,"line":317},[5018],{"type":39,"tag":305,"props":5019,"children":5020},{},[5021],{"type":45,"value":5022},"-- DataFrame:\n",{"type":39,"tag":305,"props":5024,"children":5025},{"class":307,"line":326},[5026],{"type":39,"tag":305,"props":5027,"children":5028},{},[5029],{"type":45,"value":5030},"result = main.join(other, on=\"key\", how=\"semi\")\n",{"type":39,"tag":305,"props":5032,"children":5033},{"class":307,"line":373},[5034],{"type":39,"tag":305,"props":5035,"children":5036},{"emptyLinePlaceholder":359},[5037],{"type":45,"value":362},{"type":39,"tag":305,"props":5039,"children":5040},{"class":307,"line":382},[5041],{"type":39,"tag":305,"props":5042,"children":5043},{},[5044],{"type":45,"value":5045},"-- SQL: WHERE NOT EXISTS (SELECT 1 FROM other WHERE other.key = main.key)\n",{"type":39,"tag":305,"props":5047,"children":5048},{"class":307,"line":391},[5049],{"type":39,"tag":305,"props":5050,"children":5051},{},[5052],{"type":45,"value":5022},{"type":39,"tag":305,"props":5054,"children":5055},{"class":307,"line":400},[5056],{"type":39,"tag":305,"props":5057,"children":5058},{},[5059],{"type":45,"value":5060},"result = main.join(other, on=\"key\", how=\"anti\")\n",{"type":39,"tag":551,"props":5062,"children":5064},{"id":5063},"computed-columns",[5065],{"type":45,"value":5066},"Computed Columns",{"type":39,"tag":295,"props":5068,"children":5070},{"className":297,"code":5069,"language":18,"meta":299,"style":299},"# Add computed columns while keeping all originals\ndf = df.with_column(\"full_name\", F.concat(col(\"first\"), lit(\" \"), col(\"last\")))\ndf = df.with_column(\"discounted\", col(\"price\") * lit(0.9))\n",[5071],{"type":39,"tag":69,"props":5072,"children":5073},{"__ignoreMap":299},[5074,5082,5090],{"type":39,"tag":305,"props":5075,"children":5076},{"class":307,"line":308},[5077],{"type":39,"tag":305,"props":5078,"children":5079},{},[5080],{"type":45,"value":5081},"# Add computed columns while keeping all originals\n",{"type":39,"tag":305,"props":5083,"children":5084},{"class":307,"line":317},[5085],{"type":39,"tag":305,"props":5086,"children":5087},{},[5088],{"type":45,"value":5089},"df = df.with_column(\"full_name\", F.concat(col(\"first\"), lit(\" \"), col(\"last\")))\n",{"type":39,"tag":305,"props":5091,"children":5092},{"class":307,"line":326},[5093],{"type":39,"tag":305,"props":5094,"children":5095},{},[5096],{"type":45,"value":5097},"df = df.with_column(\"discounted\", col(\"price\") * lit(0.9))\n",{"type":39,"tag":48,"props":5099,"children":5101},{"id":5100},"available-functions-categorized",[5102],{"type":45,"value":5103},"Available Functions (Categorized)",{"type":39,"tag":55,"props":5105,"children":5106},{},[5107,5109,5114,5116,5122],{"type":45,"value":5108},"The ",{"type":39,"tag":69,"props":5110,"children":5112},{"className":5111},[],[5113],{"type":45,"value":239},{"type":45,"value":5115}," module (imported as ",{"type":39,"tag":69,"props":5117,"children":5119},{"className":5118},[],[5120],{"type":45,"value":5121},"F",{"type":45,"value":5123},") provides 290+ functions. Key categories:",{"type":39,"tag":55,"props":5125,"children":5126},{},[5127,5132,5133,5139,5140,5146,5147,5153,5154,5160,5161,5167,5168,5174,5175,5181,5183,5189,5190,5196,5197,5203,5204,5210,5211,5217,5218,5224,5225,5231,5232,5238,5239,5245,5246,5252,5253,5259,5260,5266,5267,5273,5274,5280,5281,5287,5288,5294,5295,5301,5302,5308,5309,5315,5316,5322],{"type":39,"tag":61,"props":5128,"children":5129},{},[5130],{"type":45,"value":5131},"Aggregate",{"type":45,"value":2606},{"type":39,"tag":69,"props":5134,"children":5136},{"className":5135},[],[5137],{"type":45,"value":5138},"sum",{"type":45,"value":1157},{"type":39,"tag":69,"props":5141,"children":5143},{"className":5142},[],[5144],{"type":45,"value":5145},"avg",{"type":45,"value":1157},{"type":39,"tag":69,"props":5148,"children":5150},{"className":5149},[],[5151],{"type":45,"value":5152},"min",{"type":45,"value":1157},{"type":39,"tag":69,"props":5155,"children":5157},{"className":5156},[],[5158],{"type":45,"value":5159},"max",{"type":45,"value":1157},{"type":39,"tag":69,"props":5162,"children":5164},{"className":5163},[],[5165],{"type":45,"value":5166},"count",{"type":45,"value":1157},{"type":39,"tag":69,"props":5169,"children":5171},{"className":5170},[],[5172],{"type":45,"value":5173},"count_star",{"type":45,"value":1157},{"type":39,"tag":69,"props":5176,"children":5178},{"className":5177},[],[5179],{"type":45,"value":5180},"median",{"type":45,"value":5182},",\n",{"type":39,"tag":69,"props":5184,"children":5186},{"className":5185},[],[5187],{"type":45,"value":5188},"stddev",{"type":45,"value":1157},{"type":39,"tag":69,"props":5191,"children":5193},{"className":5192},[],[5194],{"type":45,"value":5195},"stddev_pop",{"type":45,"value":1157},{"type":39,"tag":69,"props":5198,"children":5200},{"className":5199},[],[5201],{"type":45,"value":5202},"var_samp",{"type":45,"value":1157},{"type":39,"tag":69,"props":5205,"children":5207},{"className":5206},[],[5208],{"type":45,"value":5209},"var_pop",{"type":45,"value":1157},{"type":39,"tag":69,"props":5212,"children":5214},{"className":5213},[],[5215],{"type":45,"value":5216},"corr",{"type":45,"value":1157},{"type":39,"tag":69,"props":5219,"children":5221},{"className":5220},[],[5222],{"type":45,"value":5223},"covar",{"type":45,"value":1157},{"type":39,"tag":69,"props":5226,"children":5228},{"className":5227},[],[5229],{"type":45,"value":5230},"approx_distinct",{"type":45,"value":5182},{"type":39,"tag":69,"props":5233,"children":5235},{"className":5234},[],[5236],{"type":45,"value":5237},"approx_median",{"type":45,"value":1157},{"type":39,"tag":69,"props":5240,"children":5242},{"className":5241},[],[5243],{"type":45,"value":5244},"approx_percentile_cont",{"type":45,"value":1157},{"type":39,"tag":69,"props":5247,"children":5249},{"className":5248},[],[5250],{"type":45,"value":5251},"array_agg",{"type":45,"value":1157},{"type":39,"tag":69,"props":5254,"children":5256},{"className":5255},[],[5257],{"type":45,"value":5258},"string_agg",{"type":45,"value":5182},{"type":39,"tag":69,"props":5261,"children":5263},{"className":5262},[],[5264],{"type":45,"value":5265},"first_value",{"type":45,"value":1157},{"type":39,"tag":69,"props":5268,"children":5270},{"className":5269},[],[5271],{"type":45,"value":5272},"last_value",{"type":45,"value":1157},{"type":39,"tag":69,"props":5275,"children":5277},{"className":5276},[],[5278],{"type":45,"value":5279},"bit_and",{"type":45,"value":1157},{"type":39,"tag":69,"props":5282,"children":5284},{"className":5283},[],[5285],{"type":45,"value":5286},"bit_or",{"type":45,"value":1157},{"type":39,"tag":69,"props":5289,"children":5291},{"className":5290},[],[5292],{"type":45,"value":5293},"bit_xor",{"type":45,"value":1157},{"type":39,"tag":69,"props":5296,"children":5298},{"className":5297},[],[5299],{"type":45,"value":5300},"bool_and",{"type":45,"value":5182},{"type":39,"tag":69,"props":5303,"children":5305},{"className":5304},[],[5306],{"type":45,"value":5307},"bool_or",{"type":45,"value":1157},{"type":39,"tag":69,"props":5310,"children":5312},{"className":5311},[],[5313],{"type":45,"value":5314},"grouping",{"type":45,"value":1157},{"type":39,"tag":69,"props":5317,"children":5319},{"className":5318},[],[5320],{"type":45,"value":5321},"regr_*",{"type":45,"value":5323}," (9 regression functions)",{"type":39,"tag":55,"props":5325,"children":5326},{},[5327,5332,5333,5339,5340,5346,5347,5353,5354,5360,5361,5367,5368,5374,5375,5381,5382,5388,5389,5394,5395,5400,5401],{"type":39,"tag":61,"props":5328,"children":5329},{},[5330],{"type":45,"value":5331},"Window",{"type":45,"value":2606},{"type":39,"tag":69,"props":5334,"children":5336},{"className":5335},[],[5337],{"type":45,"value":5338},"row_number",{"type":45,"value":1157},{"type":39,"tag":69,"props":5341,"children":5343},{"className":5342},[],[5344],{"type":45,"value":5345},"rank",{"type":45,"value":1157},{"type":39,"tag":69,"props":5348,"children":5350},{"className":5349},[],[5351],{"type":45,"value":5352},"dense_rank",{"type":45,"value":1157},{"type":39,"tag":69,"props":5355,"children":5357},{"className":5356},[],[5358],{"type":45,"value":5359},"percent_rank",{"type":45,"value":1157},{"type":39,"tag":69,"props":5362,"children":5364},{"className":5363},[],[5365],{"type":45,"value":5366},"cume_dist",{"type":45,"value":5182},{"type":39,"tag":69,"props":5369,"children":5371},{"className":5370},[],[5372],{"type":45,"value":5373},"ntile",{"type":45,"value":1157},{"type":39,"tag":69,"props":5376,"children":5378},{"className":5377},[],[5379],{"type":45,"value":5380},"lag",{"type":45,"value":1157},{"type":39,"tag":69,"props":5383,"children":5385},{"className":5384},[],[5386],{"type":45,"value":5387},"lead",{"type":45,"value":1157},{"type":39,"tag":69,"props":5390,"children":5392},{"className":5391},[],[5393],{"type":45,"value":5265},{"type":45,"value":1157},{"type":39,"tag":69,"props":5396,"children":5398},{"className":5397},[],[5399],{"type":45,"value":5272},{"type":45,"value":1157},{"type":39,"tag":69,"props":5402,"children":5404},{"className":5403},[],[5405],{"type":45,"value":5406},"nth_value",{"type":39,"tag":55,"props":5408,"children":5409},{},[5410,5415,5416,5422,5423,5429,5430,5436,5437,5443,5444,5450,5451,5457,5458,5464,5465,5471,5472,5478,5479,5485,5486,5492,5493,5499,5500,5506,5507,5513,5514,5520,5521,5527,5528,5534,5535,5541,5542,5548,5549,5555,5556,5562,5563,5569,5570,5576,5577,5583,5584,5590,5591,5597,5598,5604,5605],{"type":39,"tag":61,"props":5411,"children":5412},{},[5413],{"type":45,"value":5414},"String",{"type":45,"value":2606},{"type":39,"tag":69,"props":5417,"children":5419},{"className":5418},[],[5420],{"type":45,"value":5421},"length",{"type":45,"value":1157},{"type":39,"tag":69,"props":5424,"children":5426},{"className":5425},[],[5427],{"type":45,"value":5428},"lower",{"type":45,"value":1157},{"type":39,"tag":69,"props":5431,"children":5433},{"className":5432},[],[5434],{"type":45,"value":5435},"upper",{"type":45,"value":1157},{"type":39,"tag":69,"props":5438,"children":5440},{"className":5439},[],[5441],{"type":45,"value":5442},"trim",{"type":45,"value":1157},{"type":39,"tag":69,"props":5445,"children":5447},{"className":5446},[],[5448],{"type":45,"value":5449},"ltrim",{"type":45,"value":1157},{"type":39,"tag":69,"props":5452,"children":5454},{"className":5453},[],[5455],{"type":45,"value":5456},"rtrim",{"type":45,"value":1157},{"type":39,"tag":69,"props":5459,"children":5461},{"className":5460},[],[5462],{"type":45,"value":5463},"lpad",{"type":45,"value":5182},{"type":39,"tag":69,"props":5466,"children":5468},{"className":5467},[],[5469],{"type":45,"value":5470},"rpad",{"type":45,"value":1157},{"type":39,"tag":69,"props":5473,"children":5475},{"className":5474},[],[5476],{"type":45,"value":5477},"starts_with",{"type":45,"value":1157},{"type":39,"tag":69,"props":5480,"children":5482},{"className":5481},[],[5483],{"type":45,"value":5484},"ends_with",{"type":45,"value":1157},{"type":39,"tag":69,"props":5487,"children":5489},{"className":5488},[],[5490],{"type":45,"value":5491},"contains",{"type":45,"value":1157},{"type":39,"tag":69,"props":5494,"children":5496},{"className":5495},[],[5497],{"type":45,"value":5498},"substr",{"type":45,"value":1157},{"type":39,"tag":69,"props":5501,"children":5503},{"className":5502},[],[5504],{"type":45,"value":5505},"substring",{"type":45,"value":5182},{"type":39,"tag":69,"props":5508,"children":5510},{"className":5509},[],[5511],{"type":45,"value":5512},"replace",{"type":45,"value":1157},{"type":39,"tag":69,"props":5515,"children":5517},{"className":5516},[],[5518],{"type":45,"value":5519},"reverse",{"type":45,"value":1157},{"type":39,"tag":69,"props":5522,"children":5524},{"className":5523},[],[5525],{"type":45,"value":5526},"repeat",{"type":45,"value":1157},{"type":39,"tag":69,"props":5529,"children":5531},{"className":5530},[],[5532],{"type":45,"value":5533},"split_part",{"type":45,"value":1157},{"type":39,"tag":69,"props":5536,"children":5538},{"className":5537},[],[5539],{"type":45,"value":5540},"concat",{"type":45,"value":1157},{"type":39,"tag":69,"props":5543,"children":5545},{"className":5544},[],[5546],{"type":45,"value":5547},"concat_ws",{"type":45,"value":5182},{"type":39,"tag":69,"props":5550,"children":5552},{"className":5551},[],[5553],{"type":45,"value":5554},"initcap",{"type":45,"value":1157},{"type":39,"tag":69,"props":5557,"children":5559},{"className":5558},[],[5560],{"type":45,"value":5561},"ascii",{"type":45,"value":1157},{"type":39,"tag":69,"props":5564,"children":5566},{"className":5565},[],[5567],{"type":45,"value":5568},"chr",{"type":45,"value":1157},{"type":39,"tag":69,"props":5571,"children":5573},{"className":5572},[],[5574],{"type":45,"value":5575},"left",{"type":45,"value":1157},{"type":39,"tag":69,"props":5578,"children":5580},{"className":5579},[],[5581],{"type":45,"value":5582},"right",{"type":45,"value":1157},{"type":39,"tag":69,"props":5585,"children":5587},{"className":5586},[],[5588],{"type":45,"value":5589},"strpos",{"type":45,"value":1157},{"type":39,"tag":69,"props":5592,"children":5594},{"className":5593},[],[5595],{"type":45,"value":5596},"translate",{"type":45,"value":1157},{"type":39,"tag":69,"props":5599,"children":5601},{"className":5600},[],[5602],{"type":45,"value":5603},"overlay",{"type":45,"value":5182},{"type":39,"tag":69,"props":5606,"children":5608},{"className":5607},[],[5609],{"type":45,"value":5610},"levenshtein",{"type":39,"tag":55,"props":5612,"children":5613},{},[5614,5620,5622,5627,5629,5635,5637,5643,5645,5651,5653,5659,5661,5667],{"type":39,"tag":69,"props":5615,"children":5617},{"className":5616},[],[5618],{"type":45,"value":5619},"F.substr(str, start)",{"type":45,"value":5621}," takes ",{"type":39,"tag":61,"props":5623,"children":5624},{},[5625],{"type":45,"value":5626},"only two arguments",{"type":45,"value":5628}," and returns the tail of\nthe string from ",{"type":39,"tag":69,"props":5630,"children":5632},{"className":5631},[],[5633],{"type":45,"value":5634},"start",{"type":45,"value":5636}," onward — passing a third length argument raises\n",{"type":39,"tag":69,"props":5638,"children":5640},{"className":5639},[],[5641],{"type":45,"value":5642},"TypeError: substr() takes 2 positional arguments but 3 were given",{"type":45,"value":5644},". For the\nSQL-style 3-arg form (",{"type":39,"tag":69,"props":5646,"children":5648},{"className":5647},[],[5649],{"type":45,"value":5650},"SUBSTRING(str FROM start FOR length)",{"type":45,"value":5652},"), use\n",{"type":39,"tag":69,"props":5654,"children":5656},{"className":5655},[],[5657],{"type":45,"value":5658},"F.substring(col(\"s\"), lit(start), lit(length))",{"type":45,"value":5660},". For a fixed-length prefix,\n",{"type":39,"tag":69,"props":5662,"children":5664},{"className":5663},[],[5665],{"type":45,"value":5666},"F.left(col(\"s\"), lit(n))",{"type":45,"value":5668}," is cleanest.",{"type":39,"tag":295,"props":5670,"children":5672},{"className":297,"code":5671,"language":18,"meta":299,"style":299},"# WRONG — substr does not accept a length argument\nF.substr(col(\"c_phone\"), lit(1), lit(2))\n# CORRECT\nF.substring(col(\"c_phone\"), lit(1), lit(2)) # explicit length\nF.left(col(\"c_phone\"), lit(2)) # prefix shortcut\n",[5673],{"type":39,"tag":69,"props":5674,"children":5675},{"__ignoreMap":299},[5676,5684,5692,5700,5708],{"type":39,"tag":305,"props":5677,"children":5678},{"class":307,"line":308},[5679],{"type":39,"tag":305,"props":5680,"children":5681},{},[5682],{"type":45,"value":5683},"# WRONG — substr does not accept a length argument\n",{"type":39,"tag":305,"props":5685,"children":5686},{"class":307,"line":317},[5687],{"type":39,"tag":305,"props":5688,"children":5689},{},[5690],{"type":45,"value":5691},"F.substr(col(\"c_phone\"), lit(1), lit(2))\n",{"type":39,"tag":305,"props":5693,"children":5694},{"class":307,"line":326},[5695],{"type":39,"tag":305,"props":5696,"children":5697},{},[5698],{"type":45,"value":5699},"# CORRECT\n",{"type":39,"tag":305,"props":5701,"children":5702},{"class":307,"line":373},[5703],{"type":39,"tag":305,"props":5704,"children":5705},{},[5706],{"type":45,"value":5707},"F.substring(col(\"c_phone\"), lit(1), lit(2)) # explicit length\n",{"type":39,"tag":305,"props":5709,"children":5710},{"class":307,"line":382},[5711],{"type":39,"tag":305,"props":5712,"children":5713},{},[5714],{"type":45,"value":5715},"F.left(col(\"c_phone\"), lit(2)) # prefix shortcut\n",{"type":39,"tag":55,"props":5717,"children":5718},{},[5719,5724,5725,5731,5732,5738,5739,5745,5746,5752,5753,5759,5760,5766,5767,5773,5774,5780,5781,5787,5788,5794,5795,5801,5802,5808,5809,5815,5816,5822,5823,5829,5830,5836,5837,5843,5844,5850,5851,5857,5858,5864,5865,5871],{"type":39,"tag":61,"props":5720,"children":5721},{},[5722],{"type":45,"value":5723},"Math",{"type":45,"value":2606},{"type":39,"tag":69,"props":5726,"children":5728},{"className":5727},[],[5729],{"type":45,"value":5730},"abs",{"type":45,"value":1157},{"type":39,"tag":69,"props":5733,"children":5735},{"className":5734},[],[5736],{"type":45,"value":5737},"ceil",{"type":45,"value":1157},{"type":39,"tag":69,"props":5740,"children":5742},{"className":5741},[],[5743],{"type":45,"value":5744},"floor",{"type":45,"value":1157},{"type":39,"tag":69,"props":5747,"children":5749},{"className":5748},[],[5750],{"type":45,"value":5751},"round",{"type":45,"value":1157},{"type":39,"tag":69,"props":5754,"children":5756},{"className":5755},[],[5757],{"type":45,"value":5758},"trunc",{"type":45,"value":1157},{"type":39,"tag":69,"props":5761,"children":5763},{"className":5762},[],[5764],{"type":45,"value":5765},"sqrt",{"type":45,"value":1157},{"type":39,"tag":69,"props":5768,"children":5770},{"className":5769},[],[5771],{"type":45,"value":5772},"cbrt",{"type":45,"value":1157},{"type":39,"tag":69,"props":5775,"children":5777},{"className":5776},[],[5778],{"type":45,"value":5779},"exp",{"type":45,"value":5182},{"type":39,"tag":69,"props":5782,"children":5784},{"className":5783},[],[5785],{"type":45,"value":5786},"ln",{"type":45,"value":1157},{"type":39,"tag":69,"props":5789,"children":5791},{"className":5790},[],[5792],{"type":45,"value":5793},"log",{"type":45,"value":1157},{"type":39,"tag":69,"props":5796,"children":5798},{"className":5797},[],[5799],{"type":45,"value":5800},"log2",{"type":45,"value":1157},{"type":39,"tag":69,"props":5803,"children":5805},{"className":5804},[],[5806],{"type":45,"value":5807},"log10",{"type":45,"value":1157},{"type":39,"tag":69,"props":5810,"children":5812},{"className":5811},[],[5813],{"type":45,"value":5814},"pow",{"type":45,"value":1157},{"type":39,"tag":69,"props":5817,"children":5819},{"className":5818},[],[5820],{"type":45,"value":5821},"signum",{"type":45,"value":1157},{"type":39,"tag":69,"props":5824,"children":5826},{"className":5825},[],[5827],{"type":45,"value":5828},"pi",{"type":45,"value":1157},{"type":39,"tag":69,"props":5831,"children":5833},{"className":5832},[],[5834],{"type":45,"value":5835},"random",{"type":45,"value":1157},{"type":39,"tag":69,"props":5838,"children":5840},{"className":5839},[],[5841],{"type":45,"value":5842},"factorial",{"type":45,"value":5182},{"type":39,"tag":69,"props":5845,"children":5847},{"className":5846},[],[5848],{"type":45,"value":5849},"gcd",{"type":45,"value":1157},{"type":39,"tag":69,"props":5852,"children":5854},{"className":5853},[],[5855],{"type":45,"value":5856},"lcm",{"type":45,"value":1157},{"type":39,"tag":69,"props":5859,"children":5861},{"className":5860},[],[5862],{"type":45,"value":5863},"greatest",{"type":45,"value":1157},{"type":39,"tag":69,"props":5866,"children":5868},{"className":5867},[],[5869],{"type":45,"value":5870},"least",{"type":45,"value":5872},", sin\u002Fcos\u002Ftan and inverse\u002Fhyperbolic variants",{"type":39,"tag":55,"props":5874,"children":5875},{},[5876,5881,5882,5888,5889,5895,5896,5902,5903,5909,5910,5916,5917,5923,5924,5930,5931,5937,5938,5944,5945,5951,5952,5958,5959,5965,5966,5972,5973,5979,5980,5986,5987,5993,5994,6000,6001,6007,6008,6014,6015,6021,6022,6028,6029],{"type":39,"tag":61,"props":5877,"children":5878},{},[5879],{"type":45,"value":5880},"Date\u002FTime",{"type":45,"value":2606},{"type":39,"tag":69,"props":5883,"children":5885},{"className":5884},[],[5886],{"type":45,"value":5887},"now",{"type":45,"value":1157},{"type":39,"tag":69,"props":5890,"children":5892},{"className":5891},[],[5893],{"type":45,"value":5894},"today",{"type":45,"value":1157},{"type":39,"tag":69,"props":5897,"children":5899},{"className":5898},[],[5900],{"type":45,"value":5901},"current_date",{"type":45,"value":1157},{"type":39,"tag":69,"props":5904,"children":5906},{"className":5905},[],[5907],{"type":45,"value":5908},"current_time",{"type":45,"value":5182},{"type":39,"tag":69,"props":5911,"children":5913},{"className":5912},[],[5914],{"type":45,"value":5915},"current_timestamp",{"type":45,"value":1157},{"type":39,"tag":69,"props":5918,"children":5920},{"className":5919},[],[5921],{"type":45,"value":5922},"date_part",{"type":45,"value":1157},{"type":39,"tag":69,"props":5925,"children":5927},{"className":5926},[],[5928],{"type":45,"value":5929},"date_trunc",{"type":45,"value":1157},{"type":39,"tag":69,"props":5932,"children":5934},{"className":5933},[],[5935],{"type":45,"value":5936},"date_bin",{"type":45,"value":1157},{"type":39,"tag":69,"props":5939,"children":5941},{"className":5940},[],[5942],{"type":45,"value":5943},"extract",{"type":45,"value":5182},{"type":39,"tag":69,"props":5946,"children":5948},{"className":5947},[],[5949],{"type":45,"value":5950},"to_timestamp",{"type":45,"value":1157},{"type":39,"tag":69,"props":5953,"children":5955},{"className":5954},[],[5956],{"type":45,"value":5957},"to_timestamp_millis",{"type":45,"value":1157},{"type":39,"tag":69,"props":5960,"children":5962},{"className":5961},[],[5963],{"type":45,"value":5964},"to_timestamp_micros",{"type":45,"value":5182},{"type":39,"tag":69,"props":5967,"children":5969},{"className":5968},[],[5970],{"type":45,"value":5971},"to_timestamp_nanos",{"type":45,"value":1157},{"type":39,"tag":69,"props":5974,"children":5976},{"className":5975},[],[5977],{"type":45,"value":5978},"to_timestamp_seconds",{"type":45,"value":1157},{"type":39,"tag":69,"props":5981,"children":5983},{"className":5982},[],[5984],{"type":45,"value":5985},"to_unixtime",{"type":45,"value":1157},{"type":39,"tag":69,"props":5988,"children":5990},{"className":5989},[],[5991],{"type":45,"value":5992},"from_unixtime",{"type":45,"value":5182},{"type":39,"tag":69,"props":5995,"children":5997},{"className":5996},[],[5998],{"type":45,"value":5999},"make_date",{"type":45,"value":1157},{"type":39,"tag":69,"props":6002,"children":6004},{"className":6003},[],[6005],{"type":45,"value":6006},"make_time",{"type":45,"value":1157},{"type":39,"tag":69,"props":6009,"children":6011},{"className":6010},[],[6012],{"type":45,"value":6013},"to_date",{"type":45,"value":1157},{"type":39,"tag":69,"props":6016,"children":6018},{"className":6017},[],[6019],{"type":45,"value":6020},"to_time",{"type":45,"value":1157},{"type":39,"tag":69,"props":6023,"children":6025},{"className":6024},[],[6026],{"type":45,"value":6027},"to_local_time",{"type":45,"value":1157},{"type":39,"tag":69,"props":6030,"children":6032},{"className":6031},[],[6033],{"type":45,"value":6034},"date_format",{"type":39,"tag":55,"props":6036,"children":6037},{},[6038,6043,6044,6050,6051,6057,6058,6064,6065,6071,6072,6078,6079,6085,6086],{"type":39,"tag":61,"props":6039,"children":6040},{},[6041],{"type":45,"value":6042},"Conditional",{"type":45,"value":2606},{"type":39,"tag":69,"props":6045,"children":6047},{"className":6046},[],[6048],{"type":45,"value":6049},"case",{"type":45,"value":1157},{"type":39,"tag":69,"props":6052,"children":6054},{"className":6053},[],[6055],{"type":45,"value":6056},"when",{"type":45,"value":1157},{"type":39,"tag":69,"props":6059,"children":6061},{"className":6060},[],[6062],{"type":45,"value":6063},"coalesce",{"type":45,"value":1157},{"type":39,"tag":69,"props":6066,"children":6068},{"className":6067},[],[6069],{"type":45,"value":6070},"nullif",{"type":45,"value":1157},{"type":39,"tag":69,"props":6073,"children":6075},{"className":6074},[],[6076],{"type":45,"value":6077},"ifnull",{"type":45,"value":1157},{"type":39,"tag":69,"props":6080,"children":6082},{"className":6081},[],[6083],{"type":45,"value":6084},"nvl",{"type":45,"value":1157},{"type":39,"tag":69,"props":6087,"children":6089},{"className":6088},[],[6090],{"type":45,"value":6091},"nvl2",{"type":39,"tag":55,"props":6093,"children":6094},{},[6095,6100,6101,6107,6108,6114,6115,6120,6121,6127,6128,6134,6135,6141,6142,6148,6149,6155,6156,6162,6163,6169,6170,6176,6177,6183,6184,6190,6191,6197,6198,6204,6205,6211,6212,6218,6219,6225,6226,6232,6233,6239,6240,6246,6247,6253,6254,6260,6261,6267,6269,6275,6277,6283],{"type":39,"tag":61,"props":6096,"children":6097},{},[6098],{"type":45,"value":6099},"Array\u002FList",{"type":45,"value":2606},{"type":39,"tag":69,"props":6102,"children":6104},{"className":6103},[],[6105],{"type":45,"value":6106},"array",{"type":45,"value":1157},{"type":39,"tag":69,"props":6109,"children":6111},{"className":6110},[],[6112],{"type":45,"value":6113},"make_array",{"type":45,"value":1157},{"type":39,"tag":69,"props":6116,"children":6118},{"className":6117},[],[6119],{"type":45,"value":5251},{"type":45,"value":1157},{"type":39,"tag":69,"props":6122,"children":6124},{"className":6123},[],[6125],{"type":45,"value":6126},"array_length",{"type":45,"value":5182},{"type":39,"tag":69,"props":6129,"children":6131},{"className":6130},[],[6132],{"type":45,"value":6133},"array_element",{"type":45,"value":1157},{"type":39,"tag":69,"props":6136,"children":6138},{"className":6137},[],[6139],{"type":45,"value":6140},"array_slice",{"type":45,"value":1157},{"type":39,"tag":69,"props":6143,"children":6145},{"className":6144},[],[6146],{"type":45,"value":6147},"array_append",{"type":45,"value":1157},{"type":39,"tag":69,"props":6150,"children":6152},{"className":6151},[],[6153],{"type":45,"value":6154},"array_prepend",{"type":45,"value":5182},{"type":39,"tag":69,"props":6157,"children":6159},{"className":6158},[],[6160],{"type":45,"value":6161},"array_concat",{"type":45,"value":1157},{"type":39,"tag":69,"props":6164,"children":6166},{"className":6165},[],[6167],{"type":45,"value":6168},"array_contains",{"type":45,"value":1157},{"type":39,"tag":69,"props":6171,"children":6173},{"className":6172},[],[6174],{"type":45,"value":6175},"array_has",{"type":45,"value":1157},{"type":39,"tag":69,"props":6178,"children":6180},{"className":6179},[],[6181],{"type":45,"value":6182},"array_has_all",{"type":45,"value":1157},{"type":39,"tag":69,"props":6185,"children":6187},{"className":6186},[],[6188],{"type":45,"value":6189},"array_has_any",{"type":45,"value":1157},{"type":39,"tag":69,"props":6192,"children":6194},{"className":6193},[],[6195],{"type":45,"value":6196},"array_position",{"type":45,"value":5182},{"type":39,"tag":69,"props":6199,"children":6201},{"className":6200},[],[6202],{"type":45,"value":6203},"array_remove",{"type":45,"value":1157},{"type":39,"tag":69,"props":6206,"children":6208},{"className":6207},[],[6209],{"type":45,"value":6210},"array_distinct",{"type":45,"value":1157},{"type":39,"tag":69,"props":6213,"children":6215},{"className":6214},[],[6216],{"type":45,"value":6217},"array_sort",{"type":45,"value":1157},{"type":39,"tag":69,"props":6220,"children":6222},{"className":6221},[],[6223],{"type":45,"value":6224},"array_reverse",{"type":45,"value":1157},{"type":39,"tag":69,"props":6227,"children":6229},{"className":6228},[],[6230],{"type":45,"value":6231},"flatten",{"type":45,"value":5182},{"type":39,"tag":69,"props":6234,"children":6236},{"className":6235},[],[6237],{"type":45,"value":6238},"array_to_string",{"type":45,"value":1157},{"type":39,"tag":69,"props":6241,"children":6243},{"className":6242},[],[6244],{"type":45,"value":6245},"array_intersect",{"type":45,"value":1157},{"type":39,"tag":69,"props":6248,"children":6250},{"className":6249},[],[6251],{"type":45,"value":6252},"array_union",{"type":45,"value":1157},{"type":39,"tag":69,"props":6255,"children":6257},{"className":6256},[],[6258],{"type":45,"value":6259},"array_except",{"type":45,"value":5182},{"type":39,"tag":69,"props":6262,"children":6264},{"className":6263},[],[6265],{"type":45,"value":6266},"generate_series",{"type":45,"value":6268},"\n(Most ",{"type":39,"tag":69,"props":6270,"children":6272},{"className":6271},[],[6273],{"type":45,"value":6274},"array_*",{"type":45,"value":6276}," functions also have ",{"type":39,"tag":69,"props":6278,"children":6280},{"className":6279},[],[6281],{"type":45,"value":6282},"list_*",{"type":45,"value":6284}," aliases.)",{"type":39,"tag":55,"props":6286,"children":6287},{},[6288,6293,6294,6300,6301,6307,6308,6314,6315,6321,6322,6328,6329,6335,6336,6342,6343],{"type":39,"tag":61,"props":6289,"children":6290},{},[6291],{"type":45,"value":6292},"Struct\u002FMap",{"type":45,"value":2606},{"type":39,"tag":69,"props":6295,"children":6297},{"className":6296},[],[6298],{"type":45,"value":6299},"struct",{"type":45,"value":1157},{"type":39,"tag":69,"props":6302,"children":6304},{"className":6303},[],[6305],{"type":45,"value":6306},"named_struct",{"type":45,"value":1157},{"type":39,"tag":69,"props":6309,"children":6311},{"className":6310},[],[6312],{"type":45,"value":6313},"get_field",{"type":45,"value":1157},{"type":39,"tag":69,"props":6316,"children":6318},{"className":6317},[],[6319],{"type":45,"value":6320},"make_map",{"type":45,"value":1157},{"type":39,"tag":69,"props":6323,"children":6325},{"className":6324},[],[6326],{"type":45,"value":6327},"map_keys",{"type":45,"value":5182},{"type":39,"tag":69,"props":6330,"children":6332},{"className":6331},[],[6333],{"type":45,"value":6334},"map_values",{"type":45,"value":1157},{"type":39,"tag":69,"props":6337,"children":6339},{"className":6338},[],[6340],{"type":45,"value":6341},"map_entries",{"type":45,"value":1157},{"type":39,"tag":69,"props":6344,"children":6346},{"className":6345},[],[6347],{"type":45,"value":6348},"map_extract",{"type":39,"tag":55,"props":6350,"children":6351},{},[6352,6357,6358,6364,6365,6371,6372,6378,6379,6385,6386],{"type":39,"tag":61,"props":6353,"children":6354},{},[6355],{"type":45,"value":6356},"Regex",{"type":45,"value":2606},{"type":39,"tag":69,"props":6359,"children":6361},{"className":6360},[],[6362],{"type":45,"value":6363},"regexp_like",{"type":45,"value":1157},{"type":39,"tag":69,"props":6366,"children":6368},{"className":6367},[],[6369],{"type":45,"value":6370},"regexp_match",{"type":45,"value":1157},{"type":39,"tag":69,"props":6373,"children":6375},{"className":6374},[],[6376],{"type":45,"value":6377},"regexp_replace",{"type":45,"value":1157},{"type":39,"tag":69,"props":6380,"children":6382},{"className":6381},[],[6383],{"type":45,"value":6384},"regexp_count",{"type":45,"value":5182},{"type":39,"tag":69,"props":6387,"children":6389},{"className":6388},[],[6390],{"type":45,"value":6391},"regexp_instr",{"type":39,"tag":55,"props":6393,"children":6394},{},[6395,6400,6401,6407,6408,6414,6415,6421,6422,6428,6429,6435,6436],{"type":39,"tag":61,"props":6396,"children":6397},{},[6398],{"type":45,"value":6399},"Hash",{"type":45,"value":2606},{"type":39,"tag":69,"props":6402,"children":6404},{"className":6403},[],[6405],{"type":45,"value":6406},"md5",{"type":45,"value":1157},{"type":39,"tag":69,"props":6409,"children":6411},{"className":6410},[],[6412],{"type":45,"value":6413},"sha224",{"type":45,"value":1157},{"type":39,"tag":69,"props":6416,"children":6418},{"className":6417},[],[6419],{"type":45,"value":6420},"sha256",{"type":45,"value":1157},{"type":39,"tag":69,"props":6423,"children":6425},{"className":6424},[],[6426],{"type":45,"value":6427},"sha384",{"type":45,"value":1157},{"type":39,"tag":69,"props":6430,"children":6432},{"className":6431},[],[6433],{"type":45,"value":6434},"sha512",{"type":45,"value":1157},{"type":39,"tag":69,"props":6437,"children":6439},{"className":6438},[],[6440],{"type":45,"value":6441},"digest",{"type":39,"tag":55,"props":6443,"children":6444},{},[6445,6450,6451,6457,6458,6464,6465,6471,6472,6478,6479,6485,6486,6492,6493],{"type":39,"tag":61,"props":6446,"children":6447},{},[6448],{"type":45,"value":6449},"Type",{"type":45,"value":2606},{"type":39,"tag":69,"props":6452,"children":6454},{"className":6453},[],[6455],{"type":45,"value":6456},"arrow_typeof",{"type":45,"value":1157},{"type":39,"tag":69,"props":6459,"children":6461},{"className":6460},[],[6462],{"type":45,"value":6463},"arrow_cast",{"type":45,"value":1157},{"type":39,"tag":69,"props":6466,"children":6468},{"className":6467},[],[6469],{"type":45,"value":6470},"arrow_try_cast",{"type":45,"value":1157},{"type":39,"tag":69,"props":6473,"children":6475},{"className":6474},[],[6476],{"type":45,"value":6477},"arrow_field",{"type":45,"value":5182},{"type":39,"tag":69,"props":6480,"children":6482},{"className":6481},[],[6483],{"type":45,"value":6484},"arrow_metadata",{"type":45,"value":1157},{"type":39,"tag":69,"props":6487,"children":6489},{"className":6488},[],[6490],{"type":45,"value":6491},"cast_to_type",{"type":45,"value":1157},{"type":39,"tag":69,"props":6494,"children":6496},{"className":6495},[],[6497],{"type":45,"value":6498},"with_metadata",{"type":39,"tag":55,"props":6500,"children":6501},{},[6502,6504,6510,6512,6517,6518,6524,6526,6532],{"type":45,"value":6503},"Note: ",{"type":39,"tag":69,"props":6505,"children":6507},{"className":6506},[],[6508],{"type":45,"value":6509},"cast_to_type(value, type_ref, *, try_cast=False)",{"type":45,"value":6511}," is the single\nPython entry point for both upstream ",{"type":39,"tag":69,"props":6513,"children":6515},{"className":6514},[],[6516],{"type":45,"value":6491},{"type":45,"value":687},{"type":39,"tag":69,"props":6519,"children":6521},{"className":6520},[],[6522],{"type":45,"value":6523},"try_cast_to_type",{"type":45,"value":6525},";\npass ",{"type":39,"tag":69,"props":6527,"children":6529},{"className":6528},[],[6530],{"type":45,"value":6531},"try_cast=True",{"type":45,"value":6533}," for the variant that returns NULL on failure.",{"type":39,"tag":55,"props":6535,"children":6536},{},[6537,6542,6543,6549,6550,6555,6556,6562,6563,6569,6570,6576,6577,6583,6584,6590,6591,6597,6598,6604,6605,6611,6612,6618,6619],{"type":39,"tag":61,"props":6538,"children":6539},{},[6540],{"type":45,"value":6541},"Other",{"type":45,"value":2606},{"type":39,"tag":69,"props":6544,"children":6546},{"className":6545},[],[6547],{"type":45,"value":6548},"in_list",{"type":45,"value":1157},{"type":39,"tag":69,"props":6551,"children":6553},{"className":6552},[],[6554],{"type":45,"value":4307},{"type":45,"value":1157},{"type":39,"tag":69,"props":6557,"children":6559},{"className":6558},[],[6560],{"type":45,"value":6561},"alias",{"type":45,"value":1157},{"type":39,"tag":69,"props":6564,"children":6566},{"className":6565},[],[6567],{"type":45,"value":6568},"col",{"type":45,"value":1157},{"type":39,"tag":69,"props":6571,"children":6573},{"className":6572},[],[6574],{"type":45,"value":6575},"encode",{"type":45,"value":1157},{"type":39,"tag":69,"props":6578,"children":6580},{"className":6579},[],[6581],{"type":45,"value":6582},"decode",{"type":45,"value":5182},{"type":39,"tag":69,"props":6585,"children":6587},{"className":6586},[],[6588],{"type":45,"value":6589},"to_hex",{"type":45,"value":1157},{"type":39,"tag":69,"props":6592,"children":6594},{"className":6593},[],[6595],{"type":45,"value":6596},"to_char",{"type":45,"value":1157},{"type":39,"tag":69,"props":6599,"children":6601},{"className":6600},[],[6602],{"type":45,"value":6603},"uuid",{"type":45,"value":1157},{"type":39,"tag":69,"props":6606,"children":6608},{"className":6607},[],[6609],{"type":45,"value":6610},"version",{"type":45,"value":1157},{"type":39,"tag":69,"props":6613,"children":6615},{"className":6614},[],[6616],{"type":45,"value":6617},"bit_length",{"type":45,"value":1157},{"type":39,"tag":69,"props":6620,"children":6622},{"className":6621},[],[6623],{"type":45,"value":6624},"octet_length",{"type":39,"tag":551,"props":6626,"children":6628},{"id":6627},"spark-compatible-functions",[6629],{"type":45,"value":6630},"Spark-Compatible Functions",{"type":39,"tag":55,"props":6632,"children":6633},{},[6634,6636,6642,6644,6649],{"type":45,"value":6635},"A separate ",{"type":39,"tag":69,"props":6637,"children":6639},{"className":6638},[],[6640],{"type":45,"value":6641},"datafusion.functions.spark",{"type":45,"value":6643}," namespace mirrors the\n",{"type":39,"tag":69,"props":6645,"children":6647},{"className":6646},[],[6648],{"type":45,"value":276},{"type":45,"value":6650}," API for callers porting code from PySpark.",{"type":39,"tag":295,"props":6652,"children":6654},{"className":297,"code":6653,"language":18,"meta":299,"style":299},"from datafusion.functions import spark\n",[6655],{"type":39,"tag":69,"props":6656,"children":6657},{"__ignoreMap":299},[6658],{"type":39,"tag":305,"props":6659,"children":6660},{"class":307,"line":308},[6661],{"type":39,"tag":305,"props":6662,"children":6663},{},[6664],{"type":45,"value":6653},{"type":39,"tag":55,"props":6666,"children":6667},{},[6668],{"type":45,"value":6669},"Use it for DataFrame work; for SQL, register the Spark UDFs first:",{"type":39,"tag":295,"props":6671,"children":6673},{"className":297,"code":6672,"language":18,"meta":299,"style":299},"ctx = SessionContext()\nctx.enable_spark_functions() # makes Spark UDFs visible to SQL\nctx.sql(\"SELECT sha2('hello', 256)\").show()\n",[6674],{"type":39,"tag":69,"props":6675,"children":6676},{"__ignoreMap":299},[6677,6684,6692],{"type":39,"tag":305,"props":6678,"children":6679},{"class":307,"line":308},[6680],{"type":39,"tag":305,"props":6681,"children":6682},{},[6683],{"type":45,"value":353},{"type":39,"tag":305,"props":6685,"children":6686},{"class":307,"line":317},[6687],{"type":39,"tag":305,"props":6688,"children":6689},{},[6690],{"type":45,"value":6691},"ctx.enable_spark_functions() # makes Spark UDFs visible to SQL\n",{"type":39,"tag":305,"props":6693,"children":6694},{"class":307,"line":326},[6695],{"type":39,"tag":305,"props":6696,"children":6697},{},[6698],{"type":45,"value":6699},"ctx.sql(\"SELECT sha2('hello', 256)\").show()\n",{"type":39,"tag":55,"props":6701,"children":6702},{},[6703,6705,6711,6713,6719],{"type":45,"value":6704},"Coverage spans aggregate, array, bitmap, bitwise, datetime, hash, JSON,\nmap, math, string, URL, and conditional categories. The authoritative\nlist of what is currently exposed is the ",{"type":39,"tag":69,"props":6706,"children":6708},{"className":6707},[],[6709],{"type":45,"value":6710},"__all__",{"type":45,"value":6712}," in\n",{"type":39,"tag":69,"props":6714,"children":6716},{"className":6715},[],[6717],{"type":45,"value":6718},"python\u002Fdatafusion\u002Ffunctions\u002Fspark.py",{"type":45,"value":3184},{"type":39,"tag":295,"props":6721,"children":6725},{"className":6722,"code":6723,"language":6724,"meta":299,"style":299},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python -c \"from datafusion.functions import spark; print(sorted(spark.__all__))\"\n","bash",[6726],{"type":39,"tag":69,"props":6727,"children":6728},{"__ignoreMap":299},[6729],{"type":39,"tag":305,"props":6730,"children":6731},{"class":307,"line":308},[6732,6737,6743,6749,6754],{"type":39,"tag":305,"props":6733,"children":6735},{"style":6734},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[6736],{"type":45,"value":18},{"type":39,"tag":305,"props":6738,"children":6740},{"style":6739},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[6741],{"type":45,"value":6742}," -c",{"type":39,"tag":305,"props":6744,"children":6746},{"style":6745},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[6747],{"type":45,"value":6748}," \"",{"type":39,"tag":305,"props":6750,"children":6751},{"style":6739},[6752],{"type":45,"value":6753},"from datafusion.functions import spark; print(sorted(spark.__all__))",{"type":39,"tag":305,"props":6755,"children":6756},{"style":6745},[6757],{"type":45,"value":6758},"\"\n",{"type":39,"tag":55,"props":6760,"children":6761},{},[6762,6764,6769],{"type":45,"value":6763},"When you need to know whether a specific pyspark function is available,\ncheck ",{"type":39,"tag":69,"props":6765,"children":6767},{"className":6766},[],[6768],{"type":45,"value":6710},{"type":45,"value":6770}," rather than this skill — the list there moves with the\ncode; any enumeration here would drift.",{"type":39,"tag":55,"props":6772,"children":6773},{},[6774,6779,6781,6786,6787,6792],{"type":39,"tag":61,"props":6775,"children":6776},{},[6777],{"type":45,"value":6778},"Semantic divergences vs the default namespace.",{"type":45,"value":6780}," Functions that exist in\nboth ",{"type":39,"tag":69,"props":6782,"children":6784},{"className":6783},[],[6785],{"type":45,"value":239},{"type":45,"value":687},{"type":39,"tag":69,"props":6788,"children":6790},{"className":6789},[],[6791],{"type":45,"value":265},{"type":45,"value":6793}," may behave differently:",{"type":39,"tag":123,"props":6795,"children":6796},{},[6797,6826],{"type":39,"tag":127,"props":6798,"children":6799},{},[6800],{"type":39,"tag":131,"props":6801,"children":6802},{},[6803,6808,6818],{"type":39,"tag":135,"props":6804,"children":6805},{},[6806],{"type":45,"value":6807},"Function",{"type":39,"tag":135,"props":6809,"children":6810},{},[6811,6813],{"type":45,"value":6812},"Default ",{"type":39,"tag":69,"props":6814,"children":6816},{"className":6815},[],[6817],{"type":45,"value":239},{"type":39,"tag":135,"props":6819,"children":6820},{},[6821],{"type":39,"tag":69,"props":6822,"children":6824},{"className":6823},[],[6825],{"type":45,"value":265},{"type":39,"tag":151,"props":6827,"children":6828},{},[6829,6850,6871],{"type":39,"tag":131,"props":6830,"children":6831},{},[6832,6840,6845],{"type":39,"tag":158,"props":6833,"children":6834},{},[6835],{"type":39,"tag":69,"props":6836,"children":6838},{"className":6837},[],[6839],{"type":45,"value":5540},{"type":39,"tag":158,"props":6841,"children":6842},{},[6843],{"type":45,"value":6844},"NULL inputs treated as empty",{"type":39,"tag":158,"props":6846,"children":6847},{},[6848],{"type":45,"value":6849},"NULL inputs propagate to NULL",{"type":39,"tag":131,"props":6851,"children":6852},{},[6853,6861,6866],{"type":39,"tag":158,"props":6854,"children":6855},{},[6856],{"type":39,"tag":69,"props":6857,"children":6859},{"className":6858},[],[6860],{"type":45,"value":5751},{"type":39,"tag":158,"props":6862,"children":6863},{},[6864],{"type":45,"value":6865},"HALF_EVEN (banker's)",{"type":39,"tag":158,"props":6867,"children":6868},{},[6869],{"type":45,"value":6870},"HALF_UP",{"type":39,"tag":131,"props":6872,"children":6873},{},[6874,6882,6887],{"type":39,"tag":158,"props":6875,"children":6876},{},[6877],{"type":39,"tag":69,"props":6878,"children":6880},{"className":6879},[],[6881],{"type":45,"value":5758},{"type":39,"tag":158,"props":6883,"children":6884},{},[6885],{"type":45,"value":6886},"Numeric truncation",{"type":39,"tag":158,"props":6888,"children":6889},{},[6890],{"type":45,"value":6891},"Date truncation",{"type":39,"tag":55,"props":6893,"children":6894},{},[6895,6897,6903],{"type":45,"value":6896},"Pick the namespace whose semantics match your intent — both stay imported\nside by side; ",{"type":39,"tag":69,"props":6898,"children":6900},{"className":6899},[],[6901],{"type":45,"value":6902},"enable_spark_functions()",{"type":45,"value":6904}," only affects SQL.",{"type":39,"tag":55,"props":6906,"children":6907},{},[6908,6913,6915,6920,6921,6927,6928,6934,6935,6941],{"type":39,"tag":61,"props":6909,"children":6910},{},[6911],{"type":45,"value":6912},"Parameter names match pyspark exactly.",{"type":45,"value":6914}," The spark namespace uses\npyspark parameter names (",{"type":39,"tag":69,"props":6916,"children":6918},{"className":6917},[],[6919],{"type":45,"value":6568},{"type":45,"value":1157},{"type":39,"tag":69,"props":6922,"children":6924},{"className":6923},[],[6925],{"type":45,"value":6926},"str",{"type":45,"value":1157},{"type":39,"tag":69,"props":6929,"children":6931},{"className":6930},[],[6932],{"type":45,"value":6933},"numBits",{"type":45,"value":1157},{"type":39,"tag":69,"props":6936,"children":6938},{"className":6937},[],[6939],{"type":45,"value":6940},"partToExtract",{"type":45,"value":6942},", ...) so\nyou can paste pyspark code and keep keyword arguments working. The default\nnamespace keeps DataFusion's parameter names.",{"type":39,"tag":6944,"props":6945,"children":6946},"style",{},[6947],{"type":45,"value":6948},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":6950,"total":308},[6951],{"slug":4,"name":4,"fn":5,"description":6,"org":6952,"tags":6953,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6954,6955,6956],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"items":6958,"total":7101},[6959,6965,6981,6994,7007,7020,7038,7049,7059,7070,7080,7090],{"slug":4,"name":4,"fn":5,"description":6,"org":6960,"tags":6961,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6962,6963,6964],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"slug":6966,"name":6966,"fn":6967,"description":6968,"org":6969,"tags":6970,"stars":6978,"repoUrl":6979,"updatedAt":6980},"bydbql","generate and execute BanyanDB BydbQL queries","Generate, validate, and optionally execute read-only BanyanDB BydbQL for STREAM, MEASURE, TRACE, and PROPERTY resources. Use when the user asks to query BanyanDB, translate natural language to BydbQL, inspect BanyanDB schema or data, validate BydbQL, or fetch raw BanyanDB records.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6971,6974,6977],{"name":6972,"slug":6973,"type":15},"Analytics","analytics",{"name":6975,"slug":6976,"type":15},"Database","database",{"name":20,"slug":21,"type":15},344,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fskywalking-banyandb","2026-07-12T08:31:01.294423",{"slug":6982,"name":6982,"fn":6983,"description":6984,"org":6985,"tags":6986,"stars":6978,"repoUrl":6979,"updatedAt":6993},"compiling","compile and build BanyanDB projects","Compile and build the SkyWalking BanyanDB project. Use when the user asks to compile, build, or generate code for this project.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6987,6990],{"name":6988,"slug":6989,"type":15},"Build","build",{"name":6991,"slug":6992,"type":15},"Engineering","engineering","2026-07-12T08:31:06.373309",{"slug":6995,"name":6995,"fn":6996,"description":6997,"org":6998,"tags":6999,"stars":6978,"repoUrl":6979,"updatedAt":7006},"gh-pull-request","create GitHub pull requests for BanyanDB","Create a GitHub pull request for SkyWalking BanyanDB. Use when the user asks to create a PR, submit changes, or open a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7000,7003],{"name":7001,"slug":7002,"type":15},"GitHub","github",{"name":7004,"slug":7005,"type":15},"Pull Requests","pull-requests","2026-07-12T08:31:03.792415",{"slug":7008,"name":7008,"fn":7009,"description":7010,"org":7011,"tags":7012,"stars":6978,"repoUrl":6979,"updatedAt":7019},"vendor-update","update Go and Node.js vendor dependencies","Upgrade Go\u002FNode.js vendor dependencies and sync tool versions. Use whenever the user says \"upgrade dependencies\", \"update vendors\", \"vendor update\", \"run vendor-upgrade\", \"bump dependencies\", \"update packages\", or asks to run the `vendor-update` Make target. This skill also checks `scripts\u002Fbuild\u002Fversion.mk` after upgrading to see if any tracked tool versions need updating too, and removes stale binaries from `bin\u002F` when versions change.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7013,7016],{"name":7014,"slug":7015,"type":15},"Go","go",{"name":7017,"slug":7018,"type":15},"Node.js","node-js","2026-07-12T08:31:02.555555",{"slug":7021,"name":7021,"fn":7022,"description":7023,"org":7024,"tags":7025,"stars":7035,"repoUrl":7036,"updatedAt":7037},"cayenne-cgen","generate Cayenne entity Java classes","Use this skill whenever the user wants to (re)generate Cayenne entity Java classes from a DataMap. Trigger on phrases like 'generate Java classes', 'regenerate entities', 'run cgen', 'create the entity classes', 'why is the Artist class missing fields', 'where did the `_Abstract*` classes come from', 'sync the entity classes with the model', or any request to materialize Java from the DataMap. Also trigger as a follow-up after modeling changes (someone added an entity, attribute, or relationship and now the Java side is stale). This skill exclusively uses the `mcp__cayenne__cgen_run` MCP tool — it does NOT use `mvn cayenne:cgen` or the Gradle cgen task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7026,7029,7032],{"name":7027,"slug":7028,"type":15},"Data Modeling","data-modeling",{"name":7030,"slug":7031,"type":15},"Java","java",{"name":7033,"slug":7034,"type":15},"ORM","orm",343,"https:\u002F\u002Fgithub.com\u002Fapache\u002Fcayenne","2026-07-12T08:32:33.575211",{"slug":7039,"name":7039,"fn":7040,"description":7041,"org":7042,"tags":7043,"stars":7035,"repoUrl":7036,"updatedAt":7048},"cayenne-db-import","import database schema into Cayenne DataMaps","Use this skill when the user wants to import database schema metadata into a Cayenne DataMap — the *model\u002Fmapping only*, not names or Java classes. Trigger on phrases like 'reverse engineer the database', 'import the schema', 'generate a DataMap from my DB', 'add the new tables from the DB into the model', 'import the customer table', 'create entities from these tables', or any request to read database metadata to populate or update a DataMap's XML. This is for *full schema* or *bulk table* import; one-off a-la-carte entity additions belong in the cayenne-modeling skill. IMPORTANT — scope: this imports the mapping ONLY; it does not clean up the Object-layer names or (re)generate Java classes. When the user wants their whole project brought in line with the DB ('sync my project with the database', 'my schema changed, update everything', 'update my entities\u002Fclasses from the DB'), that is the end-to-end `cayenne-full-db-sync` skill, which runs this import and then name cleanup and class generation. To regenerate classes alone use `cayenne-cgen`. The skill runs reverse engineering directly via the `mcp__cayenne__dbimport_run` MCP tool when a DBConnector is already configured; otherwise it opens the CayenneModeler GUI via `mcp__cayenne__open_project` to configure the connection first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7044,7045,7046,7047],{"name":6975,"slug":6976,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},{"name":20,"slug":21,"type":15},"2026-07-19T05:40:33.655062",{"slug":7050,"name":7050,"fn":7051,"description":7052,"org":7053,"tags":7054,"stars":7035,"repoUrl":7036,"updatedAt":7058},"cayenne-full-db-sync","synchronize Cayenne projects with database","Use this skill when the user wants to bring their WHOLE Cayenne project in line with the database in one shot — the mapping, the Object-layer names, and the generated Java classes together. This is the end-to-end 'sync with the DB' workflow, and it orchestrates three skills in order: `cayenne-db-import` (import schema metadata into the DataMap) → `cayenne-model-naming` (polish the just-imported names) → `cayenne-cgen` (regenerate Java classes). Trigger on holistic phrases like 'sync my project with the database', 'sync with the DB', 'my schema changed, update everything', 'update my entities\u002Fclasses from the database', 'reverse engineer and regenerate the classes', 'import the new tables and rebuild the entities', 'full DB sync', 'bring the model and classes up to date with the DB'. The distinguishing signal is scope: the user wants the whole project (mapping + names + Java code), not just one stage. For the *model\u002Fmapping only* (no name cleanup, no class generation) use `cayenne-db-import`; to (re)generate classes alone use `cayenne-cgen`; to clean names alone use `cayenne-model-naming`. Uses the `mcp__cayenne__dbimport_run` and `mcp__cayenne__cgen_run` MCP tools via the sub-skills; does NOT use Maven or Gradle goals.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7055,7056,7057],{"name":6975,"slug":6976,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},"2026-07-19T06:03:49.112969",{"slug":7060,"name":7060,"fn":7061,"description":7062,"org":7063,"tags":7064,"stars":7035,"repoUrl":7036,"updatedAt":7069},"cayenne-model-naming","clean up Cayenne object-layer names","Use this skill to clean up Object-layer names in a Cayenne DataMap — ObjEntity, ObjAttribute, and ObjRelationship names, plus DbRelationship names (the first-class unit of relationship cleanup — every FK has one whether or not an ObjRelationship was generated; the ObjRelationship name is synced to it when one exists) — so they read as descriptive, consistent Java. Trigger on phrases like 'clean up the model names', 'fix the entity names', 'these names look ugly', 'make the names descriptive', 'normalize the ObjEntity\u002Fattribute\u002Frelationship names', 'why is this relationship called team1', 'rename entities to be consistent', 'the import produced Gametype instead of GameType'. Invoke it on an explicit user request, or as a manual follow-up after a `cayenne-db-import` to polish the just-imported additions — it is never triggered automatically. IMPORTANT: this is a LIGHT polish pass — CayenneModeler's reverse-engineering already produces good names for the common case; only improve the specific things its deterministic algorithm cannot (run-together names with no separators like `gametype`, meaningless numbered names like `team1` from multiple relationships between two tables, and a common entity prefix that leaks into relationship names like `aaOrders`). Do NOT rewrite names that are already correct. This is Obj-layer naming polish; for structural model edits use `cayenne-modeling`, and for regenerating classes afterward use `cayenne-cgen`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7065,7066,7067,7068],{"name":7027,"slug":7028,"type":15},{"name":6975,"slug":6976,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},"2026-07-22T05:35:32.342548",{"slug":7071,"name":7071,"fn":7072,"description":7073,"org":7074,"tags":7075,"stars":7035,"repoUrl":7036,"updatedAt":7079},"cayenne-modeler","manage Cayenne projects with CayenneModeler","Use this skill when the user explicitly wants to open CayenneModeler (the GUI) on a Cayenne project, or when the modeling task is inherently visual — reverse engineering (delegated to cayenne-db-import), bulk relationship layout, multi-entity visual refactoring. Trigger on phrases like 'open the Modeler', 'open in CayenneModeler', 'launch the GUI', 'edit visually', 'show me the project in the Modeler'. Do NOT trigger as a fallback for ordinary a-la-carte XML edits — those belong in the cayenne-modeling skill, which is faster and doesn't require the user to context-switch.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7076,7077,7078],{"name":7027,"slug":7028,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},"2026-07-12T08:32:37.199428",{"slug":7081,"name":7081,"fn":7082,"description":7083,"org":7084,"tags":7085,"stars":7035,"repoUrl":7036,"updatedAt":7089},"cayenne-modeling","edit and extend Cayenne ORM models","Use this skill whenever the user wants to edit, inspect, or extend the Cayenne ORM model in a project — adding or modifying entities, attributes, relationships, embeddables, named queries, stored procedures, or DataNodes. Trigger on phrases like 'add an ObjEntity', 'add a DbEntity', 'add a relationship', 'expose this column as an attribute', 'create a new DataMap', 'add a named query', 'create an embeddable', 'add a stored procedure', 'change the attribute type', 'mark this column as nullable', 'rename this entity', or any mention of a Cayenne `*.map.xml` or `cayenne-*.xml` file. Also trigger when the user references modeling concepts (ObjEntity, DbEntity, ObjAttribute, DbAttribute, ObjRelationship, DbRelationship, Embeddable, dbEntityName, deleteRule, db-attribute-path, db-relationship-path, defaultPackage) in the context of a Cayenne-using app. This is the *primary* skill for a-la-carte ORM model manipulation — direct XML edits, not the Modeler GUI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7086,7087,7088],{"name":6975,"slug":6976,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},"2026-07-19T05:40:32.6889",{"slug":7091,"name":7091,"fn":7092,"description":7093,"org":7094,"tags":7095,"stars":7035,"repoUrl":7036,"updatedAt":7100},"cayenne-query","write and modify Cayenne database queries","Use this skill whenever the user wants to write or modify a Cayenne query — fetching entities by criteria, joining, prefetching to avoid N+1, ordering, paginating, aggregating, or running raw SQL through Cayenne. Trigger on phrases like 'query for X', 'fetch all artists where ...', 'write an ObjectSelect', 'use SQLSelect', 'use SelectById', 'add a prefetch', 'get distinct values', 'count rows', 'find by ID', 'load by primary key', 'build a Cayenne expression', 'why am I getting N+1', 'how do I paginate', 'select a single column', 'select columns into a DTO', 'named query in the DataMap'. Do NOT trigger for modeling changes (use cayenne-modeling) or runtime bootstrap (use cayenne-runtime).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[7096,7097,7098,7099],{"name":6975,"slug":6976,"type":15},{"name":7030,"slug":7031,"type":15},{"name":7033,"slug":7034,"type":15},{"name":20,"slug":21,"type":15},"2026-07-12T08:32:35.072322",108]