LoginSign Up
← Back to all sparks
P

Polars

ANALYTICS
Velocity5.0

High-performance DataFrame library for analytics

pola.rs ↗

A deprecation sweep and hive-partition join rewrites, shipped on two trains at once.

dataframesquery-optimizationdeprecationscloud-ioiceberg
◆Current state
Polars releases Python and Rust builds in lockstep, with each Rust tag naming the Python version its DSL matches. The recent work is concentrated in two places: query-plan performance — len() pushdown into concat and union inputs, pre-partitioning on hive-partitioned joins, split multiplexers scanning in-memory DataFrames — and cloud IO, where an adaptive HTTP rate-limiter and a global DNS cache landed. Correctness fixes reach into unsoundness in rayon block_on and undefined behaviour on empty chunks.
◆Where it's heading
The 1.43.0 release carried seven deprecations at once — numeric-to-categorical casts, casts from non-nested dtypes into lists, bitwise ops between integers and booleans, LazyFrame.profile, unnamed list.to_struct calls — and 1.43.2 added more. That density of deprecation in minor releases is how a project narrows its type semantics before a major. Alongside it, Iceberg and Delta support keeps taking fixes, which is where the lakehouse-format work is showing up.
◆Prediction
Expect the deprecation cycle to keep tightening casting and categorical semantics, with performance work staying focused on hive-partitioned and cloud-hosted data where the query planner has the most left to exploit.

◆Recent moves

  1. 13d ago

    Rust 0.55.2 adds an adaptive HTTP rate-limiter for cloud IO

    Cloud reads gain an adaptive HTTP rate-limiter, which matters when a scan fans out into enough concurrent requests to get throttled by the object store. An unsoundness in rayon block_on is also fixed.

    View source ↗
  2. 14d ago

    Rust 0.55.1 rewrites joins on hive-partitioned data

    The densest performance release in the window: inner joins on hive-partitioned data are rewritten, left, right and semi joins pre-partition, len() pushes down into concat and union inputs, and the DNS cache becomes global. Aimed squarely at partitioned data in object storage.

    View source ↗
  3. 18d ago

    Python 1.43.2 deprecates Categorical-to-integer casts

    More narrowing of categorical semantics, plus an infer_schema_files parameter for scan_csv and fixes to enum metadata propagation for Iceberg column mapping and parquet field IDs. The deprecation thread from 1.43.0 continues.

    View source ↗
  4. 22d ago

    Python 1.43.1 allows callback sinks on cloud targets

    Callback sinks work against cloud destinations, and a cluster of correctness fixes lands — panics on self-joins of scan_delta and scan_iceberg, undefined behaviour on empty chunks, and SQL NOT IN three-valued-logic behaviour with NULLs in joins.

    View source ↗
  5. 29d ago

    Python 1.43.0 lands seven deprecations in one release

    The most consequential release here for existing code: numeric-to-categorical casts, cat.get_categories, cat.to_local, LazyFrame.profile, non-nested-to-list casts, integer/boolean bitwise ops and unnamed list.to_struct calls are all deprecated together. It reads as clearing the ground ahead of a major.

    View source ↗
  6. 1mo ago

    Python 1.42.1 samples multi-file parquet metadata resolution

    A sampled resolve mode for multi-file parquet metadata cuts the cost of planning over large file sets, and path expansion stops blocking. pl.concat's strict parameter is deprecated in favour of an explicit horizontal_extend mode.

    View source ↗