data engineering

Python 3.15 Made UTF-8 the Default on Windows, and 82% of Characters Come Back Wrong in 3.14 With No Error

Python 3.15 changed what open() does on Windows when no encoding is named, from the cp1252 code page to UTF-8. Running both versions on Windows 11, none of the 159,706 assigned non-ASCII Unicode characters written by Python 3.15 came back right when Python 3.14 read it. 82% came back wrong without any error, and the rest raised. In the other direction Python 3.15 raised on every character Python 3.14 could write. R moved to UTF-8 on Windows in 2022, so every R to Python handoff tested that broke under 3.14 works under 3.15, and pandas never changed at all.

Polars 2.0 Row Order: The Same Query Returned a Different Answer on Every Run

Polars 2.0 runs lazy queries on the streaming engine by default, and the streaming engine does not guarantee row order. Running five order-dependent queries 20 times each on the same file, Polars 1.44.2 matched the explicit-order answer every time, while Polars 2.0.0 and 2.1.0 got four of the five wrong on every run, with a different answer each time and only a handful of customers affected. Keeping order in the join, or sorting inside the expression, fixed every run. Switching back to the in-memory engine did not fully restore the old answers, and DuckDB in R behaves the same way without an ORDER BY.