Python 3.15 Made UTF-8 the Default on Windows, and 82% of Characters Come Back Wrong in 3.14 With No Error

The error was luck

The three names that raised have one thing in common. Their UTF-8 bytes include one of the five byte values that cp1252 leaves undefined. Every other byte maps to some character, so cp1252 reads almost anything without complaint.

undefined = []
for b in range(256):
    try:
        bytes([b]).decode("cp1252")
    except UnicodeDecodeError:
        undefined.append(hex(b))
print(undefined)

for ch in ["Á", "á", "”", "’"]:
    b = ch.encode("utf-8")
    try:
        shown = b.decode("cp1252")
    except UnicodeDecodeError:
        shown = "raises"
    print(ch, b.hex(" "), "->", shown)
## ['0x81', '0x8d', '0x8f', '0x90', '0x9d']
## Á c3 81 -> raises
## á c3 a1 -> á
## ” e2 80 9d -> raises
## ’ e2 80 99 -> ’

Capital Á ends in 0x81 and raises. Lowercase á becomes á and does not. The closing curly quote ends in 0x9D and raises, while the apostrophe becomes ’. So whether Python 3.14 warns you depends on which letters happen to be in the file.

To see how often the error comes, I sent every assigned non-ASCII character in Unicode 17.0.0, all 159,706 of them, through Python 3.15’s default writer and Python 3.14’s default reader. Not one came back right. 82% came back wrong without an error, and the other 18% raised.

The other direction is the mirror image. Python 3.14 can write only 123 of those characters, the non-ASCII part of cp1252, and raises UnicodeEncodeError on the rest. Python 3.15 raised UnicodeDecodeError on all 123 of the ones it could write. A cp1252 byte next to plain ASCII is never valid UTF-8, so the only escape is two or more non-ASCII characters in a row that happen to spell a UTF-8 sequence, such as à followed by ©, which reads as é.

Old files fail loudly in new Python. New files fail quietly in old Python.

Where R and Python meet

R has written UTF-8 on Windows since 4.2.0, so most of what 3.15 changes is Python catching up with R. I ran the common handoffs both ways, with the same names, under both versions.

Python 3.15 fixed every handoff with R. Three rows are worth a second look.

pandas did not change. read_csv() has always assumed UTF-8, whatever the locale, so it read R’s file correctly under both versions and has been raising on cp1252 files for years.

The subprocess failure hides. On Windows, subprocess.run() decodes the child’s output in a background thread. When the decode fails, that thread prints a traceback, run() returns normally, and stdout is None (line 1682 of subprocess.py in 3.14.10). check=True does not catch it, because R exited cleanly. The same line is still in 3.15.2, which works here only because its decode succeeds.

3.15 does not fix the BOM. Files saved with the “UTF-8 with BOM” option start with three marker bytes. Plain UTF-8 keeps them as an invisible character on the first column name, so row["name"] raises KeyError under both versions. Naming the encoding fixes this row and the cp1252 one:

for path, enc in [("bom.csv", "utf-8"), ("bom.csv", "utf-8-sig"), ("ansi.csv", "cp1252")]:
    with open(path, newline="", encoding=enc) as f:
        header, first = f.readline(), f.readline()
    print(f"{path:9} {enc:10}", repr(header.split(",")[0]), first.split(",")[0])
## bom.csv   utf-8      '\ufeffname' Zoë O’Brien
## bom.csv   utf-8-sig  'name' Zoë O’Brien
## ansi.csv  cp1252     'name' Zoë O’Brien

What to change

Name the encoding everywhere. open(path, encoding="utf-8") and subprocess.run(..., encoding="utf-8") behave the same on every version and every machine. Python 3.10 and later can flag each call that leaves it out, as the call runs:

with open("person_1.csv") as f:
    f.read()
with open("person_1.csv", encoding="utf-8") as f:
    f.read()
## warn.py:1: EncodingWarning: 'encoding' argument not specified
##   with open("person_1.csv") as f:

Only the first open() is flagged. If you share files with someone still on 3.14, the switch that gives 3.14 the new behavior has existed since 3.7. PYTHONUTF8=1 in the environment, or -X utf8 on the command line:

## Nguyễn Thị Minh Khai
## Zoë O’Brien
## Łukasz Wójcik
## Müller
## 東京
## José Álvarez

That is Python 3.14 reading the same six files with -X utf8.

What this does not show

• One Windows 11 machine with cp1252 as its ANSI code page. Machines set to Japanese, Chinese, or other code pages fail in different ways, and I did not test them.
• I wrote the cp1252 and BOM files as bytes rather than saving them from Excel or Notepad.
• The large table counts single characters. A file raises if any one character raises, so long Chinese or Korean text raises more often than its row suggests, and a short Western name list raises less.

What to do on Monday

1. Search for open( without encoding=, or run your tests once with -X warn_default_encoding.
2. Pass encoding="utf-8" to open(), Path.read_text(), and subprocess.run().
3. Read files that may start with a BOM with encoding="utf-8-sig".
4. Read old ANSI exports with encoding="cp1252", on purpose.
5. Until everyone is on 3.15, set PYTHONUTF8=1 on the machines that are not.

R code on Windows needs no change for 3.15, because R got here first. Files from colleagues still on 3.14 are the exception: read those with read.csv(fileEncoding = "cp1252"). How CSV, Parquet, and other formats carry text and types between tools is covered in the data formats chapter of Advanced Data Analysis. The audit script, encoding_audit.py, sits next to this post.