Python 3.15 Made UTF-8 the Default on Windows, and 82% of Characters Come Back Wrong in 3.14 With No Error

Tue, 10 Nov 2026 00:00:00 +0000

Python 3.15 came out on 9 October, and on Windows it changes what open() does when you do not name an encoding. Python 3.14 and earlier use the ANSI code page, which is cp1252 on a US or Western European machine. Python 3.15 uses UTF-8 on every platform (PEP 686). R made the same move in R 4.2.0, in April 2022 (Kalibera). So I wrote six names from R on Windows 11, one per CSV file, and read them back with Python’s csv module and no encoding argument, under 3.14.10 and 3.15.2, the current release of each.

people <- c("Nguyễn Thị Minh Khai", "Zoë O’Brien", "Łukasz Wójcik",
"Müller", "東京", "José Álvarez")
for (i in seq_along(people)) {
write.csv(data.frame(id = i, name = people[i]), sprintf("person_%d.csv", i),
row.names = FALSE)
}

Under Python 3.14:

import csv
for i in range(1, 7):
try:
with open(f"person_{i}.csv", newline="") as f: # no encoding argument
print([row["name"] for row in csv.DictReader(f)][0])
except UnicodeDecodeError as e:
print("UnicodeDecodeError on byte", hex(e.object[e.start]))
## Nguyá»…n Thá»‹ Minh Khai
## ZoÃ« Oâ€™Brien
## UnicodeDecodeError on byte 0x81
## MÃ¼ller
## UnicodeDecodeError on byte 0x9d
## UnicodeDecodeError on byte 0x81

The same code under Python 3.15:

## Nguyễn Thị Minh Khai
## Zoë O’Brien
## Łukasz Wójcik
## Müller
## 東京
## José Álvarez

Python 3.14 returned 3 names wrong and raised on the other 3. Nguyễn, the most common Vietnamese surname and mine, came back as Nguyá»…n. Python 3.15 read all 6 correctly.

The error was luck

The three names that raised have one thing in common. Their UTF-8 bytes include one of the five byte values that cp1252 leaves undefined. Every other byte maps to some character, so cp1252 reads almost anything without complaint.

undefined = []
for b in range(256):
try:
bytes([b]).decode("cp1252")
except UnicodeDecodeError:
undefined.append(hex(b))
print(undefined)
for ch in ["Á", "á", "”", "’"]:
b = ch.encode("utf-8")
try:
shown = b.decode("cp1252")
except UnicodeDecodeError:
shown = "raises"
print(ch, b.hex(" "), "->", shown)
## ['0x81', '0x8d', '0x8f', '0x90', '0x9d']
## Á c3 81 -> raises
## á c3 a1 -> Ã¡
## ” e2 80 9d -> raises
## ’ e2 80 99 -> â€™

Capital Á ends in 0x81 and raises. Lowercase á becomes Ã¡ and does not. The closing curly quote ends in 0x9D and raises, while the apostrophe becomes â€™. So whether Python 3.14 warns you depends on which letters happen to be in the file.

To see how often the error comes, I sent every assigned non-ASCII character in Unicode 17.0.0, all 159,706 of them, through Python 3.15’s default writer and Python 3.14’s default reader. Not one came back right. 82% came back wrong without an error, and the other 18% raised.

Characters	Count	Wrong, no error	Raises
Latin-1 Supplement (é, ü, ñ)	96	95%	5%
Latin Extended-A (Ł, ő, š)	128	92%	8%
Vietnamese (Latin Extended Additional)	256	92%	8%
Greek	135	94%	6%
Cyrillic	256	92%	8%
Arabic	256	92%	8%
General Punctuation (curly quotes, dashes)	111	53%	47%
CJK Unified Ideographs	20,992	85%	15%
Hangul Syllables	11,172	84%	16%
Emoji (U+1F300 to U+1FAFF)	1,895	80%	20%

The other direction is the mirror image. Python 3.14 can write only 123 of those characters, the non-ASCII part of cp1252, and raises UnicodeEncodeError on the rest. Python 3.15 raised UnicodeDecodeError on all 123 of the ones it could write. A cp1252 byte next to plain ASCII is never valid UTF-8, so the only escape is two or more non-ASCII characters in a row that happen to spell a UTF-8 sequence, such as Ã followed by ©, which reads as é.

Old files fail loudly in new Python. New files fail quietly in old Python.

Where R and Python meet

R has written UTF-8 on Windows since 4.2.0, so most of what 3.15 changes is Python catching up with R. I ran the common handoffs both ways, with the same names, under both versions.

Path	Names	Python 3.14.10	Python 3.15.2
R `write.csv()` to Python `csv` module	all six	UnicodeDecodeError	right
R `write.csv()` to Python `csv` module	three cp1252 can hold	wrong, no error	right
Python 3.15 `csv.writer` to Python `csv` module	three cp1252 can hold	wrong, no error	right
R `write.csv()` to `pandas.read_csv()`	all six	right	right
Python `csv.writer` to R `read.csv()`	all six	UnicodeEncodeError, file holds 0 of 6 rows	right
Python `csv.writer` to R `read.csv()`	three cp1252 can hold	wrong, no error	right
Python `print()` to R `system2()`	all six	UnicodeEncodeError	right
R `writeLines()` to Python `subprocess.run(text=True)`	all six	stdout is None	right
cp1252 file to Python `csv` module	three cp1252 can hold	right	UnicodeDecodeError
cp1252 file to `pandas.read_csv()`	three cp1252 can hold	UnicodeDecodeError	UnicodeDecodeError
UTF-8 file with a BOM to `csv.DictReader`	three cp1252 can hold	KeyError	KeyError

Python 3.15 fixed every handoff with R. Three rows are worth a second look.

pandas did not change. read_csv() has always assumed UTF-8, whatever the locale, so it read R’s file correctly under both versions and has been raising on cp1252 files for years.

The subprocess failure hides. On Windows, subprocess.run() decodes the child’s output in a background thread. When the decode fails, that thread prints a traceback, run() returns normally, and stdout is None (line 1682 of subprocess.py in 3.14.10). check=True does not catch it, because R exited cleanly. The same line is still in 3.15.2, which works here only because its decode succeeds.

3.15 does not fix the BOM. Files saved with the “UTF-8 with BOM” option start with three marker bytes. Plain UTF-8 keeps them as an invisible character on the first column name, so row["name"] raises KeyError under both versions. Naming the encoding fixes this row and the cp1252 one:

for path, enc in [("bom.csv", "utf-8"), ("bom.csv", "utf-8-sig"), ("ansi.csv", "cp1252")]:
with open(path, newline="", encoding=enc) as f:
header, first = f.readline(), f.readline()
print(f"{path:9} {enc:10}", repr(header.split(",")[0]), first.split(",")[0])
## bom.csv utf-8 '\ufeffname' Zoë O’Brien
## bom.csv utf-8-sig 'name' Zoë O’Brien
## ansi.csv cp1252 'name' Zoë O’Brien

What to change

Name the encoding everywhere. open(path, encoding="utf-8") and subprocess.run(..., encoding="utf-8") behave the same on every version and every machine. Python 3.10 and later can flag each call that leaves it out, as the call runs:

with open("person_1.csv") as f:
f.read()
with open("person_1.csv", encoding="utf-8") as f:
f.read()
## warn.py:1: EncodingWarning: 'encoding' argument not specified
## with open("person_1.csv") as f:

Only the first open() is flagged. If you share files with someone still on 3.14, the switch that gives 3.14 the new behavior has existed since 3.7. PYTHONUTF8=1 in the environment, or -X utf8 on the command line:

## Nguyễn Thị Minh Khai
## Zoë O’Brien
## Łukasz Wójcik
## Müller
## 東京
## José Álvarez

That is Python 3.14 reading the same six files with -X utf8.

What this does not show

One Windows 11 machine with cp1252 as its ANSI code page. Machines set to Japanese, Chinese, or other code pages fail in different ways, and I did not test them.
I wrote the cp1252 and BOM files as bytes rather than saving them from Excel or Notepad.
The large table counts single characters. A file raises if any one character raises, so long Chinese or Korean text raises more often than its row suggests, and a short Western name list raises less.

What to do on Monday

Search for open( without encoding=, or run your tests once with -X warn_default_encoding.
Pass encoding="utf-8" to open(), Path.read_text(), and subprocess.run().
Read files that may start with a BOM with encoding="utf-8-sig".
Read old ANSI exports with encoding="cp1252", on purpose.
Until everyone is on 3.15, set PYTHONUTF8=1 on the machines that are not.