gh-140260: Fix data race in _struct module initialization with subinterpreters by ashm-dev · Pull Request #140909 · python/cpython

ashm-dev · 2025-11-02T12:29:49Z

Issue: ThreadSanitizer: data race in _struct module initialization with InterpreterPoolExecutor (free-threading build) #140260

Modules/_struct.c

ZeroIntensity

Please add a test stressing this.

Modules/_struct.c

Lib/test/test_struct.py

Modules/_struct.c

Lib/test/test_struct.py

Tools/c-analyzer/cpython/globals-to-fix.tsv

ZeroIntensity · 2025-11-05T13:46:37Z

I think the failing job will be solved by a rebase.

ZeroIntensity

LGTM

ZeroIntensity · 2025-11-05T13:52:35Z

Hm, that failure looks unrelated. @encukou, is it possible that #138190 broke something?

encukou · 2025-11-06T06:59:22Z

It did (now fixed thanks to Victor), but I don't think it broke mobile. Maybe it's sequential tests, or cross-compilation?

kumaraditya303 · 2025-11-11T20:23:38Z

Seems like Android and iOS do not support subinterpreters so the test should be skipped on those platforms.

ZeroIntensity · 2025-11-11T20:32:15Z

Good catch, I disabled auto-merge for the time being.

miss-islington-app · 2025-11-13T11:01:35Z

Thanks @ashm-dev for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…h subinterpreters (pythonGH-140909) (cherry picked from commit 63548b3) Co-authored-by: Shamil <ashm.tech@proton.me>

bedevere-app · 2025-11-13T11:01:44Z

GH-141501 is a backport of this pull request to the 3.14 branch.

bedevere-bot · 2025-11-13T11:23:38Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x (tier-1) has failed when building commit 63548b3.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/12669) and take a look at the build logs.
Check if the failure is related to this commit (63548b3) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/12669

Failed tests:

test.test_os.test_os

Failed subtests:

test_timerfd_interval - test.test_os.test_os.TimerfdTests.test_timerfd_interval

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_os/test_os.py", line 4017, in test_timerfd_interval
    self.assertEqual(self.read_count_signaled(fd), count)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 4 != 3

…th subinterpreters (GH-140909) (#141501) gh-140260: fix data race in `_struct` module initialization with subinterpreters (GH-140909) (cherry picked from commit 63548b3) Co-authored-by: Shamil <ashm.tech@proton.me>

* main: (463 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

* 'main' of github.com:python/cpython: (464 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

mhsmith · 2025-11-20T08:28:45Z

Seems like Android and iOS do not support subinterpreters so the test should be skipped on those platforms.

Based on discussions on Discord, it looks like this failure would happen on any platform if you run the tests with --single-process. So Android and iOS actually do support subintepreters, it's just that they're the only platforms that use --single-process in GitHub Actions.

…h subinterpreters (python#140909)

Fix data race in _struct module endian table initialization

243b5f0

bedevere-app bot added the awaiting review label Nov 2, 2025

bedevere-app bot mentioned this pull request Nov 2, 2025

ThreadSanitizer: data race in _struct module initialization with InterpreterPoolExecutor (free-threading build) #140260

Closed

fix doc

4c9892e

ashm-dev requested a review from ericsnowcurrently as a code owner November 2, 2025 12:58

refactor(c-analyzer): add endian_tables_init_once to globals-to-fix list

95958e8

ashm-dev force-pushed the tsan branch from 64f49c0 to 95958e8 Compare November 2, 2025 13:10

ZeroIntensity reviewed Nov 2, 2025

View reviewed changes

Modules/_struct.c Outdated Show resolved Hide resolved

ZeroIntensity added needs backport to 3.14 bugs and security fixes topic-subinterpreters labels Nov 2, 2025

ZeroIntensity reviewed Nov 2, 2025

View reviewed changes

Modules/_struct.c Outdated Show resolved Hide resolved

Use Py_UNUSED for unused arg in init_endian_tables

3130512

ashm-dev force-pushed the tsan branch from 68ffe6d to 3130512 Compare November 2, 2025 16:28

ashm-dev requested a review from ZeroIntensity November 2, 2025 16:47

ZeroIntensity reviewed Nov 2, 2025

View reviewed changes

picnixz reviewed Nov 2, 2025

View reviewed changes

Modules/_struct.c Outdated Show resolved Hide resolved

fix: remove duplicate include of pycore_lock.h in _struct.c

d33514b

ashm-dev requested a review from ZeroIntensity November 3, 2025 14:31

ZeroIntensity reviewed Nov 3, 2025

View reviewed changes

Lib/test/test_struct.py Outdated Show resolved Hide resolved

add test

c4f577f

ashm-dev force-pushed the tsan branch from 219b7df to c4f577f Compare November 3, 2025 16:13

ashm-dev requested a review from ZeroIntensity November 3, 2025 16:22

ZeroIntensity reviewed Nov 3, 2025

View reviewed changes

Lib/test/test_struct.py Outdated Show resolved Hide resolved

Lib/test/test_struct.py Outdated Show resolved Hide resolved

Lib/test/test_struct.py Outdated Show resolved Hide resolved

Modules/_struct.c Outdated Show resolved Hide resolved

Remove return check for endian table init

238c40e

ashm-dev force-pushed the tsan branch from e21838a to 238c40e Compare November 3, 2025 16:39

Use InterpreterPoolExecutor in struct tests

f4d24ff

ashm-dev requested review from ZeroIntensity and picnixz November 3, 2025 16:52

ZeroIntensity reviewed Nov 3, 2025

View reviewed changes

Lib/test/test_struct.py Outdated Show resolved Hide resolved

Lib/test/test_struct.py Show resolved Hide resolved

Lib/test/test_struct.py Outdated Show resolved Hide resolved

ZeroIntensity reviewed Nov 3, 2025

View reviewed changes

Tools/c-analyzer/cpython/globals-to-fix.tsv Show resolved Hide resolved

ZeroIntensity approved these changes Nov 5, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Nov 5, 2025

Merge branch 'main' into tsan

434dbab

kumaraditya303 approved these changes Nov 6, 2025

View reviewed changes

kumaraditya303 enabled auto-merge (squash) November 6, 2025 05:54

ZeroIntensity disabled auto-merge November 11, 2025 20:31

fix: tests

3844826

ashm-dev requested review from ZeroIntensity and kumaraditya303 November 13, 2025 09:29

Merge branch 'main' into tsan

b825107

kumaraditya303 merged commit 63548b3 into python:main Nov 13, 2025
46 checks passed

bedevere-app bot removed the awaiting merge label Nov 13, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Nov 13, 2025

ashm-dev deleted the tsan branch November 13, 2025 11:03

StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025

pythongh-140260: fix data race in _struct module initialization wit…

1ce571b

…h subinterpreters (python#140909)

Uh oh!

Conversation

ashm-dev commented Nov 2, 2025 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity commented Nov 5, 2025

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity commented Nov 5, 2025

Uh oh!

encukou commented Nov 6, 2025

Uh oh!

kumaraditya303 commented Nov 11, 2025

Uh oh!

ZeroIntensity commented Nov 11, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented Nov 13, 2025

Uh oh!

bedevere-app bot commented Nov 13, 2025

Uh oh!

bedevere-bot commented Nov 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

mhsmith commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ashm-dev commented Nov 2, 2025 •

edited by bedevere-app bot

Loading