gh-135906: Test the internal C API in test_cext

[vstinner]

• Issue: Use _PyObject_CAST in internal headers #135906

[vstinner]

I tried to check for C++ compiler warnings in internal Python C headers, but I got many C++ compiler errors in mimalloc and I don't know how to handle them. Examples:

In file included from /usr/include/features.h:524,
                 from /usr/include/assert.h:35,
                 from /home/vstinner/python/main/Include/Python.h:19,
                 from extension.cpp:13:
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:73:33: error: expected unqualified-id before 'noexcept'
   73 | #define mi_decl_throw           __THROW
      |                                 ^~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:611:31: note: in expansion of macro 'mi_decl_throw'
  611 | mi_decl_noreturn mi_decl_cold mi_decl_throw
      |                               ^~~~~~~~~~~~~

and

In file included from /home/vstinner/python/main/Include/internal/pycore_mimalloc.h:44,
                 from /home/vstinner/python/main/Include/internal/pycore_tstate.h:13,
                 from /home/vstinner/python/main/Include/internal/pycore_pystate.h:13,
                 from extension.cpp:20:
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/internal.h: In function 'bool _mi_is_aligned(void*, size_t)':
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:613:49: error: '_mi_assert_fail' was not declared in this scope; did you mean '__assert_fail'?
  613 | #define mi_assert(expr)     ((expr) ? (void)0 : _mi_assert_fail(#expr,__FILE__,__LINE__,__func__))
      |                                                 ^~~~~~~~~~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/types.h:619:31: note: in expansion of macro 'mi_assert'
  619 | #define mi_assert_internal    mi_assert
      |                               ^~~~~~~~~
/home/vstinner/python/main/Include/internal/mimalloc/mimalloc/internal.h:273:3: note: in expansion of macro 'mi_assert_internal'
  273 |   mi_assert_internal(alignment != 0);
      |   ^~~~~~~~~~~~~~~~~~

[vstinner]

I updated test_cppext to have a bare minimum test of the internal C API for now: test including pycore_frame.h. We can extend the test later.

[vstinner]

Note to mysel: GitHub Action skips these tests which require the cpu test resource (-u cpu or -u all). Only buildbots run these slow tests.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 69588da 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

test_cext fails on on multiple buildbots (s390x Fedora Stable Clang PR, AMD64 FreeBSD PR, x86-64 macOS PR, ...) with:

  In file included from extension.c:21:
  In file included from /home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/internal/pycore_gc.h:11:
  /home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/internal/pycore_interp_structs.h:132:16: error: redefinition of typedef 'atexit_datacallbackfunc' is a C11 feature [-Werror,-Wtypedef-redefinition]
    132 | typedef void (*atexit_datacallbackfunc)(void *);
        |                ^
  /home/buildbot/buildarea/pull_request.cstratak-fedora-stable-x86_64.clang/build/Include/cpython/pylifecycle.h:87:16: note: previous definition is here
     87 | typedef void (*atexit_datacallbackfunc)(void *);
        |                ^

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 49faa0c 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

test_cext fails on ARM64 Windows Non-Debug PR:

  "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\ARM64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\ARM64 -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\build\test_python_28188�\tempcwd\env\include -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC -IC:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\ATLMFC\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\cppwinrt" /Tcextension.c /Fobuild\temp.win-arm64-cpython-315\Release\extension.obj /W4 /WX -DMODULE_NAME=_test_cext -DTEST_INTERNAL_C_API=1
  cl : Command line warning D9025 : overriding '/W3' with '/W4'
  extension.c
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(509): error C2220: the following warning is treated as an error
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(509): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(512): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(513): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\mimalloc/mimalloc/types.h(518): warning C4324: 'mi_abandoned_pool_s': structure was padded due to alignment specifier
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_qsbr.h(75): warning C4200: nonstandard extension used: zero-sized array in struct/union
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_backoff.h(68): warning C4244: 'function': conversion from 'int' to 'uint16_t', possible loss of data
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_code.h(114): warning C4201: nonstandard extension used: nameless struct/union
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_object_deferred.h(20): warning C4100: 'op': unreferenced formal parameter
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_stackref.h(792): warning C4100: 'tstate': unreferenced formal parameter
  C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\include\internal\pycore_stackref.h(803): warning C4100: 'tstate': unreferenced formal parameter
  error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2
  error: subprocess-exited-with-error

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 7870668 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[vstinner]

ARM64 Windows Non-Debug PR failed. .TestInteralCAPI.test_build() failed with:

  extension.obj : error LNK2001: unresolved external symbol PyLong_FromLong
  extension.obj : error LNK2001: unresolved external symbol PyModuleDef_Init
  extension.obj : error LNK2001: unresolved external symbol PyArg_ParseTuple
  build\lib.win-arm64-cpython-315\_test_cext.cp315-win_arm64.pyd : fatal error LNK1120: 3 unresolved externals

and TestInteralCAPI.test_build_c11() fails with:

  extension.c
  C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): error C2220: the following warning is treated as an error
  C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): warning C5105: macro expansion producing 'defined' has undefined behavior
  error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2

[vstinner]

!buildbot Windows

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 30c6265 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

The command will test the builders whose names match following regular expression: Windows

The builders matched are:

• AMD64 Windows PGO NoGIL PR
• AMD64 Windows Server 2022 NoGIL PR
• AMD64 Windows10 PR
• ARM64 Windows Non-Debug PR
• AMD64 Windows11 Refleaks PR
• AMD64 Windows11 Non-Debug PR
• AMD64 Windows11 Bigmem PR
• AMD64 Windows PGO PR
• ARM64 Windows PR

[vstinner]

On ARM64 Windows Non-Debug PR, TestInteralCAPI.test_build_c11() failed with:

  extension.c
  C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): error C2220: the following warning is treated as an error
  C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\um\winbase.h(9531): warning C5105: macro expansion producing 'defined' has undefined behavior
  error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.31.31103\\bin\\HostX86\\ARM64\\cl.exe' failed with exit code 2

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @vstinner for commit 9d4d697 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F136247%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.