GH-36411: [Python] Use scikit-build-core as build backend for PyArrow and get rid of setup.py

[raulcd]

Rationale for this change

Move our PyArrow build backend from setuptools and a custom setup.py to scikit-build-core which is just build backend for CMake related projects.

What changes are included in this PR?

Move from setuptools to scikit-build-core and remove PyArrow setup.py. Update some of the build requirements and minor fixes.
A custom build backend has been also been created in order to wrap scikit-build-core in order to fix problems on License files for monorepos.
pyproject.toml metadata validation expects license files to exist before exercising the build backend that's why we create symlinks. Our thin build backend will just make those symlinks hard-links in order for license and notice files to contain the contents and be added as part of the sdist.

Remove flags that are not used anymore (were only part of setup.py) and documented and validated how the same flags have to be used now.

Are these changes tested?

Yes all Python CI tests, wheels and sdist are successful.

Are there any user-facing changes?

Yes, users building PyArrow will now require the new build dependencies to exercise the build and depending on the flags used they might require to use the new documented way of using those flags.

• GitHub Issue: [Python] Improve the pyarrow build system #36411

[raulcd]

@github-actions crossbow submit -g python

[raulcd]

@github-actions crossbow submit wheel-*-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 166ff63

Submitted crossbow builds: ursacomputing/crossbow @ actions-f1bbdf4eaa

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 36fefd4

Submitted crossbow builds: ursacomputing/crossbow @ actions-bd811d95ea

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit -g python -g wheel

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: c518c90

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea249e92ce

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 131e2c0

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c58e78aff

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit python-sdist

[github-actions]

Revision: 30e04be

Submitted crossbow builds: ursacomputing/crossbow @ actions-3179f97956

Task	Status
python-sdist

[raulcd]

@github-actions crossbow submit -g python -g wheel

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 5606749

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c4293e220

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@jorisvandenbossche this seems to happen only on conda environments and is due to scikit-build-core ignoring the default CMAKE_ARGS that conda sets (see -DCMAKE_BUILD_TYPE=Release there), this is directly from the container:

$ PANDAS=nightly NUMPY=nightly PYTHON=3.13 docker compose run -it --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=23.0.0.dev328 conda-python-pandas /bin/bash
root@6ac69eff4ef5:/# echo $CMAKE_ARGS
-DCMAKE_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-strip -DCMAKE_BUILD_TYPE=Release
root@6ac69eff4ef5:/# 

I am unsure why conda defines a default -DCMAKE_BUILD_TYPE=Release on CMAKE_ARGS, maybe @h-vetinari can help here.
In any case for our specific case, this is ok as we are just defining it ourselves on the pyproject.toml or via -C cmake.build-type=Debug and working as expected.

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: d5a61ab

Submitted crossbow builds: ursacomputing/crossbow @ actions-c69c0863fa

Task	Status
wheel-windows-cp313-cp313-amd64

[raulcd]

@github-actions crossbow submit wheel-windows-cp313-cp313-amd64

[github-actions]

Revision: 9c6cf8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-b68682bb32

Task	Status
wheel-windows-cp313-cp313-amd64