GH-29847: [C++] Build with Azure SDK for C++

[Tom-Newton]

Rationale for this change

We want to use the Azure SDK for C++ to read/write to Azure blob storage. Obviously this is pretty important for building an AzureFileSystem.

What changes are included in this PR?

Builds the the relevant parts of the azure SDK as a cmake external project. Adds a couple of simple tests that just assert that the Azure SDK is working and a couple of lines in AzureFileSystem to initialise the blob storage client to ensure the build is working correctly in all environments.

I started with the build setup from #12914 but I did make few changes.

1. Although its atypical for this project we chose to switch from cmake's ExternalProject to FetchContent. FetchContent is recomended by the Azure docs https://github.com/Azure/azure-sdk-for-cpp#cmake-project--fetch-content. It also solves a few problems including: automatically linking system curl and ssl instead of bootstrapping vcpkg and installing curl and ssl from there.
2. Only build one version of the Azure SDK for C++ because it contains all the components. Previously we were unnecessarily building 5 different versions of the whole thing on top of each other. This created race conditions for which version each component came from.
3. We are using azure-core_1.10.2 which is a very recent version. There are a couple of important reasons for this 1. an important managed identity fix, 2. fixed support for curl versions < 7.71.0.

There will be follow up PRs to enable Azure in the manylinux builds. We need to update vcpkg first so we can get a version of the Azure SDK which contains an important managed identity fix.

Are these changes tested?

Yes. There is a simple test that just runs the Azure client against azurite. Additionally just initialising the client in AzureFileSystem goes a long way towards ensuring the build is working.

Are there any user-facing changes?

No

• Closes: [C++] Add azure-sdk-for-cpp to ThirdpartyToolchain #29847

[github-actions]

⚠️ GitHub issue #29847 has been automatically assigned in GitHub to PR creator.

[pitrou]

I'm trying to build with gcc 12.3.0 and I get the following error:

In file included from /build/build-test/_deps/azure_sdk-src/sdk/identity/azure-identity/src/environment_credential.cpp:6:
/build/build-test/_deps/azure_sdk-src/sdk/identity/azure-identity/inc/azure/identity/client_certificate_credential.hpp:68:9: error: 'Azure::Identity::ClientCertificateCredential' declared with greater visibility than the type of its field 'Azure::Identity::ClientCertificateCredential::m_pkey' [-Werror=attributes]
   68 |   class ClientCertificateCredential final : public Core::Credentials::TokenCredential {
      |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1plus: all warnings being treated as errors

It seems like we're passing -Wall -Werror when building bundled dependencies? It makes us heavily dependent on maintenance policies of third-party projects.

[pitrou]

We normally use the EP_CXX_FLAGS cmake variable when compiling bundled dependencies, but it seems that isn't forwarded by the FetchContent-based directives?

[assignUser]

It seems like we're passing -Wall -Werror when building bundled dependencies? It makes us heavily dependent on maintenance policies of third-party projects.

I agree dependencies should not be build with -Werror etc. This is an issue with the use of the flag variables directly. As we can now use cmake 3.16 we should move to target based properties vs global flags but that is of course a major refactor...

@Tom-Newton the use of external project within arrow is 'historic' as we just recently increased our minimum cmake version enough to make use of fc. Eventually it would be great to move everything to fc so adding new deps with fc instead of ep is in my eyes encouraged!

[assignUser]

We normally use the EP_CXX_FLAGS cmake variable when compiling bundled dependencies, but it seems that isn't forwarded by the FetchContent-based directives?

Contrary to external project things added via fetchcontent are configured with the parent project and inherit variables and flags as if using add_subdirectory. So it likely is not necessary but I haven't looked at our flags script recently so 🤷

[Tom-Newton]

I agree dependencies should not be build with -Werror etc. This is an issue with the use of the flag variables directly. As we can now use cmake 3.16 we should move to target based properties vs global flags but that is of course a major refactor...

So what do we think is the best option today? Is disabling -Werror at a global level an acceptable solution? Being able to disable -Werror would also be helpful for supporting Ubuntu 20 #36835 (comment)

[kou]

I'll provide a patch for -Werror later. Please wait for a few days...

[kou]

This is an ad-hoc patch but this will work. We need a real improvement later.

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 1dfaf71b4..9ff91f978 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -5082,6 +5082,13 @@ function(build_azure_sdk)
   set(CMAKE_EXPORT_NO_PACKAGE_REGISTRY TRUE)
   set(DISABLE_AZURE_CORE_OPENTELEMETRY TRUE)
   set(ENV{AZURE_SDK_DISABLE_AUTO_VCPKG} TRUE)
+  if(MSVC)
+    string(REPLACE "/WX" "" CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG}")
+    string(REPLACE "/WX" "" CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG}")
+  else()
+    string(REPLACE "-Werror" "" CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG}")
+    string(REPLACE "-Werror" "" CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG}")
+  endif()
   fetchcontent_makeavailable(azure_sdk)
   set(AZURE_SDK_VENDORED
       TRUE

[kou]

@github-actions crossbow submit -g cpp

[github-actions]

Revision: e3fb84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-ea66ccd89b

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-11-cpp-amd64
test-debian-11-cpp-i386
test-fedora-35-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-no-threading

[kou]

+1

I'll merge this tomorrow if nobody objects it.

[Tom-Newton]

Thanks for your help on this @kou. I feel kind of bad about how much work I created for you during review, with my lack of C++ experience. Hopefully the native Azure support is worth it 🙂.

[felipecrv]

Thank you for this PR @Tom-Newton!

[kou]

Don't worry. :-)
I'm happy that we have more contributors like you!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 3cb4481.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.