Skip to content

ARROW-12630: [Dev][Integration] conda-integration docker build fails #10225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kszucs wants to merge 4 commits into from
Closed

ARROW-12630: [Dev][Integration] conda-integration docker build fails #10225

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
06f4860
Disable the rust integration tests by default in the docker-compose s…
kszucs May 3, 2021
d4fb6da
Doesn't know the path outside of the container
kszucs May 3, 2021
574cd6e
Explicitly enable the implementations
kszucs May 3, 2021
8cad727
Pass true flags from the CLI
kszucs May 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/integration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ on:
- 'cpp/**'
- 'java/**'
- 'format/**'
- 'rust/**'
pull_request:
paths:
- '.github/workflows/integration.yml'
Expand All @@ -41,7 +40,6 @@ on:
- 'cpp/**'
- 'java/**'
- 'format/**'
- 'rust/**'

env:
DOCKER_VOLUME_PREFIX: ".docker/"
Expand Down Expand Up @@ -82,7 +80,7 @@ jobs:
- name: Setup Archery
run: pip install -e dev/archery[docker]
- name: Execute Docker Build
run: archery docker run conda-integration
run: archery docker run -e ARCHERY_INTEGRATION_WITH_RUST=1 conda-integration
Copy link
Member Author

@kszucs kszucs May 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorgecarleitao @andygrove @alamb you'll need to update the integration workflow in arrow-rs once we merge this

Copy link
Member

@jorgecarleitao jorgecarleitao May 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fielded apache/arrow-rs#253

- name: Docker Push
if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
continue-on-error: true
Expand Down
11 changes: 7 additions & 4 deletions ci/scripts/integration_arrow.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,17 @@
set -ex

arrow_dir=${1}
source_dir=${1}/cpp
build_dir=${2}/cpp

gold_dir=$arrow_dir/testing/data/arrow-ipc-stream/integration

pip install -e $arrow_dir/dev/archery

archery integration --with-all --run-flight \
# Rust can be enabled by exporting ARCHERY_INTEGRATION_WITH_RUST=1
archery integration \
--run-flight \
--with-cpp=1 \
--with-java=1 \
--with-js=1 \
--with-go=1 \
--gold-dirs=$gold_dir/0.14.1 \
--gold-dirs=$gold_dir/0.17.1 \
--gold-dirs=$gold_dir/1.0.0-bigendian \
Expand Down
28 changes: 21 additions & 7 deletions ci/scripts/rust_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,20 +17,37 @@
# specific language governing permissions and limitations
# under the License.

set -ex
set -e

arrow_dir=${1}
source_dir=${1}/rust

# This file is used to build the rust binaries needed for the
# archery integration tests. Testing of the rust implementation
# in normal CI is handled by github workflows
# This file is used to build the rust binaries needed for the archery
# integration tests. Testing of the rust implementation in normal CI is handled
# by github workflows in the arrow-rs repository.

# Disable full debug symbol generation to speed up CI build / reduce memory required
export RUSTFLAGS="-C debuginfo=1"

export ARROW_TEST_DATA=${arrow_dir}/testing/data
export PARQUET_TEST_DATA=${arrow_dir}/cpp/submodules/parquet-testing/data

if [ "${ARCHERY_INTEGRATION_WITH_RUST}" -eq "0" ]; then
echo "====================================================================="
echo "Not building the Rust implementation."
echo "====================================================================="
exit 0;
elif [ ! -d "${source_dir}" ]; then
echo "====================================================================="
echo "The Rust source is missing. Please clone the arrow-rs repository"
echo "to arrow/rust before running the integration tests:"
echo " git clone https://github.com/apache/arrow-rs.git path/to/arrow/rust"
echo "====================================================================="
exit 1;
fi

set -x

# show activated toolchain
rustup show

Expand All @@ -39,7 +56,4 @@ pushd ${source_dir}
# build only the integration testing binaries
cargo build -p arrow-integration-testing

# Remove incremental build artifacts to save space
rm -rf target/debug/deps/ target/debug/build/

popd
3 changes: 2 additions & 1 deletion dev/archery/archery/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -726,7 +726,8 @@ def _set_default(opt, default):
@click.option('--with-go', type=bool, default=False,
help='Include Go in integration tests')
@click.option('--with-rust', type=bool, default=False,
help='Include Rust in integration tests')
help='Include Rust in integration tests',
envvar="ARCHERY_INTEGRATION_WITH_RUST")
@click.option('--write_generated_json', default=False,
help='Generate test JSON to indicated path')
@click.option('--run-flight', is_flag=True, default=False,
Expand Down
1 change: 1 addition & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1197,6 +1197,7 @@ services:
<<: *ccache
# tell archery where the arrow binaries are located
ARROW_CPP_EXE_PATH: /build/cpp/debug
ARCHERY_INTEGRATION_WITH_RUST: 0
command:
["/arrow/ci/scripts/rust_build.sh /arrow /build &&
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
Expand Down