Array SLT fails when run using `dev` profile

[pepijnve]

Describe the bug

In discord thread https://discord.com/channels/885562378132000778/1462024777643327610 it was mentioned that when run locally an SLT is failing. This is the error that's reported:

External error: 1 errors in file /Users/pepijn/RustroverProjects/datafusion/datafusion/sqllogictest/test_files/array.slt

1. query failed: DataFusion error: Arrow error: Invalid argument error: Codecs [List(RowConverter { fields: [SortField { options: SortOptions { descending: false, nulls_first: true }, data_type: Null }], codecs: [Stateless] })] did not consume all bytes for row 0, remaining bytes: [1]
[SQL] select array_union([[null]], []);
at /Users/pepijn/RustroverProjects/datafusion/datafusion/sqllogictest/test_files/array.slt:4750

A git bisect points to PR #19355 for issue #19290

To Reproduce

Run cargo test --profile=dev --test sqllogictests

Expected behavior

All SLTs pass

Additional context

No response

[pepijnve]

What's odd is that the same SLTs when run by CI are passing. I haven't investigated where the difference might be coming from.

[alamb]

I think the CI runs it like

cargo test --profile=ci --test sqllogictests

I am not sure if that matters

[pepijnve]

I am not sure if that matters

Yep, test passes with the ci profile

[kumarUjjawal]

I think it is related to this #19798, I explored it in here #19813 but after discussion with @Jefffrey we decided it should be solved in upstream in arrow-rs. #19813 (comment)

[pepijnve]

Yes, you're right. This is the same issue. I'll close this one as duplicate of #19798