Migrate Avro reader to arrow-avro and remove internal conversion code by getChan · Pull Request #17861 · apache/datafusion

getChan · 2025-10-01T14:10:36Z

Which issue does this PR close?

Closes Use arrow-avro for performance and improved type support #14097

Rationale for this change

DataFusion previously maintained custom Avro-to-Arrow conversion logic.
This PR migrates Avro reading to arrow-avro to align behavior with upstream Arrow and remove duplicated implementation.

What changes are included in this PR?

Switched DataFusion Avro reader path to arrow-avro (ReaderBuilder)
Removed internal/legacy Avro conversion paths that are no longer needed
Updated crate wiring to use arrow-avro and removed prior apache-avro dependency usage in affected paths
Updated Avro projection flow to use arrow-avro projection support
Added/updated upgrade documentation for Avro API and behavior changes

Are these changes tested?

Yes.

Added/updated Avro reader unit tests in datafusion/datasource-avro (including projection and timestamp logical types)
Updated SQL logic tests in datafusion/sqllogictest/test_files/avro.slt
Integration is covered by existing CI/test suites for affected crates

Are there any user-facing changes?

Yes.

DataFusionError::AvroError is removed.
From<apache_avro::Error> for DataFusionError is removed.
Re-export changed from datafusion::apache_avro to datafusion::arrow_avro.
Avro feature wiring changed:
- datafusion crate avro feature no longer enables datafusion-common/avro
- datafusion-proto crate avro feature no longer enables datafusion-common/avro
Avro decoding behavior now follows arrow-avro semantics, including:
- Avro string values being read as Arrow Binary in this path
- timestamp-* logical types read as UTC timezone-aware timestamps (Timestamp(..., Some("+00:00")))
- local-timestamp-* remaining timezone-naive (Timestamp(..., None))

Upgrade notes are documented in:
docs/source/library-user-guide/upgrading/53.0.0.md

alamb · 2025-10-02T10:10:04Z

❤️ amazing! Thank you @getChan
FYI @jecsand838 and @nathaniel-d-ef

alamb · 2025-10-19T20:04:52Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

getChan · 2025-10-19T23:10:10Z

Hi @getChan -- I am preparing to make an arrow release -- have you hit any blockers while integrating the new arrow-avro crate into DataFusion?

No, not yet. Thanks for release.

nathaniel-d-ef · 2025-10-20T15:21:27Z

Thanks for jumping on this @getChan; let me know if I can help!

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

alamb · 2025-10-29T11:08:03Z

FYI I merged the arrow 57 upgrade to DataFusion -- so if you rebase this PR against main you'll have access to the new arrow-avro crate

# Conflicts: # Cargo.lock

getChan · 2026-01-18T17:57:56Z

Hello. @alamb.
Thanks to the work by jecsand838, I think we’ll be able to apply the code in the next arrow-avro release.
However, as I mentioned in the post above, API change is planned. Do we need any additional steps to prepare for this?

alamb · 2026-01-19T15:11:49Z

Hello. @alamb. Thanks to the work by jecsand838, I think we’ll be able to apply the code in the next arrow-avro release. However, as I mentioned in the post above, API change is planned. Do we need any additional steps to prepare for this?

This is great news @getChan -- thank you! Very exciting.

What API change are you referring to?

If you mean API changes for any users of DataFusion, then I think the best thing to do is the same as for any breaking public API change. It should documented in the upgrading.md guide:
https://github.com/apache/datafusion/blob/main/docs/source/library-user-guide/upgrading.md

getChan · 2026-02-23T15:14:31Z

Status update:

arrow-avro 58.0.0 is now released, and I’ve updated this PR accordingly.
After Arrow 58.0.0 lands in main, I will:
resolve the remaining merge conflicts,
add the Upgrade Guides, and
move this PR to Ready for review.

alamb · 2026-02-23T22:19:44Z

Here is a PR that updates DataFusion to use arrow 58:

Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 0.13.0 #19728

alamb · 2026-02-27T19:34:01Z

I am just waiting for someone to approve #19728 and we'll be good to go

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/core/src/datasource/file_format/avro.rs

getChan · 2026-03-01T11:23:01Z

Now ready for review.

Are these breaking changes acceptable for the arrow-avro migration?

draft: using arrow-avro

e27e1f8

github-actions bot added common Related to common crate datasource Changes to the datasource crate labels Oct 1, 2025

draft: using arrow-avro

027e79d

getChan added 3 commits October 19, 2025 17:15

Merge remote-tracking branch 'origin/main' into arrow-avro

2bf2c89

cargo fmt

2c7e364

check public API's can be replaced

7c9f976

getChan added 3 commits October 27, 2025 23:52

revert arrow dependency

7bc24ea

Merge remote-tracking branch 'origin/main' into arrow-avro

185ede5

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/common/Cargo.toml

apply arrow-avro 57.0.0

51557fe

github-actions bot removed the common Related to common crate label Oct 27, 2025

timsaucer mentioned this pull request Oct 28, 2025

Switch from deprecated xz2 to libzma #18331

Closed

getChan added 2 commits October 30, 2025 00:36

BufReader

8e12f47

remove apache-avro dependencies

83ec01d

github-actions bot added core Core DataFusion crate common Related to common crate proto Related to proto crate labels Oct 29, 2025

getChan added 8 commits October 30, 2025 01:16

Merge remote-tracking branch 'origin/main' into arrow-avro

f37fcd3

# Conflicts: # Cargo.lock

remove apache-avro dependencies

b12f563

remove apache-avro dependencies

581ecae

fix test result

74792ff

pass projected schema to arrow-avro

151ea95

testing submodule checkout

78df065

taplo format

c7b3b63

replace with arrow-avro

2556591

getChan added 2 commits January 19, 2026 02:13

add test for datatype

fe11959

submodule

d65d847

jecsand838 mentioned this pull request Jan 23, 2026

AvroError enum for arrow-avro crate apache/arrow-rs#8759

Merged

upgrade arrow-avro 58.0.0

b1351b6

getChan force-pushed the arrow-avro branch 4 times, most recently from 443830c to b1351b6 Compare February 24, 2026 16:37

getChan added 4 commits February 28, 2026 21:19

Merge remote-tracking branch 'origin/main' into arrow-avro

3ab7404

# Conflicts: # Cargo.lock # Cargo.toml # datafusion/core/src/datasource/file_format/avro.rs

fix avro test expectation for timestamp timezone

d3e27f6

Reset testing and parquet-testing submodule pointers to main

a59e556

Update testing submodule to add avro timestamp test data

2e79c73

getChan mentioned this pull request Feb 28, 2026

add avro timestamp file apache/arrow-testing#124

Open

getChan added 8 commits February 28, 2026 23:25

test(avro): align sqllogictest schema expectations

b92b320

Merge remote-tracking branch 'origin/main' into arrow-avro

7020131

avro: remove reader schema coercion and align tests

e2f63c2

avro: disable repartition scan and update sqllogictest

d363ef9

avro: handle empty projection and keep schema metadata normalized

4c7944d

avro: strip inferred schema metadata for stable runtime matching

d1e4e8c

cargo fmt

c3bd1c8

docs: add avro upgrade notes for arrow-avro migration

6834f70

github-actions bot added the documentation Improvements or additions to documentation label Mar 1, 2026

getChan marked this pull request as ready for review March 1, 2026 11:18

getChan changed the title ~~WIP: using arrow-avro. remove own implementation~~ Migrate Avro reader to arrow-avro and remove internal conversion code Mar 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate Avro reader to arrow-avro and remove internal conversion code#17861

Migrate Avro reader to arrow-avro and remove internal conversion code#17861
getChan wants to merge 62 commits intoapache:mainfrom
getChan:arrow-avro

getChan commented Oct 1, 2025 •

edited

Loading

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

getChan commented Jan 18, 2026

Uh oh!

alamb commented Jan 19, 2026

Uh oh!

getChan commented Feb 23, 2026

Uh oh!

alamb commented Feb 23, 2026

Uh oh!

alamb commented Feb 27, 2026

Uh oh!

getChan commented Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

getChan commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Oct 2, 2025

Uh oh!

alamb commented Oct 19, 2025

Uh oh!

getChan commented Oct 19, 2025

Uh oh!

nathaniel-d-ef commented Oct 20, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

getChan commented Jan 18, 2026

Uh oh!

alamb commented Jan 19, 2026

Uh oh!

getChan commented Feb 23, 2026

Uh oh!

alamb commented Feb 23, 2026

Uh oh!

alamb commented Feb 27, 2026

Uh oh!

getChan commented Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

getChan commented Oct 1, 2025 •

edited

Loading