FIP-3: Support tiering Fluss data to Iceberg

[leonardBang]

Search before asking

• I searched in the issues and found nothing similar.

Description

Implement Iceberg writer related interface based on POC code refinements

Willingness to contribute

• I'm willing to submit a PR!

[simonsssu]

cc @XuQianJin-Stars Linked https://github.com/alibaba/fluss/issues/102

[SML0127]

@leonardBang Is this ticekcts in progress? if not, I'd like to try this one.

[MehulBatra]

@leonardBang @luoyuxia If no one is working and it's ready to work on, I would love to collaborate/work on this.

[EhFa1999]

mr @wuchong and mr @leonardBang I have the ability to refactor the lakehouse storage architecture to support pluggable open table formats.
Each format is organized as a separate submodule within the fluss-lakehouse project:
fluss-lakehouse-paimon
fluss-lakehouse-delta-lake
fluss-lakehouse-iceberg
fluss-lakehouse-hudi

and Automatically append system columns for fluss primary key table when create each of open table format
I can also implement iceberg while implementing delta lake if you assign me this task.
but hudi connector before assigned
#453

[EhFa1999]

In any case, implementing and testing Delta is more difficult than Iceberg. Testing Delta also requires S3 and AWS Glue, whereas Iceberg is designed for easier integration and implementation with various systems. On the other hand, Delta stores metadata as a transaction log, which requires implementing more complex logic such as ACID guarantees, compaction, and checkpointing

[MehulBatra]

@luoyuxia I'd love to collaborate with you on this issue! If it helps move things forward, I'm happy to take the lead on drafting a design document and breaking the task down into smaller, manageable components.

[leonardBang]

@luoyuxia I'd love to collaborate with you on this issue! If it helps move things forward, I'm happy to take the lead on drafting a design document and breaking the task down into smaller, manageable components.

Hope to see your collaboration and looking forward iceberg integration in 0.8 version

[xx789633]

@leonardBang @MehulBatra Please include me in this integration as well. Maybe we can divide the tasks into multiple parts including writing into Iceberg and union read part?

[luoyuxia]

@luoyuxia I'd love to collaborate with you on this issue! If it helps move things forward, I'm happy to take the lead on drafting a design document and breaking the task down into smaller, manageable components.

Thanks for @MehulBatra @cwang9208 . I have worked drafting a design document. I'll share it maybe today. Definitely, we can discuss on the design document, and you all will take more in coding. I'm happy to review~