-
Notifications
You must be signed in to change notification settings - Fork 508
Description
Search before asking
- I searched in the issues and found nothing similar.
Description
Description:
When reading a batch of records using LogRecordBatch.ReadContext#getVectorSchemaRoot, the current implementation retains memory buffers even after the batch is processed. According to the Javadoc:
"DO NOT close the vector schema root because it is shared across multiple batches."
While this is correct (as new buffer loads replace old ones), it leads to a temporary duplication of buffers during batch transitions.
For example:
- Old buffer (from previous batch) is still referenced by the VectorSchemaRoot.
- New buffer (for current batch) is loaded into the same VectorSchemaRoot.
- The old buffer is not released until loadFieldBuffers.
Between 2 and 3, old buffer and new buffer are existed at same time, and cannot reuse.
Proposed Solution:
Still "DO NOT close the vector schema root because it is shared across multiple batches". However explicitly clear the VectorSchemaRoot at the end of each batch to release old buffers immediately.
vectorSchemaRoot.clear(); // Releases buffers but retains schema structureWillingness to contribute
- I'm willing to submit a PR!
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels