GH-343: Fix BaseVariableWidthVector and BaseLargeVariableWidthVector offset buffer serialization #989
+48
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's Changed
Fix
BaseVariableWidthVector/BaseLargeVariableWidthVectorIPC serialization whenvalueCountis 0.Problem
When
valueCount == 0,setReaderAndWriterIndex()was settingoffsetBuffer.writerIndex(0), which meansreadableBytes() == 0. IPC serializer usesreadableBytes()to determine buffer size, so 0 bytes were written to the IPC stream. This crashes IPC readers in other libraries because Arrow spec requires offset buffer to have at least one entry[0].This is a follow-up to #967 which fixed the same issue in
ListVector/LargeListVector.Fix
Simplify
setReaderAndWriterIndex()to always use(valueCount + 1) * OFFSET_WIDTHfor offset buffer'swriterIndex. WhenvalueCount == 0, this correctly setswriterIndextoOFFSET_WIDTH, ensuringoffset[0]is included in serialization.Testing
Added tests for empty
VarCharVectorandLargeVarCharVectorverifying offset buffer has correctreadableBytes().related to #343