Skip to content

MINOR: Bounds check source buffer in BaseFixedWidthVector.copyFrom #1376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Support-itecz wants to merge 2 commits into
base: main
Choose a base branch
from
Open

MINOR: Bounds check source buffer in BaseFixedWidthVector.copyFrom #1376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50064a5
bounds check source buffer in BaseFixedWidthVector.copyFrom
Support-itecz Oct 20, 2026
221aa0f
bounds check source buffers in VectorAppender and concatBits
Support-itecz Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.util.ArrowBufPointer;
import org.apache.arrow.memory.util.ByteFunctionHelpers;
import org.apache.arrow.memory.util.MemoryUtil;
import org.apache.arrow.memory.util.hash.ArrowBufHasher;
import org.apache.arrow.util.Preconditions;
import org.apache.arrow.vector.compare.VectorVisitor;
Expand Down Expand Up @@ -833,10 +832,12 @@ public void copyFrom(int fromIndex, int thisIndex, ValueVector from) {
BitVectorHelper.unsetBit(this.getValidityBuffer(), thisIndex);
} else {
BitVectorHelper.setBit(this.getValidityBuffer(), thisIndex);
MemoryUtil.copyMemory(
from.getDataBuffer().memoryAddress() + (long) fromIndex * typeWidth,
this.getDataBuffer().memoryAddress() + (long) thisIndex * typeWidth,
typeWidth);
from.getDataBuffer()

@jbonofre jbonofre Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hardens only BaseFixedWidthVector.copyFrom. The same unchecked pattern, MemoryUtil.copyMemory on raw memoryAddress() plus an offset, remains in other places in vector/src/main. These include DecimalVector, Decimal256Vector, BitVectorHelper, and VectorAppender`.

If the goal is to make malformed IPC input fail with IndexOutOfBoundsException instead of an out-of-bounds read, can we say which of these are in scope?

If they are reachable from loadFieldBuffers style input, I would fix them here.

@Support-itecz Support-itecz Oct 21, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VectorAppender and BitVectorHelper.concatBits are reachable: the delta vector can be one loaded via loadFieldBuffers, and nothing checks its data, view, type or validity buffer against its value count before the raw copy. I added a source checkBytes before each of those copies and both concatBits inputs, plus two appender tests (short data buffer, short validity buffer) that fail without the change. The offset copies were already bounded by the getInt/getLong of the last offset right above them. DecimalVector/Decimal256Vector setSafe and setBigEndianSafe already checkBytes both buffers before copying, so I left those alone. The description now lists what is covered.

@jbonofre jbonofre Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, also, copyFrom is called once per element, and copyFromSafe loops and appenders call it in bulk. ArrowBuf.getBytes(long, ArrowBuf, long, int) adds bounds checks on both buffers plus argument validation, in place of a single raw copy.

Do you have a before/after number, for instance the existing JMH benchmarks for copyFrom? I would like to see the cost here.

Also, this only protects when bounds checking is on. With arrow.enable_unsafe_memory_access=true, getBytes skips validation and the read is unchecked again. That is probably acceptable, since the flag disables checks by design, but please say so in the description.

@Support-itecz Support-itecz Oct 21, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran Float8Benchmarks.copyFromBenchmark (1024 copies per op, 2 forks x 8 iterations on a laptop, so the error bars are wide):

  • checks on: main 13.98 ± 6.03 us/op, this PR 19.89 ± 5.18 us/op
  • enable_unsafe_memory_access=true: main 8.51 ± 4.87, this PR 8.79 ± 4.83

So about 6 ns per value with checks on and nothing measurable with them off. I also tried checkBytes on both buffers plus the original raw copy instead of getBytes, and it came out the same (20.38 ± 11.60), so I kept getBytes. The appender changes are one range check per buffer per append, not per value.

Agreed on the unsafe flag. With it set the source check is skipped and the read is unchecked again. That is in the description now.

.getBytes(
(long) fromIndex * typeWidth,
this.getDataBuffer(),
(long) thisIndex * typeWidth,
typeWidth);
}
}

Expand Down
2 changes: 2 additions & 0 deletions vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,8 @@ public static void concatBits(
int numBytesOut = DataSizeRoundingUtil.divideBy8Ceil(numBits1 + numBits2);

if (BoundsChecking.BOUNDS_CHECKING_ENABLED) {
input1.checkBytes(0, numBytes1);
input2.checkBytes(0, numBytes2);
output.checkBytes(0, numBytesOut);
}

Expand Down
11 changes: 11 additions & 0 deletions vector/src/main/java/org/apache/arrow/vector/util/VectorAppender.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@ public ValueVector visit(BaseFixedWidthVector deltaVector, Void value) {
deltaVector.getValueCount(),
targetVector.getDataBuffer());
} else {
deltaVector
.getDataBuffer()
.checkBytes(0, (long) deltaVector.getTypeWidth() * deltaVector.getValueCount());
MemoryUtil.copyMemory(
deltaVector.getDataBuffer().memoryAddress(),
targetVector.getDataBuffer().memoryAddress()
Expand Down Expand Up @@ -153,6 +156,7 @@ public ValueVector visit(BaseVariableWidthVector deltaVector, Void value) {
targetVector.getValidityBuffer());

// append data buffer
deltaVector.getDataBuffer().checkBytes(deltaDataStart, deltaDataStart + deltaDataSize);
MemoryUtil.copyMemory(
deltaVector.getDataBuffer().memoryAddress() + deltaDataStart,
targetVector.getDataBuffer().memoryAddress() + targetDataSize,
Expand Down Expand Up @@ -229,6 +233,7 @@ public ValueVector visit(BaseLargeVariableWidthVector deltaVector, Void value) {
targetVector.getValidityBuffer());

// append data buffer
deltaVector.getDataBuffer().checkBytes(deltaDataStart, deltaDataStart + deltaDataSize);
MemoryUtil.copyMemory(
deltaVector.getDataBuffer().memoryAddress() + deltaDataStart,
targetVector.getDataBuffer().memoryAddress() + targetDataSize,
Expand Down Expand Up @@ -300,6 +305,10 @@ public ValueVector visit(BaseVariableWidthViewVector deltaVector, Void value) {

// append view buffer
ArrowBuf targetViewBuffer = targetVector.getDataBuffer();
deltaVector
.getDataBuffer()
.checkBytes(
0, (long) BaseVariableWidthViewVector.ELEMENT_SIZE * deltaVector.getValueCount());
MemoryUtil.copyMemory(
deltaVector.getDataBuffer().memoryAddress(),
targetViewBuffer.memoryAddress()
Expand Down Expand Up @@ -606,6 +615,7 @@ public ValueVector visit(UnionVector deltaVector, Void value) {
}

// append type buffers
deltaVector.getTypeBuffer().checkBytes(0, deltaVector.getValueCount());
MemoryUtil.copyMemory(
deltaVector.getTypeBufferAddress(),
targetUnionVector.getTypeBufferAddress() + targetVector.getValueCount(),
Expand Down Expand Up @@ -671,6 +681,7 @@ public ValueVector visit(DenseUnionVector deltaVector, Void value) {
}

// append type buffers
deltaVector.getTypeBuffer().checkBytes(0, deltaVector.getValueCount());
MemoryUtil.copyMemory(
deltaVector.getTypeBuffer().memoryAddress(),
targetDenseUnionVector.getTypeBuffer().memoryAddress() + targetVector.getValueCount(),
Expand Down
27 changes: 27 additions & 0 deletions vector/src/test/java/org/apache/arrow/vector/TestCopyFrom.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,19 @@
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Period;
import java.util.Arrays;
import java.util.Objects;
import org.apache.arrow.memory.ArrowBuf;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.ipc.message.ArrowFieldNode;
import org.apache.arrow.vector.types.Types.MinorType;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
Expand Down Expand Up @@ -1098,4 +1102,27 @@ public void testCopySafeArrow7837() {
assertEquals("1234567890", Objects.requireNonNull(vc2.getObject(5)).toString());
}
}

@Test
public void testCopyFromFixedWidthSourceBufferTooShort() {
try (IntVector source = new IntVector("source", allocator);
IntVector target = new IntVector("target", allocator);
ArrowBuf body = allocator.buffer(64)) {
body.setZero(0, body.capacity());
// all 4 values marked valid
body.setByte(0, 0x0F);
body.setInt(8, 1);
body.setInt(20, 0xBAD);
// Field node claims 4 values, but the data buffer only holds one, as a malformed IPC

@jbonofre jbonofre Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source is loaded with a zero-length validity buffer (body.slice(0, 0)) and a null count of 0. The first copyFromSafe(0, 0, source) then needs isSet(0) to read that buffer without throwing.

Is that guaranteed, or does it work only because of how isSet handles this case today?
If a later change makes isSet check the buffer length, this test fails for a reason unrelated to the data-buffer bounds check it's meant to cover.

Giving the source a real validity buffer (all bits set) would make the test say what it means.

@Support-itecz Support-itecz Oct 21, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Today it works because loadValidityBuffer replaces an empty validity buffer with an all-ones one when the null count is 0, not because of isSet, but the test should not rely on that. The source now gets a real 8-byte validity buffer with all 4 bits set, and the new appender tests do the same.

// record batch can.
ArrowBuf validity = body.slice(0, 8);
ArrowBuf data = body.slice(8, IntVector.TYPE_WIDTH);
source.loadFieldBuffers(new ArrowFieldNode(4, 0), Arrays.asList(validity, data));

target.allocateNew(4);
target.copyFromSafe(0, 0, source);
assertEquals(1, target.get(0));
assertThrows(IndexOutOfBoundsException.class, () -> target.copyFromSafe(3, 1, source));
}
}
}
43 changes: 43 additions & 0 deletions vector/src/test/java/org/apache/arrow/vector/util/TestVectorAppender.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,49 @@ public void testAppendFixedWidthVector() {
}
}

@Test
public void testAppendFixedWidthVectorSourceBufferTooShort() {
try (IntVector target = new IntVector("", allocator);
IntVector delta = new IntVector("", allocator);
ArrowBuf body = allocator.buffer(64)) {
body.setZero(0, body.capacity());
// all 4 values marked valid
body.setByte(0, 0x0F);
body.setInt(8, 1);
// Field node claims 4 values, but the data buffer only holds one, as a malformed IPC
// record batch can.
ArrowBuf validity = body.slice(0, 8);
ArrowBuf data = body.slice(8, IntVector.TYPE_WIDTH);
delta.loadFieldBuffers(new ArrowFieldNode(4, 0), Arrays.asList(validity, data));

target.allocateNew(4);
ValueVectorDataPopulator.setVector(target, 0, 1, 2, 3);

VectorAppender appender = new VectorAppender(target);
assertThrows(IndexOutOfBoundsException.class, () -> delta.accept(appender, null));
}
}

@Test
public void testAppendValidityBufferTooShort() {
try (IntVector target = new IntVector("", allocator);
IntVector delta = new IntVector("", allocator);
ArrowBuf body = allocator.buffer(128)) {
body.setZero(0, body.capacity());
body.setByte(0, 0x0F);
// Field node claims 16 values, but the validity buffer only covers 8.
ArrowBuf validity = body.slice(0, 1);
ArrowBuf data = body.slice(64, 16 * IntVector.TYPE_WIDTH);
delta.loadFieldBuffers(new ArrowFieldNode(16, 4), Arrays.asList(validity, data));

target.allocateNew(8);
ValueVectorDataPopulator.setVector(target, 0, 1, 2, 3, 4, 5, 6, 7);

VectorAppender appender = new VectorAppender(target);
assertThrows(IndexOutOfBoundsException.class, () -> delta.accept(appender, null));
}
}

@Test
public void testAppendBitVector() {
final int length1 = 10;
Expand Down
Loading