Skip to content

OpenVINO: fuse the 2D im2col patch embed into Convolution; NPU properties passthrough - #63

Open
sandlbn wants to merge 1 commit into
VinRobotics:mainfrom
sandlbn:fuse-2-im2col
Open

sandlbn wants to merge 1 commit into
VinRobotics:mainfrom
sandlbn:fuse-2-im2col

Conversation

@sandlbn

@sandlbn sandlbn commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

What

  • FuseToConv also folds the 2D im2col chain that ggml_conv_2d exports (Pad + ExtractImagePatches + Transpose/Reshape) into one Convolution. Before, it matched only the 1D chain. GGML_OPENVINO_FUSE_CONV2D=0 keeps the old graph for A/B.
  • GGML_OPENVINO_NPU_PROPS="K=V,..." passes extra properties into the NPU compile config, for example NPU_TURBO=YES.

Why

  • The NPU runs ExtractImagePatches on its SHAVE cores: 349.9 ms of a 382.8 ms SmolVLA SigLIP view at 512 px, by the per-op profile. The same tower built natively with a Convolution runs in 32.9 ms. SDPA fusion, f16 weights and the execution-mode, performance-hint and dynamic-quantization settings were each tested on their own; none moved the tower by more than ~5%.
  • The im2col cost grows with resolution: ~350 ms per view at 512 px, ~30 at 448, ~4 at 224. GR00T N1.6, VLA-JEPA and TurboVLA use a linear patch embed and do not change.
  • The NPU drops its clock between requests. With a server's 50–500 ms gaps that costs 50–70 ms per request, and NPU_TURBO=YES prevents it.

Tests

  • Latency, vla-bench p50, warmup 3, reps 20, back to back, before → after:

    Model Before After
    SmolVLA (2 × 512 px) 926 ms 177 ms (vision 810 → 63)
    Evo-1 (3 × 448 px) 572 ms 482 ms (vision 354 → 263)
    π0.5 500 ms 486 ms
    GR00T N1.5 263 ms 255 ms
    GR00T N1.6 241 ms 241 ms
    VLA-JEPA 161 ms 161 ms
    TurboVLA 27 ms 27 ms
  • NPU_TURBO=YES with 200 ms gaps (vla-bench --gap-ms 200, PR 4): SmolVLA 247 → 178 ms, π0.5 549 → 487 ms, GR00T N1.6 292 → 241 ms. No change back to back, and no change in numerics.

  • Correctness, max action error vs CPU f32, two fresh processes per model, all finite:

    Model Error
    π0.5 0.0015
    SmolVLA 0.0021
    GR00T N1.6 0.0023
    TurboVLA 0.0032
    Evo-1 0.0065
    VLA-JEPA 0.0068
    GR00T N1.5 0.0176 (0.0198 with the fusion off)

    On the CPU plugin, the fused graph is within 3e-5 of the im2col graph.

  • LIBERO-object on the NPU, PRs 1 + 2 + 3, NPU_TURBO=YES, median server time per request:

    Model Success Median per request
    SmolVLA 89/100 182 ms (was 90/100 at 981 ms)
    Evo-1 98/100 481 ms (was 97/100 at 648 ms)
    GR00T N1.5 95/100 383 ms (was 94/100 at 445 ms)
    π0.5, GR00T N1.6, VLA-JEPA, TurboVLA in progress
  • Patch script: applies to a fresh llama.cpp checkout and is idempotent.

…ties passthrough

ggml_conv_2d reaches OpenVINO as Pad + ExtractImagePatches + a Transpose /
Reshape chain, and the NPU runs ExtractImagePatches on its SHAVE cores:
349.9 ms of a 382.8 ms SmolVLA SigLIP view at 512 px. FuseToConv only matched
the 1D chain; it now also folds the 2D one into a single Convolution, and the
exported tower runs at the speed of a natively built one (31.7 vs 32.9 ms per
view). The cost grows with resolution (~350 ms per view at 512 px, ~30 at 448,
~4 at 224); towers with a linear patch embed do not change.
GGML_OPENVINO_FUSE_CONV2D=0 keeps the im2col graph for A/B.

GGML_OPENVINO_NPU_PROPS="K=V,..." passes extra properties to the NPU compile
config. NPU_TURBO=YES keeps the clock up between requests; with 200 ms gaps
(a server's pattern) SmolVLA 247 -> 178 ms, pi0.5 549 -> 487, GR00T N1.6
292 -> 241. No change back to back, no numerics change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant