Repository navigation
Conversation
…ties passthrough ggml_conv_2d reaches OpenVINO as Pad + ExtractImagePatches + a Transpose / Reshape chain, and the NPU runs ExtractImagePatches on its SHAVE cores: 349.9 ms of a 382.8 ms SmolVLA SigLIP view at 512 px. FuseToConv only matched the 1D chain; it now also folds the 2D one into a single Convolution, and the exported tower runs at the speed of a natively built one (31.7 vs 32.9 ms per view). The cost grows with resolution (~350 ms per view at 512 px, ~30 at 448, ~4 at 224); towers with a linear patch embed do not change. GGML_OPENVINO_FUSE_CONV2D=0 keeps the im2col graph for A/B. GGML_OPENVINO_NPU_PROPS="K=V,..." passes extra properties to the NPU compile config. NPU_TURBO=YES keeps the clock up between requests; with 200 ms gaps (a server's pattern) SmolVLA 247 -> 178 ms, pi0.5 549 -> 487, GR00T N1.6 292 -> 241. No change back to back, no numerics change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
FuseToConvalso folds the 2D im2col chain thatggml_conv_2dexports (Pad + ExtractImagePatches + Transpose/Reshape) into one Convolution. Before, it matched only the 1D chain.GGML_OPENVINO_FUSE_CONV2D=0keeps the old graph for A/B.GGML_OPENVINO_NPU_PROPS="K=V,..."passes extra properties into the NPU compile config, for exampleNPU_TURBO=YES.Why
ExtractImagePatcheson its SHAVE cores: 349.9 ms of a 382.8 ms SmolVLA SigLIP view at 512 px, by the per-op profile. The same tower built natively with a Convolution runs in 32.9 ms. SDPA fusion, f16 weights and the execution-mode, performance-hint and dynamic-quantization settings were each tested on their own; none moved the tower by more than ~5%.NPU_TURBO=YESprevents it.Tests
Latency,
vla-benchp50, warmup 3, reps 20, back to back, before → after:NPU_TURBO=YESwith 200 ms gaps (vla-bench --gap-ms 200, PR 4): SmolVLA 247 → 178 ms, π0.5 549 → 487 ms, GR00T N1.6 292 → 241 ms. No change back to back, and no change in numerics.Correctness, max action error vs CPU f32, two fresh processes per model, all finite:
On the CPU plugin, the fused graph is within 3e-5 of the im2col graph.
LIBERO-object on the NPU, PRs 1 + 2 + 3,
NPU_TURBO=YES, median server time per request:Patch script: applies to a fresh llama.cpp checkout and is idempotent.