Skip to content

smolvla: one camera view's vision encoder on the NPU (prototype) - #59

Open
sandlbn wants to merge 3 commits into
VinRobotics:mainfrom
sandlbn:smolvla-npu-view
Open

smolvla: one camera view's vision encoder on the NPU (prototype)#59
sandlbn wants to merge 3 commits into
VinRobotics:mainfrom
sandlbn:smolvla-npu-view

Conversation

@sandlbn

@sandlbn sandlbn commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Depends on the SmolVLA fused PR (patch 2). Opt-in; with the variable unset, SmolVLA's graph and actions are unchanged (bit-identical).

With VLA_SMOLVLA_NPU_VISION=<socket> and 2 or more camera views:

  • Split: SmolVLA sends its last view to a worker on the Intel NPU, while the GPU encodes the other views in one pass. Vision then costs the slower of the two instead of their sum.

  • Fallback: if the NPU fails, that view is encoded on the GPU.

  • scripts/build_vision_ir.py (new): builds SmolVLA's vision encoder as an OpenVINO graph straight from the GGUF: patch embedding, 12 SigLIP blocks, post-LayerNorm, pixel shuffle and connector.

    • It matches the ggml graph to 3e-6 (OpenVINO CPU plugin, f32).
    • It runs in 31 ms per view on the NPU. The graph ggml-openvino exports for the same tower takes ~330 ms there.

  • scripts/npu_vision_worker.py: serves that graph on the NPU. It gains --turbo (NPU_TURBO): without it the NPU clocks down between a policy loop's requests, and each view costs about twice as much.

  • src/modules/npu_vision.h: the socket client. SmolVLA's vision code becomes one encode(first, count) helper, shared by both paths.

Usage

python3 scripts/build_vision_ir.py smolvla-libero.gguf smolvla_vision.xml      # once
# in a shell without oneAPI's setvars (OpenVINO and oneAPI ship conflicting TBB/OpenCL):
python3 scripts/npu_vision_worker.py --ir smolvla_vision.xml --sock /tmp/smolvla_npu.sock --turbo
# the SYCL server, as usual, plus:
VLA_SMOLVLA_NPU_VISION=/tmp/smolvla_npu.sock build-sycl/vla-server ...

Results

Core Ultra 7 356H (Panther Lake, Xe3 iGPU + NPU 5), OpenVINO 2026.4.1, NPU driver 1.38, 512 px.

vla-bench p50, back-to-back requests, interleaved over two rounds:

GPU only GPU + NPU last view
3 views 151.5 ms (vision 66 ms) 135.5 ms (vision 50 ms)
2 views 120.8 ms (vision 44 ms) 121.5 ms (vision 39 ms)

LIBERO-object, 10 tasks x 10 episodes per arm, same binary, 2 views (LIBERO's cameras):

success Wilson 95% median ms / request
GPU only 80/100 71-87% 142
GPU + NPU second view 87/100 79-92% 122
  • No fallbacks: the NPU was busy for 64 s of the NPU arm, with zero GPU fallbacks and zero worker errors.
  • Accuracy: 87 vs 80 is within noise at n = 100 (p ~ 0.2). The same GPU-only path scored 85/100 in an earlier run, so read this as no accuracy loss.
  • Actions: with the NPU view they move by at most 2.2e-3 (mean 1e-4) against GPU-only (vla_predict_check, fixed inputs).

Why it is faster, and when

  • Per view, the GPU is faster: 22 ms against the NPU's 31 ms. The gain comes only from overlap: the NPU view is hidden while the GPU encodes the other views.
  • 3 views: the NPU view hides completely behind the GPU's two views (vision 66 -> 50 ms).
  • 2 views: vision still drops 44 -> 39 ms, but with back-to-back requests the rest of the request gets ~5 ms slower, so vla-bench shows no net gain. In the LIBERO loop, which has gaps between requests, the median request is 20 ms faster.
  • Memory contention: a fully busy NPU slows the GPU by 22-25% through shared memory; one view per request cost the GPU 5-10% in these runs.
  • Calibrated INT8 on the NPU: the next lever, ~22 ms per view, not in this patch.

Verified

  • Builds clean under -Wall -Wextra (first-party code)
  • ctest passes
  • Numeric output unchanged (vla_predict_check diff), or the change is
    meant to move it and a LIBERO sweep is below

Archs and backends tested:

Linux x86 / PTL

@sandlbn
sycl: fused nodes for SmolLM / Qwen2 / InternViT / DiT layers
d3a3d06 
layers/fused_ew.h, with CPU references and SYCL kernels:
- fused_rms: RMSNorm into F32 / BF16 rows (fused_rms_q8 wrote only q8 blobs)
- fused_swiglu: the GLU node with SiLU, into BF16 rows or a q8 blob
- FUSED_GELU_ERF: an exact-GELU epilogue for fused_linear and qlinear
  (oneDNN eltwise_gelu_erf); the old bool argument still means tanh
- fused_lnorm without affine (null weight and bias)
- fused_rope_kv_table: RoPE from per-token cos/sin tables (rope_2d), with a
  BF16 output so a batch of views can be read in place by the attention

modules/fused_lm.h: a Qwen2 / Llama decoder layer as 8 fused nodes.
modules/encoder: EncCfg::gelu_erf, and fold_layer_scale for folding
LayerScale into resident BF16 weights.

tests/test_fused_sycl_ops checks every new node against the CPU reference.
pi0.5's actions are bit-identical to before (default and W8A8).
@sandlbn
smolvla: fused vision tower, LM and action expert on SYCL
1f8c124 
On SYCL with BF16 weights and view-taking flash attention:
- flash attention (oneDNN's fused SDPA) on by default on SYCL unless
  --flash-attn is given; the fused paths below need it
- SigLIP as EncCfg::fused blocks, both views in one pass (q, k and v stream
  into row slices of one q|k|v weight at load)
- every SmolLM2 layer (VLM prefix and expert) as 9 fused nodes, K/V written
  in place into per-layer F16 buffers; cross-attention layers reproject the
  prefix rows once per request; residual stream F32
- opt-in VLA_SMOLVLA_W8A8=all|vit,vlm,expert: those GEMMs as W8A8

--flash-attn 0, VLA_SMOLVLA_NO_VIT_FUSE=1 / VLA_SMOLVLA_NO_FUSE=1 keep the
per-op graphs.

Core Ultra 7 356H, 2 views at 512 px, vla-bench p50: 558 ms (--flash-attn 0)
-> 259 ms (flash attention) -> 121-131 ms (fused). LIBERO-object x100:
85/100 (flash attention only 88/100, --flash-attn 0 92/100); not
significant at n = 100, needs a larger run.
@sandlbn
smolvla: prototype - one camera view's vision encoder on the NPU
098dff8 
VLA_SMOLVLA_NPU_VISION=<socket> with 2+ views: the last view goes to
scripts/npu_vision_worker.py on the Intel NPU while the GPU encodes the
others, so vision costs max(GPU, NPU) instead of the sum. Any NPU failure
falls back to the GPU for that view. Without the variable the graph and the
actions are unchanged (bit-identical).

scripts/build_vision_ir.py builds SmolVLA's vision encoder (patch embed,
SigLIP blocks, post-LN, pixel shuffle, connector) as an OpenVINO graph straight
from the GGUF: it matches the ggml graph to 3e-6 (CPU f32) and runs in 31 ms
per view on the NPU, where the graph ggml-openvino exports takes ~330 ms.
The worker gains --turbo (NPU_TURBO, keeps the NPU clocked between requests).

Core Ultra 7 356H, 512 px, vla-bench p50: 3 views 151.5 -> 135.5 ms (vision
66 -> 50 ms); 2 views 120.8 -> 121 ms (vision 44 -> 39 ms, not a net gain).
Actions vs GPU-only: max 2.2e-3, mean 1e-4. LIBERO-object x100, same
binary, 2 views: 87/100 with the NPU view (median 122 ms per request) vs
80/100 GPU-only (142 ms); no fallbacks to the GPU.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sandlbn