Repository navigation
Conversation
layers/fused_ew.h, with CPU references and SYCL kernels: - fused_rms: RMSNorm into F32 / BF16 rows (fused_rms_q8 wrote only q8 blobs) - fused_swiglu: the GLU node with SiLU, into BF16 rows or a q8 blob - FUSED_GELU_ERF: an exact-GELU epilogue for fused_linear and qlinear (oneDNN eltwise_gelu_erf); the old bool argument still means tanh - fused_lnorm without affine (null weight and bias) - fused_rope_kv_table: RoPE from per-token cos/sin tables (rope_2d), with a BF16 output so a batch of views can be read in place by the attention modules/fused_lm.h: a Qwen2 / Llama decoder layer as 8 fused nodes. modules/encoder: EncCfg::gelu_erf, and fold_layer_scale for folding LayerScale into resident BF16 weights. tests/test_fused_sycl_ops checks every new node against the CPU reference. pi0.5's actions are bit-identical to before (default and W8A8).
On SYCL with BF16 weights and view-taking flash attention: - flash attention (oneDNN's fused SDPA) on by default on SYCL unless --flash-attn is given; the fused paths below need it - SigLIP as EncCfg::fused blocks, both views in one pass (q, k and v stream into row slices of one q|k|v weight at load) - every SmolLM2 layer (VLM prefix and expert) as 9 fused nodes, K/V written in place into per-layer F16 buffers; cross-attention layers reproject the prefix rows once per request; residual stream F32 - opt-in VLA_SMOLVLA_W8A8=all|vit,vlm,expert: those GEMMs as W8A8 --flash-attn 0, VLA_SMOLVLA_NO_VIT_FUSE=1 / VLA_SMOLVLA_NO_FUSE=1 keep the per-op graphs. Core Ultra 7 356H, 2 views at 512 px, vla-bench p50: 558 ms (--flash-attn 0) -> 259 ms (flash attention) -> 121-131 ms (fused). LIBERO-object x100: 85/100 (flash attention only 88/100, --flash-attn 0 92/100); not significant at n = 100, needs a larger run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
smolvla: fused vision tower, LM and action expert on SYCL
Depends on the SYCL fused-nodes PR. On SYCL with BF16 weights and view-taking flash attention (the SYCL default since #52):
EncCfg::fusedblocks with BF16 activations, both views in one pass. Every GEMM carries its bias, GELU and residual, and q|k|v is one GEMM (q, k and v stream into row slices of it at load). Attention reads that GEMM's output in place.VLA_SMOLVLA_W8A8=all(orvit,vlm,expert): those GEMMs run on oneDNN's INT8 path with per-token activations. The weights are quantized at load.VLA_SMOLVLA_NO_VIT_FUSE=1/VLA_SMOLVLA_NO_FUSE=1keep the per-op graphs.Results
Core Ultra 7 356H (Panther Lake), 2 views at 512 px,
vla-benchp50 (interleaved runs):VLA_SMOLVLA_W8A8=allActions vs
--flash-attn 0(fixed inputs), max / mean: this PR 3.7e-3 / 1.4e-4 (main with FA 1.6e-3), W8A8 2.0e-2 / 6.3e-4.LIBERO-object, 10 tasks x 10 episodes per arm:
--flash-attn 0(default before #52)VLA_SMOLVLA_W8A8=allThe intervals overlap, and 85 vs 92 is not significant at n = 100 (p ~ 0.12). The failures move between tasks from arm to arm: task 2 is 6/10 here and 10/10 with W8A8, and task 5 is 4-7/10 everywhere. The three BF16 arms do all sit below the F32 one, though, so this needs a larger run before the fused path is called neutral. Draft until then; the switches above keep the old graph.
Verified
-Wall -Wextra(first-party code)ctestpassesvla_predict_checkdiff), or the change ismeant to move it and a LIBERO sweep is below
Archs and backends tested: