Repository navigation
Conversation
layers/fused_ew.h, with CPU references and SYCL kernels: - fused_rms: RMSNorm into F32 / BF16 rows (fused_rms_q8 wrote only q8 blobs) - fused_swiglu: the GLU node with SiLU, into BF16 rows or a q8 blob - FUSED_GELU_ERF: an exact-GELU epilogue for fused_linear and qlinear (oneDNN eltwise_gelu_erf); the old bool argument still means tanh - fused_lnorm without affine (null weight and bias) - fused_rope_kv_table: RoPE from per-token cos/sin tables (rope_2d), with a BF16 output so a batch of views can be read in place by the attention modules/fused_lm.h: a Qwen2 / Llama decoder layer as 8 fused nodes. modules/encoder: EncCfg::gelu_erf, and fold_layer_scale for folding LayerScale into resident BF16 weights. tests/test_fused_sycl_ops checks every new node against the CPU reference. pi0.5's actions are bit-identical to before (default and W8A8).
On SYCL with BF16 weights and view-taking flash attention, without FoldQuant (--flash-attn 0 or VLA_GR00T_NO_FUSE=1 keep the per-op graph): - DitHead (shared by N1.5 / N1.6 / N1.7): each block as 7 fused nodes, the adaLN as a LayerNorm with weight 1 + scale (FlowTimes keeps it) and bias shift, attention reading the q|k|v / hoisted cross K/V outputs in place, residual adds and GELU as GEMM epilogues; dit_enable_fused is the switch - N1.5 / N1.6: SigLIP as EncStack's fused blocks, every view in one pass - N1.7: the Qwen3-VL tower fused too (RoPE from tables, views batched); opt-in (VLA_GR00T_N17_FUSE=1) until it has a LIBERO run Core Ultra 7 356H, 2 views, vla-bench p50: N1.5 671 -> 421 ms, N1.6 455 -> 214 ms, N1.7 (opt-in) 449 -> 227 ms. LIBERO-object x100: N1.5 95/100 (per-op 98/100), N1.6 90/100 (per-op 89/100). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on the SYCL fused-nodes PR. On SYCL with BF16 weights and view-taking flash attention, without FoldQuant:
modules/dit_head, shared by N1.5, N1.6 and N1.7): each block runs as 7 nodes.FlowTimesstores 1 + scale when the head is fused.proj_outis a LayerNorm plus one GEMM.dit_enable_fusedis the single switch all three models call.EncStack's fused blocks, every view in one pass. N1.5 previously encoded one view per compute.VLA_GR00T_N17_FUSE=1) because it has no LIBERO run yet; the eval client's tokenizer repo,nvidia/Cosmos-Reason2-2B, is gated. N1.7's default graph is unchanged: actions identical to main.--flash-attn 0orVLA_GR00T_NO_FUSE=1keep the per-op graph.Results
Core Ultra 7 356H (Panther Lake), 2 views,
vla-benchp50 (per-op → fused, interleaved runs):N1.7's larger error comes from reduced-precision attention in the tower: the per-op graph with
--flash-attn 1measures the same 1.3e-2.LIBERO-object, 10 tasks x 10 episodes per arm, same binary:
VLA_GR00T_NO_FUSE=1)--flash-attn 1(not default)N1.5's 95 vs 98 is within noise, and the misses are spread over four tasks. With
--flash-attn 1on top (LM and VL self-attention), N1.5 drops to 90/100 with misses on six tasks. That is why flash attention stays opt-in for GR00T.Verified
-Wall -Wextra(first-party code)ctestpassesvla_predict_checkdiff), or the change ismeant to move it and a LIBERO sweep is below
Archs and backends tested:
Linux x86 / PTL