Repository navigation
Conversation
layers/fused_ew.h, with CPU references and SYCL kernels: - fused_rms: RMSNorm into F32 / BF16 rows (fused_rms_q8 wrote only q8 blobs) - fused_swiglu: the GLU node with SiLU, into BF16 rows or a q8 blob - FUSED_GELU_ERF: an exact-GELU epilogue for fused_linear and qlinear (oneDNN eltwise_gelu_erf); the old bool argument still means tanh - fused_lnorm without affine (null weight and bias) - fused_rope_kv_table: RoPE from per-token cos/sin tables (rope_2d), with a BF16 output so a batch of views can be read in place by the attention modules/fused_lm.h: a Qwen2 / Llama decoder layer as 8 fused nodes. modules/encoder: EncCfg::gelu_erf, and fold_layer_scale for folding LayerScale into resident BF16 weights. tests/test_fused_sycl_ops checks every new node against the CPU reference. pi0.5's actions are bit-identical to before (default and W8A8).
On SYCL with BF16 weights and view-taking flash attention, DualTower runs both towers as EncStack's fused blocks (DINOv2's LayerScale folded into its weights), every view in one pass, the projector as three fused GEMMs; the Qwen2 LM layers use modules/fused_lm.h. --flash-attn 0 or VLA_ADAPTER_NO_FUSE=1 keep the per-op F32 graph. Core Ultra 7 356H, 224 px, vla-bench p50: 1 view 351 -> 124 ms, 2 views 753 -> 209 ms. LIBERO-object x100: 100/100 (per-op F32: 99/100).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on the SYCL fused-nodes PR. On SYCL with BF16 weights and view-taking flash attention (
--flash-attn 0orVLA_ADAPTER_NO_FUSE=1keep the per-op F32 graph):DualTower::enable_fusedruns both towers asEncStack's fused blocks with the erf GELU. DINOv2's LayerScale is folded into its proj / fc2 weights. Every view goes through each tower in one pass, and the projector is three fused GEMMs. The per-op graph ran each view's towers separately with explicit attention. OpenVLA-OFT uses the sameDualTowerand could callenable_fusedtoo; that is not enabled or measured here.modules/fused_lm.h. Each layer's F32 output still feeds its Bridge-Attention block.Results
Core Ultra 7 356H (Panther Lake), 224 px,
vla-benchp50:Actions vs the F32 graph (fixed inputs), max / mean: 5.4e-3 / 1.2e-3 (1 view), 3.7e-3 / 7.3e-4 (2 views), on actions up to 0.66.
LIBERO-object, 10 tasks x 10 episodes per arm, same binary:
VLA_ADAPTER_NO_FUSE=1)Verified
-Wall -Wextra(first-party code)ctestpassesvla_predict_checkdiff), or the change ismeant to move it and a LIBERO sweep is below
Archs and backends tested:
Linux x86 / PTL