Repository navigation
Conversation
The guest builds its transport context once and keeps the ring addresses. For scratch to change size across a snapshot, the transport arena (virtqueue rings, pools and the checkpoint mailbox) needs an address that does not depend on the scratch size. The arena ends one free page below the reserved top pages. The page tables start at the scratch base. The free page takes a small overrun of the exception stack above it. The guest's page allocator stops at the arena GPA the host publishes, and aborts if that is not between the scratch base and the end of the transport range. The guest's transport memory access is bound to the arena. scratch_allocator_limit_gpa is renamed scratch_reserved_top_gpa, so a guest that stops its allocator at the old limit fails to compile. The minimum scratch size grows by the free page. This changes the snapshot layout: snapshot ABI 7, goldens v7.0. Existing snapshots must be regenerated, and guests rebuilt. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Sandbox::snapshot_with_scratch_size takes a snapshot whose layout has a smaller scratch size. Its page tables are built for that size, walking the live tables with the sandbox's own layout. Its virtqueue rings carry over unchanged, since the transport arena's address does not depend on the scratch size. Restore replaces scratch when the size differs, and restoring a larger snapshot grows it back. A size larger than the current scratch, too small for the transport, or not a whole number of host pages is rejected before the guest is entered. Room for the page tables is checked when they are built. The snapshot loader refuses a scratch size this host cannot map. A guest can boot with the scratch its initialisation needs and run guest function calls from a snapshot with only the scratch they need, so restores that zero or replace all of scratch cost less. Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added
kind/enhancement
danbugs
requested review from
andreiltd,
dblnz,
devigned,
jprendes,
jsturtevant,
ludfjig,
simongdavies,
squillace and
syntactically
as code owners
October 16, 2026 23:32
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Memory-layout and ABI changes need cross-platform human validation, and a host-page-dependent test failure remains unresolved.
1 open issue
What changed in this PR
Enables snapshots with smaller scratch regions to reduce restore costs while preserving transport addresses.
Changes:
- Adds scratch-size selection and validation for snapshots.
- Fixes transport placement and bounds guest allocation below it.
- Updates snapshot ABI, tests, and documentation.
| File | Description |
|---|---|
| src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs | Selects v7.0 goldens. |
| src/hyperlight_host/src/sandbox/snapshot/tripwires.rs | Pins ABI 7. |
| src/hyperlight_host/src/sandbox/snapshot/mod.rs | Builds snapshots with target layouts. |
| src/hyperlight_host/src/sandbox/snapshot/file/mod.rs | Validates host-page alignment. |
| src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs | Sets ABI 7. |
| src/hyperlight_host/src/sandbox/snapshot/file_tests.rs | Tests alignment and ABI rejection. |
| src/hyperlight_host/src/sandbox/initialized.rs | Adds resizing API and runtime tests. |
| src/hyperlight_host/src/mem/virtq/tests.rs | Updates transport address tests. |
| src/hyperlight_host/src/mem/virtq/mod.rs | Supports resized transport snapshots. |
| src/hyperlight_host/src/mem/mgr.rs | Passes target layouts and mailbox offsets. |
| src/hyperlight_host/src/mem/layout.rs | Defines fixed transport placement and sizing. |
| src/hyperlight_guest/src/transport/mem.rs | Bounds memory access to transport ranges. |
| src/hyperlight_guest/src/transport/context.rs | Uses scratch-independent transport bounds. |
| src/hyperlight_guest/src/layout.rs | Defines the allocator’s transport boundary. |
| src/hyperlight_guest/src/arch/amd64/prim_alloc.rs | Applies the allocator boundary. |
| src/hyperlight_guest/src/arch/aarch64/prim_alloc.rs | Applies the allocator boundary. |
| src/hyperlight_common/src/layout.rs | Defines transport placement helpers. |
| docs/virtio-host-guest-communication.md | Documents transport placement. |
| docs/snapshot-versioning.md | Documents ABI 7 compatibility. |
| docs/paging-development-notes.md | Explains resizing and allocation limits. |
| CHANGELOG.md | Records the API and breaking changes. |
Review effort: Balanced
Signed-off-by: danbugs <danilochiarlone@gmail.com>
This comment has been minimized.
This comment has been minimized.
Benchmark ResultsMeasured commit: kvm / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |
Member
|
Hey! Numbers look great. Could you explain why transport needs to be moved? Would it be possible to only move pools out of arena and reserve the top region for g2h/h2g pools only? This way we could remove all the knowledge about the pool placement from the host which would simplify the code quite a bit. Currently the pools need to be pinned in scratch memory between restores. This is a bit annoying but so far it has been the only way to avoid an extra round trip on restore. This is because host request can already be queued on restore in h2g queue. IIUC, this patch doesn't change this behavior? |
Contributor
Author
Thanks! Currently, the transport arena location is calculated once at boot and cached. With scratch resizing, those values end up wrong (since the base depends on the scratch size) and would have to be recalculated. If we move the transport arena just beneath the reserved pages, the remembered addresses stay valid. We could move only the pools, which would also let the host drop the pool placement logic, but then the guest would have to re-learn the ring and mailbox addresses after a resize. I figured moving the whole arena avoids that extra work... And right, this doesn't change the pool pinning: pools and rings keep their addresses across restores, including at a different scratch size, so host requests can still be queued in the H2G queue before the guest resumes. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sandbox::snapshot_with_scratch_sizetakes a snapshot that restores with a smaller scratch region. A guest boots with the scratch its initialisation needs and runs guest function calls from a snapshot with only what they need, so each restore resets less memory. Follows the discussion on #1901.snapshot_with_scratch_sizerejects a size that is larger, too small for the transport, or not whole host pages, before the guest is entered. The snapshot loader refuses a scratch size the host cannot map.