Repository navigation
Conversation
The guest builds its transport context once and keeps the ring addresses. For scratch to change size across a snapshot, the transport arena (virtqueue rings, pools and the checkpoint mailbox) needs an address that does not depend on the scratch size. The arena ends one free page below the reserved top pages. The page tables start at the scratch base. The free page takes a small overrun of the exception stack above it. The guest's page allocator stops at the arena GPA the host publishes, and aborts if that is not between the scratch base and the end of the transport range. The guest's transport memory access is bound to the arena. scratch_allocator_limit_gpa is renamed scratch_reserved_top_gpa, so a guest that stops its allocator at the old limit fails to compile. The minimum scratch size grows by the free page. This changes the snapshot layout: snapshot ABI 7, goldens v7.0. Existing snapshots must be regenerated, and guests rebuilt. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Sandbox::snapshot_with_scratch_size takes a snapshot whose layout has a smaller scratch size. Its page tables are built for that size, walking the live tables with the sandbox's own layout. Its virtqueue rings carry over unchanged, since the transport arena's address does not depend on the scratch size. Restore replaces scratch when the size differs, and restoring a larger snapshot grows it back. A size larger than the current scratch, too small for the transport, or not a whole number of host pages is rejected before the guest is entered. Room for the page tables is checked when they are built. The snapshot loader refuses a scratch size this host cannot map. A guest can boot with the scratch its initialisation needs and run guest function calls from a snapshot with only the scratch they need, so restores that zero or replace all of scratch cost less. Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added
kind/enhancement
danbugs
requested review from
andreiltd,
dblnz,
devigned,
jprendes,
jsturtevant,
ludfjig,
simongdavies,
squillace and
syntactically
as code owners
October 16, 2026 23:32
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Memory-layout and ABI changes need cross-platform human validation, and a host-page-dependent test failure remains unresolved.
1 open issue
What changed in this PR
Enables snapshots with smaller scratch regions to reduce restore costs while preserving transport addresses.
Changes:
- Adds scratch-size selection and validation for snapshots.
- Fixes transport placement and bounds guest allocation below it.
- Updates snapshot ABI, tests, and documentation.
| File | Description |
|---|---|
| src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs | Selects v7.0 goldens. |
| src/hyperlight_host/src/sandbox/snapshot/tripwires.rs | Pins ABI 7. |
| src/hyperlight_host/src/sandbox/snapshot/mod.rs | Builds snapshots with target layouts. |
| src/hyperlight_host/src/sandbox/snapshot/file/mod.rs | Validates host-page alignment. |
| src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs | Sets ABI 7. |
| src/hyperlight_host/src/sandbox/snapshot/file_tests.rs | Tests alignment and ABI rejection. |
| src/hyperlight_host/src/sandbox/initialized.rs | Adds resizing API and runtime tests. |
| src/hyperlight_host/src/mem/virtq/tests.rs | Updates transport address tests. |
| src/hyperlight_host/src/mem/virtq/mod.rs | Supports resized transport snapshots. |
| src/hyperlight_host/src/mem/mgr.rs | Passes target layouts and mailbox offsets. |
| src/hyperlight_host/src/mem/layout.rs | Defines fixed transport placement and sizing. |
| src/hyperlight_guest/src/transport/mem.rs | Bounds memory access to transport ranges. |
| src/hyperlight_guest/src/transport/context.rs | Uses scratch-independent transport bounds. |
| src/hyperlight_guest/src/layout.rs | Defines the allocator’s transport boundary. |
| src/hyperlight_guest/src/arch/amd64/prim_alloc.rs | Applies the allocator boundary. |
| src/hyperlight_guest/src/arch/aarch64/prim_alloc.rs | Applies the allocator boundary. |
| src/hyperlight_common/src/layout.rs | Defines transport placement helpers. |
| docs/virtio-host-guest-communication.md | Documents transport placement. |
| docs/snapshot-versioning.md | Documents ABI 7 compatibility. |
| docs/paging-development-notes.md | Explains resizing and allocation limits. |
| CHANGELOG.md | Records the API and breaking changes. |
Review effort: Balanced
Signed-off-by: danbugs <danilochiarlone@gmail.com>
Benchmark ResultsMeasured commit: kvm / amd (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (❌ 2.28x)Top regressions
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (➖ stable)No benchmark improved or regressed. Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sandbox::snapshot_with_scratch_sizetakes a snapshot that restores with a smaller scratch region. A guest boots with the scratch its initialisation needs and runs guest function calls from a snapshot with only what they need, so each restore resets less memory. Follows the discussion on #1901.snapshot_with_scratch_sizerejects a size that is larger, too small for the transport, or not whole host pages, before the guest is entered. The snapshot loader refuses a scratch size the host cannot map.