Layered/incremental snapshots by ludfjig · Pull Request #1926 · hyperlight-dev/hyperlight

ludfjig · 2026-10-12T20:00:48Z

Summary

Snapshots share immutable memory layers. A capture copies only the pages changed since its parent snapshot, and Sandbox::snapshot switches the sandbox to the capture so the next one is incremental too. This makes snapshot() much faster for larger sandboxes. It also improves memory density: each snapshot holds only its changed pages and shares the rest with its parent, where main holds a full copy of guest memory per snapshot. See HIP 0003.

Deviations from the merged HIP 0003 draft (the HIP is updated):

The VM maps each layer whole, one mapping per layer. A layer stays in a snapshot while it gives at least one page. Its pages that a newer layer replaced stay mapped, so a guest can read data it overwrote before a snapshot from the older layers of that snapshot.
Each blob holds guest data followed by the page tables of the snapshot that created it, as on main. A restore copies the newest page tables into scratch.
Sandbox::snapshot installs the capture, because otherwise every later capture copies all pages changed since the last restore again. Installing removes map_region and map_file_cow regions, because the snapshot holds their mapped pages and later blobs may reuse their addresses.
Snapshots from earlier versions are rejected instead of loaded as one layer. This is simpler, and there is little reason to keep backwards compatibility here.

Changes

Snapshot memory is a list of immutable shared blobs, one per layer, each mapped whole.
Capture keeps parent layers that still give a page and copies changed pages into one new blob.
Sandbox::snapshot installs the capture into the running sandbox. Scratch pages it frees are zeroed.
The OCI format stores ordered data layers and a transport layer (config v4, ABI 7). The config names the layer whose page tables a restore uses.
Snapshot performance: adjacent page-table walk mappings are merged, and page-table pages are cached during capture.
KVM: KVM_X86_QUIRK_SLOT_ZAP_ALL is disabled where supported, so deleting a memory slot flushes only that slot.
New benchmarks snapshots/call_and_create and snapshots/restore_alternate cover the cost moved into the next call and restores to a non-current snapshot.

Breaking changes

Sandbox::snapshot removes map_region and map_file_cow regions. Pages of these regions the guest has not mapped are lost.
PtRootFinder receives a SnapshotMemoryReader and returns Result<Vec<u64>>.
A snapshot holds at most 30 memory layers, and a capture fails when it would exceed this. This bounds restore cost by limiting the number of hypercalls to map memory.

Commits

Add call_and_create and restore_alternate snapshot benchmarks. Lands first so the benchmarks run before and after.
Implement layered and incremental snapshots.
Install captured snapshots into the sandbox.
vmem: merge adjacent mappings in walk_va_spaces.
Cache page-table pages in PageTableReader.
Disable KVM_X86_QUIRK_SLOT_ZAP_ALL.

Benchmarks

Compared to main. restore, restore_alternate and call_and_create compare to the first commit, which adds the last two.

MSHV: Linux 6.6 mshv.
KVM: Linux 6.17.

Highlights

Benchmark	MSHV	KVM
`snapshots/create/default`	440 µs → 249 µs (−43%)	380 µs → 262 µs (−31%)
`snapshots/create/large`	171 ms → 1.68 ms (−99%)	162 ms → 1.7 ms (−99%)
`snapshots/call_and_create/medium`	38.1 ms → 726 µs (−98%)	35 ms → 778 µs (−98%)
`snapshots/restore_alternate/large`	4.41 ms → 106 µs (−98%)	1.66 ms → 131 µs (−92%)
`snapshots/restore/large`	1.2 ms → 91.6 µs (−92%)	403 µs → 52 µs (−87%)

Guest calls show no consistent change.

Regressions

Benchmark	MSHV	KVM	Cause
`snapshot_files/load_snapshot_unverified/default`	88.5 µs → 112 µs (+27%)	93.3 µs → 123 µs (+31%)	Opens and maps 2 data blobs instead of 1
`snapshot_files/load_snapshot/default`	617 µs → 701 µs (+14%)	613 µs → 701 µs (+14%)	Also hashes 88 KiB more: base layer pages that the newer layer replaced, and the base layer's page tables
`snapshot_files/save_snapshot/default`	853 µs → 941 µs (+10%)	35.5 ms → 43.4 ms (+22%)	Writes, hashes and fsyncs 2 data blobs and 88 KiB more. fsync is slow on the KVM host
`snapshot_files/cold_start_via_snapshot/default`	1.8 ms → 1.97 ms (+9%)	2.59 ms → 2.85 ms (+10%)	Includes `load_snapshot`
`sandboxes/sandbox_from_snapshot/small`	823 µs → 812 µs (~)	877 µs → 978 µs (+12%)	Maps 2 layers instead of 1. Each costs a memory slot on KVM and a map call on MSHV. MSHV shows it only at `default` (+11%)

Kernels before 6.12 cannot disable KVM_X86_QUIRK_SLOT_ZAP_ALL, so each memory slot change flushes all guest mappings. WHP and aarch64 were not benchmarked.

A saved snapshot stores each layer in full. Base layer pages that a newer layer replaced stay in the file, which adds 88 KiB for default.

Full results

All benchmarks on both hosts (click to expand)

Before → after (change). Negative is faster. Bold is a regression. ~ is no consistent change. A change needs agreement between a 0.5 s and a 3 s run, or a 3 s run alone.

snapshots

Benchmark	Size	MSHV	KVM
`call_and_create`	default	556 µs → 372 µs (−33%)	471 µs → 409 µs (−13%)
`call_and_create`	small	2.67 ms → 416 µs (−84%)	2.65 ms → 466 µs (−82%)
`call_and_create`	medium	38.1 ms → 726 µs (−98%)	35 ms → 778 µs (−98%)
`call_and_create`	large	172 ms → 1.78 ms (−99%)	161 ms → 1.85 ms (−99%)
`create`	default	440 µs → 249 µs (−43%)	380 µs → 262 µs (−31%)
`create`	small	2.57 ms → 291 µs (−89%)	2.47 ms → 324 µs (−87%)
`create`	medium	41.4 ms → 606 µs (−99%)	36.9 ms → 606 µs (−98%)
`create`	large	171 ms → 1.68 ms (−99%)	162 ms → 1.7 ms (−99%)
`restore`	default	64.9 µs → 65.7 µs (~)	20.6 µs → 21 µs (~)
`restore`	small	66.3 µs → 67 µs (~)	21.4 µs → 21.5 µs (~)
`restore`	medium	74.5 µs → 71.1 µs (−5%)	26.4 µs → 25.7 µs (~)
`restore`	large	1.2 ms → 91.6 µs (−92%)	403 µs → 52 µs (−87%)
`restore_alternate`	default	164 µs → 78.7 µs (−52%)	248 µs → 96.5 µs (−61%)
`restore_alternate`	small	270 µs → 79.7 µs (−71%)	268 µs → 93 µs (−65%)
`restore_alternate`	medium	901 µs → 83.4 µs (−91%)	255 µs → 103 µs (−60%)
`restore_alternate`	large	4.41 ms → 106 µs (−98%)	1.66 ms → 131 µs (−92%)

guest_calls

Benchmark	Size	MSHV	KVM
`call`	default	45.8 µs → 45.3 µs (~)	18.4 µs → 17.3 µs (~)
`call`	small	46.2 µs → 43.8 µs (−5%)	18.5 µs → 17.7 µs (~)
`call`	medium	47.3 µs → 43.8 µs (~)	19.8 µs → 17.3 µs (~)
`call`	large	45.4 µs → 44 µs (~)	18.5 µs → 18 µs (~)
`call_with_host_function`	default	77.5 µs → 75.2 µs (~)	32.9 µs → 34.5 µs (~)
`call_with_host_function`	small	75.4 µs → 74.1 µs (~)	32.8 µs → 34.7 µs (~)
`call_with_host_function`	medium	74 µs → 75.2 µs (~)	34.8 µs → 32.8 µs (~)
`call_with_host_function`	large	73.5 µs → 75.2 µs (~)	33.5 µs → 32.7 µs (~)
`call_with_restore`	default	140 µs → 133 µs (~)	64 µs → 62.9 µs (~)
`call_with_restore`	small	141 µs → 140 µs (~)	64.4 µs → 63.8 µs (~)
`call_with_restore`	medium	147 µs → 138 µs (~)	70.4 µs → 67.9 µs (~)
`call_with_restore`	large	201 µs → 166 µs (~)	102 µs → 89.1 µs (~)
`different_thread`	-	75.5 µs → 74 µs (~)	42.4 µs → 42.9 µs (~)
`interrupt_latency`	-	47.8 µs → 48.9 µs (~)	32.8 µs → 35.9 µs (~)

guest_functions_with_large_parameters

Benchmark	Size	MSHV	KVM
`guest_call_with_large_parameters`	-	164 ms → 163 ms (~)	872 ms → 916 ms (~)

sample_workloads

Benchmark	Size	MSHV	KVM
`24K_in_8K_out_c`	-	56.3 µs → 56.5 µs (~)	26.9 µs → 27.6 µs (~)
`24K_in_8K_out_rust`	-	54.6 µs → 56.1 µs (~)	26.6 µs → 27.9 µs (~)

sandboxes

Benchmark	Size	MSHV	KVM
`create_initialized`	default	1.59 ms → 1.72 ms (~)	2.62 ms → 2.8 ms (~)
`create_initialized`	small	4.44 ms → 4.35 ms (~)	4.72 ms → 4.58 ms (~)
`create_initialized`	medium	11.8 ms → 11.8 ms (~)	9.9 ms → 9.81 ms (~)
`create_initialized`	large	37.8 ms → 38.2 ms (~)	27.6 ms → 27.5 ms (~)
`create_initialized_and_drop`	default	22.4 ms → 21.4 ms (~)	10.9 ms → 14.1 ms (~)
`create_initialized_and_drop`	small	29.4 ms → 28.9 ms (~)	14.8 ms → 13.3 ms (~)
`create_initialized_and_drop`	medium	31.9 ms → 32.2 ms (~)	19.9 ms → 20.7 ms (~)
`create_initialized_and_drop`	large	58.8 ms → 57.9 ms (~)	41.7 ms → 39.6 ms (~)
`sandbox_from_snapshot`	default	599 µs → 665 µs (+11%)	865 µs → 971 µs (~)
`sandbox_from_snapshot`	small	823 µs → 812 µs (~)	877 µs → 978 µs (+12%)
`sandbox_from_snapshot`	medium	2.17 ms → 2.02 ms (~)	975 µs → 1.01 ms (~)
`sandbox_from_snapshot`	large	6.29 ms → 6.06 ms (~)	1.06 ms → 1.22 ms (+15%)

snapshot_files

Benchmark	Size	MSHV	KVM
`cold_start_via_evolve`	default	1.67 ms → 1.61 ms (~)	3.05 ms → 2.96 ms (~)
`cold_start_via_evolve`	small	4.74 ms → 4.55 ms (~)	5.12 ms → 5.76 ms (~)
`cold_start_via_evolve`	medium	12 ms → 12.3 ms (~)	10.3 ms → 11 ms (~)
`cold_start_via_evolve`	large	37.6 ms → 38.8 ms (~)	27.9 ms → 29.1 ms (~)
`cold_start_via_snapshot`	default	1.8 ms → 1.97 ms (+9%)	2.59 ms → 2.85 ms (+10%)
`cold_start_via_snapshot`	small	9.14 ms → 8.93 ms (~)	8.55 ms → 8.95 ms (~)
`cold_start_via_snapshot`	medium	54.7 ms → 53.8 ms (~)	49.8 ms → 50.9 ms (~)
`cold_start_via_snapshot`	large	204 ms → 205 ms (~)	193 ms → 194 ms (~)
`cold_start_via_snapshot_unverified`	default	1.31 ms → 1.4 ms (~)	2.06 ms → 2.2 ms (~)
`cold_start_via_snapshot_unverified`	small	2.18 ms → 2.08 ms (−4%)	2.03 ms → 2.17 ms (~)
`cold_start_via_snapshot_unverified`	medium	3.68 ms → 3.59 ms (~)	2.2 ms → 2.25 ms (~)
`cold_start_via_snapshot_unverified`	large	7.7 ms → 7.73 ms (~)	2.33 ms → 2.59 ms (~)
`load_snapshot`	default	617 µs → 701 µs (+14%)	613 µs → 701 µs (+14%)
`load_snapshot`	small	6.65 ms → 6.71 ms (~)	6.49 ms → 6.68 ms (~)
`load_snapshot`	medium	50.3 ms → 50.4 ms (~)	48.1 ms → 48.8 ms (~)
`load_snapshot`	large	197 ms → 198 ms (~)	190 ms → 192 ms (~)
`load_snapshot_unverified`	default	88.5 µs → 112 µs (+27%)	93.3 µs → 123 µs (+31%)
`load_snapshot_unverified`	small	89.6 µs → 113 µs (+26%)	95.1 µs → 119 µs (+25%)
`load_snapshot_unverified`	medium	90.4 µs → 114 µs (+26%)	96.3 µs → 120 µs (+24%)
`load_snapshot_unverified`	large	94 µs → 118 µs (+26%)	99 µs → 125 µs (+27%)
`save_snapshot`	default	853 µs → 941 µs (+10%)	35.5 ms → 43.4 ms (+22%)
`save_snapshot`	small	8.97 ms → 9.09 ms (~)	55.2 ms → 61.5 ms (~)
`save_snapshot`	medium	68.2 ms → 69.6 ms (~)	462 ms → 402 ms (~)
`save_snapshot`	large	270 ms → 271 ms (~)	2.03 s → 1.75 s (~)

Copilot

🔵 Needs a closer look

The VMM memory boundary changes substantially, and OCI loading maps oversized blobs before enforcing resource limits.

1 open finding

Preflight descriptor sizes before mapping to prevent excessive reservations · New

What changed in this PR

Adds layered incremental snapshots that share immutable memory, persist OCI data and page-table layers, and optimize restore mappings.

Changes:

Adds shared snapshot layers with incremental capture and installation.
Introduces snapshot ABI 6 and OCI config v4.
Optimizes page-table walks, KVM invalidation, tests, and benchmarks.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Adds zero-init mapping test support.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Bumps golden snapshots to v6.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Documents capture behavior and uses GPA spans.
`src/hyperlight_host/src/sandbox/trace/mem_profile.rs`	Reads layered memory by GPA.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pins ABI 6 media types.
`src/hyperlight_host/src/sandbox/snapshot/memory/mod.rs`	Defines layered snapshot memory.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Maps and accesses snapshot layers.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Saves and loads layered OCI snapshots.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Adds config v4 and new layer media types.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Defines the layered snapshot schema.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Tests layered persistence and validation.
`src/hyperlight_host/src/sandbox/mod.rs`	Exports `SnapshotMemoryReader`.
`src/hyperlight_host/src/sandbox/initialized.rs`	Installs captures and updates snapshot APIs.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Updates scratch transport tests.
`src/hyperlight_host/src/mem/shared_mem.rs`	Adds immutable memory and range mappings.
`src/hyperlight_host/src/mem/layout.rs`	Removes flat snapshot layout state.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Disables the slot-wide KVM flush quirk.
`src/hyperlight_host/src/hypervisor/virtual_machine/hvf/mod.rs`	Uses range-based snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/test_support.rs`	Updates mapping-state fault tests.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Manages multiple snapshot mappings.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Adopts layered VM memory.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Resolves layered debug memory.
`src/hyperlight_host/build.rs`	Removes obsolete memory-read configuration.
`src/hyperlight_host/benches/benchmarks.rs`	Adds incremental snapshot benchmarks.
`src/hyperlight_common/src/vmem.rs`	Coalesces adjacent mappings.
`src/hyperlight_common/src/arch/amd64/vmem.rs`	Applies mapping coalescing on AMD64.
`src/hyperlight_common/src/arch/aarch64/vmem.rs`	Applies mapping coalescing on AArch64.
`proposals/0003-incremental-snapshots/README.md`	Updates the snapshot design.
`docs/snapshot-versioning.md`	Documents ABI 6 versioning.
`docs/snapshot-oci-format.md`	Documents layered OCI storage.
`docs/paging-development-notes.md`	Describes incremental capture behavior.
`CHANGELOG.md`	Records breaking API and format changes.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        let mut snap_files = data_descs
+            .iter()
+            .chain([pt_desc])
+            .map(|desc| open_snapshot_blob(&blobs_dir, desc))
+            .collect::<crate::Result<Vec<_>>>()?;


call_and_create times a guest call plus snapshot creation, so the call pays for memory remapping done by the previous snapshot. restore_alternate restores two snapshots in turn, so each restore targets memory that is not current. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Share immutable data blobs between related snapshots. Each blob holds guest data followed by the page tables of the snapshot that created it. Persist layered snapshots as OCI image layouts. Capture copies only pages that changed since the snapshot the sandbox was created from or last restored to. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Snapshot capture switches the running sandbox to the captured snapshot. The next capture reuses its layers and copies only pages changed after it. Scratch pages used before the capture become free again. They are zeroed so guest allocations return zeroed pages, as after restore. Capture removes map_region and map_file_cow regions. Pages of these regions that the guest has not mapped are lost. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

walk_va_spaces merges mappings that have contiguous physical and virtual addresses and the same kind. Snapshot capture classifies and rebuilds each merged mapping once. Criterion snapshots/create: * MSHV: medium 1.18 ms to 950 us (-20%), large 3.56 ms to 2.85 ms (-20%) * KVM: medium 1.34 ms to 1.04 ms (-22%), large 3.59 ms to 2.91 ms (-19%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

A page-table walk reads every entry through GuestPhysicalMemoryView, which resolves the backing and copies 8 bytes through a locked volatile copy. The reader keeps a copy of the last table page it read, so each table costs one copy. Snapshot capture and crash dumps use the reader. Criterion snapshots/create: * MSHV: default 344 us to 248 us (-28%), medium 950 us to 594 us (-37%) * KVM: default 499 us to 379 us (-24%), medium 1.04 ms to 746 us (-28%) Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

By default, deleting a memslot flushes the guest mappings of every memslot. KVM keeps that flush as a workaround for VMs with assigned GPUs. Hyperlight assigns no devices and disables the quirk when the kernel supports it (Linux 6.12 and later). Deleting a memslot then flushes only that memslot's mappings. Snapshot capture deletes a memslot when a layer drops out of the snapshot. Restore to another snapshot and region unmapping also delete memslots. Criterion on KVM (Linux 6.17): * snapshots/call_and_create: 24% to 58% faster * snapshots/restore_alternate: 32% to 52% faster Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-10-13T23:43:17Z

Benchmark Results

Measured commit: 67561dc342b8
Baseline commit: ad05110e17bf

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`826.77 ns` (➖ 1.09x slower)
`vec_bytes`	`591.63 ns` (➖ 1.01x slower)
		`371.97 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`522.33 ns` (➖ 1.00x faster)
`65536`	`142.24 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`79.43 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`7.78 ns` (➖ 1.01x slower)	`7.72 ns` (➖ 1.00x faster)	`7.80 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`117.36 µs` (➖ 1.27x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.10 µs` (➖ 1.00x faster)	`7.23 µs` (➖ 1.05x slower)
`65536`	`2.07 µs` (➖ 1.07x slower)	`1.93 µs` (➖ 1.09x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`6.30 µs` (➖ 1.03x faster)	`6.37 µs` (➖ 1.01x slower)
`8192`	`1.09 µs` (➖ 1.02x faster)	`1.13 µs` (➖ 1.05x slower)
`262144`		`27.57 µs` (➖ 1.01x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`710.53 ns` (➖ 1.00x faster)
`vec_bytes`	`537.19 ns` (➖ 1.02x faster)
		`695.30 µs` (➖ 1.04x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`498.38 ns` (➖ 1.21x faster)
`65536`	`135.52 ns` (➖ 1.13x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`77.79 ms` (➖ 1.10x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`6.89 ns` (➖ 1.21x faster)	`6.89 ns` (➖ 1.08x faster)	`6.89 ns` (➖ 1.18x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`60.74 µs` (➖ 1.29x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.71 µs` (➖ 1.09x faster)	`7.68 µs` (➖ 1.08x faster)
`65536`	`2.17 µs` (➖ 1.12x faster)	`2.11 µs` (➖ 1.09x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.43 µs` (➖ 1.04x faster)	`7.32 µs` (➖ 1.10x faster)
`8192`	`859.85 ns` (➖ 1.00x slower)	`746.65 ns` (➖ 1.11x faster)
`262144`		`31.99 µs` (➖ 1.00x slower)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`927.54 ns` (➖ 1.04x faster)
`vec_bytes`	`704.27 ns` (➖ 1.08x faster)
		`332.50 µs` (➖ 1.11x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`718.94 ns` (➖ 1.00x slower)
`65536`	`190.46 ns` (➖ 1.02x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`58.68 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.41 ns` (➖ 1.00x slower)	`9.57 ns` (➖ 1.00x slower)	`9.53 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`107.94 µs` (➖ 1.29x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`10.71 µs` (➖ 1.02x slower)	`9.80 µs` (➖ 1.01x faster)
`65536`	`2.43 µs` (➖ 1.02x faster)	`2.23 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.89 µs` (➖ 1.10x faster)	`8.42 µs` (➖ 1.17x faster)
`8192`	`1.43 µs` (➖ 1.02x slower)	`1.28 µs` (➖ 1.02x slower)
`262144`		`36.51 µs` (➖ 1.09x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`933.56 ns` (➖ 1.02x slower)
`vec_bytes`	`665.35 ns` (➖ 1.00x slower)
		`888.32 µs` (➖ 1.08x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`648.59 ns` (➖ 1.00x slower)
`65536`	`168.01 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`75.52 ms` (➖ 1.10x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`9.59 ns` (➖ 1.05x faster)	`9.51 ns` (➖ 1.03x faster)	`9.15 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`63.98 µs` (➖ 1.28x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.02 µs` (➖ 1.01x slower)	`8.16 µs` (➖ 1.03x slower)
`65536`	`2.31 µs` (➖ 1.02x slower)	`2.27 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`7.69 µs` (➖ 1.01x faster)	`7.55 µs` (➖ 1.02x faster)
`8192`	`936.03 ns` (➖ 1.02x faster)	`940.76 ns` (➖ 1.02x slower)
`262144`		`39.03 µs` (➖ 1.03x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.03x slower)
`vec_bytes`	`760.23 ns` (➖ 1.14x faster)
		`1.84 ms` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`782.63 ns` (➖ 1.03x faster)
`65536`	`229.16 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`95.97 ms` (➖ 1.18x slower)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`11.01 ns` (➖ 1.09x faster)	`11.01 ns` (➖ 1.00x faster)	`10.89 ns` (➖ 1.02x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`657.83 µs` (➖ 1.08x faster)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`8.71 µs` (➖ 1.01x faster)	`8.57 µs` (➖ 1.01x faster)
`65536`	`2.58 µs` (➖ 1.00x slower)	`2.55 µs` (➖ 1.00x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`8.87 µs` (➖ 1.01x slower)	`8.74 µs` (➖ 1.01x slower)
`8192`	`1.40 µs` (➖ 1.02x faster)	`1.45 µs` (➖ 1.02x faster)
`262144`		`37.84 µs` (➖ 1.03x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.20 µs` (➖ 1.04x faster)
`vec_bytes`	`782.66 ns` (➖ 1.00x slower)
		`2.86 ms` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`731.84 ns` (➖ 1.01x faster)
`65536`	`246.70 ns` (➖ 1.18x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`105.09 ms` (➖ 1.09x faster)

slot_pool

	`alloc_dealloc_1500`	`alloc_dealloc_4096`	`alloc_dealloc_128`
	`10.38 ns` (➖ 1.04x slower)	`10.11 ns` (➖ 1.01x slower)	`10.20 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`677.18 µs` (➖ 1.33x slower)

virtq_readonly

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`262144`	`7.79 µs` (➖ 1.02x slower)	`8.05 µs` (➖ 1.04x slower)
`65536`	`2.34 µs` (➖ 1.01x slower)	`2.34 µs` (➖ 1.00x faster)

virtq_readwrite

	`slot_pool_segmented_fragmented`	`slot_pool_segmented`
`65536`	`9.06 µs` (➖ 1.05x slower)	`8.57 µs` (➖ 1.00x faster)
`8192`	`1.13 µs` (➖ 1.12x faster)	`1.11 µs` (➖ 1.00x faster)
`262144`		`41.35 µs` (➖ 1.20x faster)

_{Reported by cargo ci bench-report --candidate run:38481552259 --config-file bench_report.toml.}

andreiltd · 2026-10-14T11:29:57Z

    unsafe {
        internal_walk_va_spaces(&op, roots.iter().cloned(), true, address, len)
-            .map(|(id, mappings)| (id, mappings.collect::<Vec<_>>()))
+            .map(|(id, mappings)| {


It's hard to understand what happens here, but IIUC this is just coalescing regions? If so, I think we could use: coalesce from itertools: https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.coalesce.

coalesce() expects Ok(merged) or Err((previous, next)), so we could have for example:

impl SpaceAwareMapping { pub(crate) fn try_merge(mut self, other: Self) -> Result<Self, (Self, Self)>{} }

This would then read like this:

map(|(id, mappings)| (id, mappings.coalesce(SpaceAwareMapping::try_merge).collect()))

It looks like coalescing when walking is quite common and that could be reused in the guest code as well, see :

hyperlight/src/hyperlight_guest/src/transport/backing.rs

Lines 131 to 154 in 67561dc

/// Yield coalesced aliases for the same pages checked by [`validate_source_pages`].

fn coalesce_mappings(

source: u64,

start: u64,

pages: impl Iterator<Item = Mapping>,

) -> impl Iterator<Item = Mapping> {

let runs = pages.coalesce(|mut run, page| {

if run.phys_base.checked_add(run.len) != Some(page.phys_base) {

return Err((run, page));

}

run.len += page.len;

Ok(run)

});

runs.map(move |run| Mapping {

virt_base: start + (run.virt_base - source),

kind: MappingKind::Basic(PERMISSIONS),

..run

})

}

/// Update owned alias entries and invalidate cleared translations.

///

andreiltd · 2026-10-14T11:54:35Z

-            self.vm.unmap_memory((self.snapshot_slot, &old_rgn))?;
+        let target_mappings = snapshot.mappings();
+
+        let mut index = self.snapshot_mappings.len();


Suggested change

let mut index = self.snapshot_mappings.len();

for index in (0..self.snapshot_mappings.len()).rev() {

let (slot, curr) = &self.snapshot_mappings[index];

if target_mappings.contains(curr) {

continue;

}

self.vm.unmap_memory((*slot, curr))?;

let (slot, _) = self.snapshot_mappings.remove(index);

self.freed_slots.push(slot);

}

I think is the same and reads slightly better, but feel free to ignore.

andreiltd · 2026-10-14T12:23:02Z

+        Ok(page_gpa)
+    }
+
+    pub(crate) fn read(&mut self, gva: u64, destination: &mut [u8]) -> Result<()> {


I would call this read_exact or read_exact_at to be somewhat compatible with std. This would make it clear that this "reads the exact number of bytes required to fill buf".

andreiltd · 2026-10-14T12:30:54Z

-            self.shared_mem = hsnapshot;
-            Some(gsnapshot)
-        };
+        let gsnapshot = self.replace_snapshot_memory(snapshot)?;


"g" as global?

andreiltd · 2026-10-14T12:39:40Z

+                    .checked_add(chunk_len)
+                    .ok_or_else(|| new_error!("crash-dump GVA range overflows"))?;
+
+                if !try_coalesce_region(&mut regions, virt_base, virt_end, host_region.start, flags)


I think this could also use the same coalescing pattern:

resolved_regions .coalesce(CrashDumpRegion::try_merge) .collect()

andreiltd · 2026-10-14T12:59:24Z

+
+/// Copies guest data and page tables into one blob. The data is padded to a
+/// host page.
+#[cfg(test)]


Should we keep all those test functions in mod tests and make mod pub(crate)?

andreiltd · 2026-10-14T13:13:35Z

-            layout,
+            memory,
            root,
+            failure: Cell::new(None),


Shouldn't TableReadOps have a fallible variant of read_entry? I guess we could then avoid this dance with finish() and caching failures?

andreiltd · 2026-10-14T13:15:27Z

+    if mapping.kind == MappingKind::Unmapped {
+        return Ok(false);
+    }
+    if mapping.len == 0


Are those checks common enough to have them in Mapping constructor? I think, in general, we should think about making all sort of mapping and pages operations more centralized in the common crate.

ludfjig added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. regen-goldens Regenerate snapshot golden fixtures labels Oct 12, 2026

Copilot AI balanced review requested due to automatic review settings October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested review from danbugs, dblnz and devigned as code owners October 12, 2026 20:00

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Oct 12, 2026

ludfjig requested review from jprendes, jsturtevant and syntactically as code owners October 12, 2026 20:00

ludfjig added the regen-goldens Regenerate snapshot golden fixtures label Oct 12, 2026

ludfjig requested review from andreiltd and simongdavies as code owners October 12, 2026 20:00

ludfjig added the ready-for-review PR is ready for (re-)review label Oct 12, 2026

ludfjig requested a review from squillace as a code owner October 12, 2026 20:00

ludfjig added this pull request to stack #1929 October 12, 2026 20:01

ludfjig mentioned this pull request Oct 12, 2026

Layered/incremental snapshots #1902

Closed

Copilot started reviewing on behalf of ludfjig October 12, 2026 20:05 View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/file/mod.rs

Comment on lines +901 to +905

let mut snap_files = data_descs

.iter()

.chain([pt_desc])

.map(|desc| open_snapshot_blob(&blobs_dir, desc))

.collect::<crate::Result<Vec<_>>>()?;

This comment has been minimized.

Sign in to view

ludfjig force-pushed the ll/1-layered-snapshots branch from 8c92fa0 to 67c5cc8 Compare October 13, 2026 18:56

ludfjig force-pushed the ll/1-layered-snapshots branch from 67c5cc8 to 7ca7b5e Compare October 13, 2026 21:01

This comment has been minimized.

Sign in to view

ludfjig added 5 commits October 13, 2026 15:42

ludfjig force-pushed the ll/1-layered-snapshots branch from 7ca7b5e to 67561dc Compare October 13, 2026 23:00

andreiltd reviewed Oct 14, 2026

View reviewed changes

	/// Yield coalesced aliases for the same pages checked by [`validate_source_pages`].
	fn coalesce_mappings(
	source: u64,
	start: u64,
	pages: impl Iterator<Item = Mapping>,
	) -> impl Iterator<Item = Mapping> {
	let runs = pages.coalesce(\|mut run, page\| {
	if run.phys_base.checked_add(run.len) != Some(page.phys_base) {
	return Err((run, page));
	}

	run.len += page.len;
	Ok(run)
	});

	runs.map(move \|run\| Mapping {
	virt_base: start + (run.virt_base - source),
	kind: MappingKind::Basic(PERMISSIONS),
	..run
	})
	}

	/// Update owned alias entries and invalidate cleared translations.
	///

-        let mut index = self.snapshot_mappings.len();
+    for index in (0..self.snapshot_mappings.len()).rev() {
+        let (slot, curr) = &self.snapshot_mappings[index];
+        if target_mappings.contains(curr) {
+            continue;
+        }
+        self.vm.unmap_memory((*slot, curr))?;
+        let (slot, _) = self.snapshot_mappings.remove(index);
+        self.freed_slots.push(slot);
+    }

Conversation

ludfjig commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Breaking changes

Commits

Benchmarks

Highlights

Regressions

Full results

snapshots

guest_calls

guest_functions_with_large_parameters

sample_workloads

sandboxes

snapshot_files

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Oct 13, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

andreiltd Oct 14, 2026

Choose a reason for hiding this comment

Uh oh!

andreiltd Oct 14, 2026

Choose a reason for hiding this comment

ludfjig commented Oct 12, 2026 •

edited

Loading

andreiltd Oct 14, 2026 •

edited

Loading

andreiltd Oct 14, 2026 •

edited

Loading