Repository navigation
[fence 5/9] libsql-server: migration target lifecycle and import capability - #45
Closed
tszymczyszyn-shopify wants to merge 9 commits into
Closed
tszymczyszyn-shopify wants to merge 9 commits into
tszymczyszyn-shopify wants to merge 9 commits into
Conversation
This was referenced Oct 5, 2026
tszymczyszyn-shopify
force-pushed
the
namespace-fence/3-read-fence
branch
from
October 13, 2026 19:41
bd0052e to
da708e3
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/4-target-lifecycle
branch
from
October 13, 2026 19:41
e3b4ab4 to
5a38436
Compare
Author
|
New in this PR after the rebase onto #53 (summary in #41): the dump importers now run on any connection, so |
load_dump and both importers took a PrimaryConnection. Take any Connection instead, so that an import into a quarantined migration target can run the same importers on its capability connection. The buffered importer shares the connection with its per-statement blocking task through an Arc rather than requiring Clone. No behaviour change.
Add `NamespaceStore::create_target_quarantined` (and route the admin `CreateTargetQuarantined` command through it), which creates a namespace as a quarantined migration target atomically with namespace creation: - A name the server already knows (config in memory, or a namespace cache entry) is refused with FENCE_PRECONDITION_FAILED/namespace_exists without touching its gate. - Otherwise the controller publishes an in-memory target-creation gate before the metastore transaction writes the marker, config row, record and receipt. The gate refuses every class but maintenance and observability, and `check_available` refuses the name, so create, fork and `with()` neither store nor set anything up for it. - The committed record's quarantine gate replaces it; only then is the config published into the in-memory map (from the durable row, with the record's own block values) and the namespace loaded, so its first connection maker is created behind the quarantine gate. - The command runs on its own task under the transition lock; a replay returns the stored result and completes the publication and load of a commit that was not acknowledged, or of a creation interrupted between its marker and its commit. `CreateTargetRequest` is the typed entry point for the admin route and bulk import. The target's log id is not written back into the record, to keep the marker rule for same-revision records intact (documented). Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Add the operation-owned import path into a quarantined migration target and
the seal that ends it (docs/NAMESPACE_FENCE.md sections 7, 10.2 and 11).
- MigrationCapability (server-issued, fields private) and CapabilityPurpose.
The fence controller keeps the live capability set and a count of running
import calls; every published transition drops capabilities whose state,
owner or revision no longer match.
- FenceConnState::with_capability: the WAL admits a capability connection's
write transaction only while its capability matches the fence and is live;
a validation connection never writes.
- NamespaceStore::open_import_session and ImportSession::{with_raw,
load_dump}: the only way to write into TARGET_QUARANTINED. The dump loader
is split into load_dump_sql so that the loader used for namespaces created
from a dump also runs under an import capability.
- SealTargetImport closes import admission in memory, persists
TARGET_IMPORT_DRAINING (invalidating every import capability), waits on
release notifications for running import calls and for any import
transaction holding the write slot (force_rollback rolls it back at the
deadline), then persists TARGET_VALIDATING. A deadline leaves
TARGET_IMPORT_DRAINING durable and closed until the owner's seal resumes it.
Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Keep stale DRAINING receipts historical once their exact drain state and revision have been superseded, and preserve resumed attribution when a live drain completes. This prevents old commands from cancelling or completing work in newer states while keeping replay responses and audit classification accurate. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Walk a source and two targets through every stored fence state and check the protection an older binary gets (docs/NAMESPACE_FENCE.md section 13.2): the config row's block_* fields hold the fence's mirror and nothing else in the row changes, a config write through the metastore is refused and changes nothing, and an older binary's delete of the config row fails on the fence row's foreign key. Release and write enable put the namespace's own values back. The test found that a restart overwrote the in-memory config of a released source or a writable target with the block_* values saved when the fence was acquired, although config writes after the operation had stored the namespace's own values in the row since: a namespace blocked after its release could come back unblocked. Loading the metastore, the target config publication and adoption now take the namespace's own config through fence_store::own_config, which uses the row as it is once the record no longer mirrors the fence. Crash the server at each persistence boundary of SetSourceReadFence, SealTargetImport and EnableTargetWrites, and while a read or seal drain waits for a reader or an import call, then restart it on the same directory: the prior or the committed state is recovered with no in-memory gate, reader or import call left, reads and import stay closed once their draining state committed, target writes open only if TARGET_WRITABLE committed, the marker is repaired, and a replay of the same command completes an interrupted drain at once. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Add a test that the admin shell can neither read nor write a quarantined migration target, through its own entry point and with a raw write that skips its read admission, while the operation's import session keeps working. Map every acceptance requirement in docs/NAMESPACE_FENCE.md section 17 to the tests that cover it, correct the names of the corrupt-state tests, list the transition and CAS tests that cover ownership, revision and replay outcomes, and note the replica connection path in the code-path coverage table. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
name_in_use looked at the metastore and the namespace cache, but a name reserved by a create, fork or reset in progress is pinned in in_flight, and the cache may evict the reserved entry before the creation puts its config in memory. A target created in that window would have its config row deleted when the reservation is undone. Consult in_flight as well.
Load the representative dump through ImportSession::load_dump_with with the streaming importer and compare the sealed target with the source, and check that a streaming import whose caller goes away mid-dump leaves the target empty and quarantined.
tszymczyszyn-shopify
force-pushed
the
namespace-fence/4-target-lifecycle
branch
from
October 13, 2026 20:51
5a38436 to
950c901
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/3-read-fence
branch
from
October 13, 2026 20:51
da708e3 to
10d7857
Compare
Author
|
Closing. This stack is parked; the note at the top of the description explains why. The branch is kept at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creates migration targets in quarantine, issues operation-owned import and validation capabilities, and implements the seal drain, validation, publication (readable, writes fenced) and the irreversible
EnableTargetWrites.open_import_sessioninto a quarantined target is the internal API that bulk import builds on.Also includes, next to the code they cover:
block_*mirror fix and its tests, plus crash-boundary tests for the read fence, seal and write enable;Review focus
Capability checks at the WAL (only the owning operation's import capability can write a quarantined target) and the seal/validate/publish/enable transitions.
Since the rebase onto #53:
ImportSessionloads throughnamespace::dump_import(buffered by default;load_dump_withselects the streaming importer and returns its statistics), and a migration target is refused over a name that a create, fork or reset still holds reserved.Commits
Stack
Part 5 of 9, based on
namespace-fence/3-read-fence. The stack sits on #53 (tszymczyszyn/atomic-namespace-create), which sits on #51: review and land #51 → #53 → #41 → … → #49 bottom-up, restacking after each squash or rebase merge.The first version of these PRs carried #35's fence diff unchanged on
v0.9.30-shopify-patches. On 2026-10-13 the stack was rebased onto #53 (how the conflicts were resolved is summarised in a comment on #41) and gained the commits marked above.