Repository navigation
[fence 3/9] libsql-server: fence controller, WAL write gate and positive source drain - #43
Closed
tszymczyszyn-shopify wants to merge 6 commits into
Closed
tszymczyszyn-shopify wants to merge 6 commits into
tszymczyszyn-shopify wants to merge 6 commits into
Conversation
This was referenced Oct 5, 2026
tszymczyszyn-shopify
force-pushed
the
namespace-fence/2-write-gate
branch
from
October 13, 2026 19:41
eb21592 to
8b1c0d9
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/1-model-state
branch
from
October 13, 2026 19:41
872576b to
2e91ab3
Compare
…t connection Add the in-memory authority for namespace fences: - `FenceRegistry`, held by `NamespaceStore` outside the namespace cache and seeded from `MetaStore::load_fences()` before anything is served, so an evicted and reloaded namespace gets the controller it had. Namespaces without fence state get an UNFENCED controller on first load; deleting a namespace drops its controller. - `FenceController`: a per-namespace transition lock and a `watch` gate (`GateSnapshot`: the durable fence, a write generation and an indeterminate flag). Commands commit in the metastore, are published to the gate and only then answered, on their own task so a lost response does not lose the publication. An error before COMMIT leaves the gate unchanged; a failed COMMIT closes the gate and refuses other commands with FENCE_COMMIT_INDETERMINATE until the same command is replayed. - `FenceConnState`, bound to the controller for every connection a `MakeLegacyConnection` opens, starting with its held connection, and shared with the connection's WAL wrapper. The checks that use it land in the next commit. - `cfg(test)` `FenceTestHooks` with the named hook points of the design. `NamespaceStore::with` and `make_namespace` refuse a namespace whose fence state is unavailable before any setup. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…tion Make ManagedConnectionWalWrapper::begin_write_txn the authoritative namespace fence check. Before queueing for the write slot it requires the live gate to admit the connection's operation class and the program and its read transaction to have been admitted under the gate's current write generation. A refusal returns SQLITE_AUTH (not BUSY, so SQLite does not retry it, and before acquire(), so no slot is released that was never held) and leaves the typed outcome in the connection's FenceConnState. - FenceConnState gains begin_program, begin_read_txn and admit_write. CoreConnection::run, every with_raw call and vacuum_if_needed start a program; the WAL wrapper records the generation of each new read transaction before its snapshot is taken. - The Vm refuses Write and DDL statements early against the live gate and reports a WAL refusal as Error::NamespaceFence instead of SQLITE_AUTH. A plain SQLITE_AUTH from an authorizer is left unchanged. - New detail stale_transaction on MIGRATION_WRITE_FENCED for a transaction or program that began under an earlier generation. - Namespaces without a fence stay at generation 0 and behave as before. Tests cover a program parked between admission and its write while the fence is acquired and released, read-to-write upgrades, DDL, a header pragma, BEGIN IMMEDIATE, VACUUM and raw writes, stale transactions after release, and unchanged behaviour of unfenced namespaces. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Queue entries and the write slot now carry the operation class. A write transaction waiting for the slot re-checks the connection's fence admission every time it takes the manager's lock, and the fence controller wakes every registered write queue after each change of the write generation, so a writer queued before a fence leaves the queue with MIGRATION_WRITE_FENCED instead of waiting for the slot and then writing. Checkpoints ask for the slot as maintenance: they are never refused and queue again when woken. The manager exposes what the positive write drain needs: the active writer and its class, a notification on every release, and abort_active(), which uses the registered rollback handle. Abort no longer panics when the connection has already closed. VACUUM is skipped, and reported as skipped rather than failed, while the fence denies normal writes, including when the fence closes between the check and the statement. TRUNCATE checkpoints run in every state. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
AcquireSourceWriteFence now runs the drain of docs/NAMESPACE_FENCE.md section 8.3 end to end, under the namespace's transition lock and on a task of its own: - an in-memory INSTALLING gate closes write admission (and moves the write generation, waking queued writers) before SOURCE_DRAINING is persisted; a command proven not to have committed removes it again; - the drain waits on the connection manager's release notification for the writer that held the slot when admission closed, never on elapsed time or the transaction timeout; at the deadline it answers DRAINING (admission stays closed) or, with force_rollback, rolls the writer back and waits for the actual release; - the frozen boundary (log id, last committed frame) is read under the write-slot lock once no writer holds it, and SOURCE_WRITE_FENCED is committed with it; - replaying a DRAINING command resumes the same drain. Primary connection makers register a write-drain source (their connection manager, held weakly, and their replication log) with the namespace's controller; NamespaceStore::execute_fence_command loads the namespace before an acquisition so that the source exists. The default drain deadline is --namespace-fence-default-write-drain-ms (30 s). FrozenBoundary.frame_no becomes optional, for a log without frames. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…ach boundary Add crash-restart tests for the namespace fence: each server lifetime runs on its own runtime and is ended without any shutdown code while a fence command is parked at a hook point, so the next start takes the real dirty-recovery path on the same directory. They cover every persistence boundary of AcquireSourceWriteFence and ReleaseSourceWriteFence (including a marker that lags the metastore commit), a restart in SOURCE_DRAINING with a writer active at the crash, indeterminate commits through the drain path (applied and not applied), and lost acquisition responses resolved by replay and inspection. The tests exposed that a source restarted while draining could never finish its drain: dirty recovery rebuilds the replication log under a new log id, and completing the drain refused a boundary on a log other than the one the fence was acquired against, leaving the namespace in SOURCE_DRAINING for good. The frozen boundary now names the log that is live when the drain is proven, the record's identity keeps the acquisition log id, and the server warns when the two differ. Write admission was durably closed throughout, so the data at the boundary is unchanged. The BeforeMetastoreCommit test hook can now report a commit as indeterminate without running it. The contract document describes the restart and log rebuild semantics. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…e registers Every connection maker registers its connection manager's write queue and drain source with the namespace's fence controller, and controllers outlive cache eviction. Dead entries were only dropped on a write generation change or during a drain, so a namespace that was never fenced kept one waker and one drain source per eviction and reload. Drop the entries whose manager is gone whenever a new one registers. The waker can now tell whether its manager is alive without waking it.
tszymczyszyn-shopify
force-pushed
the
namespace-fence/1-model-state
branch
from
October 13, 2026 20:51
2e91ab3 to
e87d1d1
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/2-write-gate
branch
from
October 13, 2026 20:51
8b1c0d9 to
0af6adc
Compare
Author
|
Closing. This stack is parked; the note at the top of the description explains why. The branch is kept at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the per-namespace fence registry and controller, installed before a namespace's first connection so eviction and reload reinstall the same gate.
Gates every write transaction in
ManagedConnectionWalWrapper::begin_write_txnby admission generation, before the writer slot is acquired, so a program or transaction admitted before a fence change cannot write after it. Queued writers are classed and woken to re-check the gate;VACUUMbecomes its own operation class.Implements the positive source write drain (close admission, wait for or force-roll-back the active writer, capture the frozen boundary) and handles indeterminate commits, restart at every persistence boundary, and response loss.
Review focus
This is the piece that changes code on every connection's hot path, fence or no fence: the WAL wrapper, the writer queue and
CoreConnection. It deserves the most careful review and possibly a benchmark.Since the first version, from review: write queues and drain sources whose connection manager is gone are dropped whenever a new one registers, so a never-fenced namespace that is evicted and reloaded does not accumulate them.
Commits
Stack
Part 3 of 9, based on
namespace-fence/1-model-state. The stack sits on #53 (tszymczyszyn/atomic-namespace-create), which sits on #51: review and land #51 → #53 → #41 → … → #49 bottom-up, restacking after each squash or rebase merge.The first version of these PRs carried #35's fence diff unchanged on
v0.9.30-shopify-patches. On 2026-10-13 the stack was rebased onto #53 (how the conflicts were resolved is summarised in a comment on #41) and gained the commits marked above.