Repository navigation
[fence 4/9] libsql-server: source read fence for SQL, dump and replication streams - #44
Closed
tszymczyszyn-shopify wants to merge 4 commits into
Closed
tszymczyszyn-shopify wants to merge 4 commits into
tszymczyszyn-shopify wants to merge 4 commits into
Conversation
This was referenced Oct 5, 2026
tszymczyszyn-shopify
force-pushed
the
namespace-fence/3-read-fence
branch
from
October 13, 2026 19:41
bd0052e to
da708e3
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/2-write-gate
branch
from
October 13, 2026 19:41
eb21592 to
8b1c0d9
Compare
SetSourceReadFence now closes read admission in memory, persists SOURCE_READ_DRAINING, and waits for every read lease already held before it persists SOURCE_READ_FENCED. Leases are taken only after checking the gate under the controller's lease lock, so once admission is closed the set of leases can only shrink. Each SQL program holds a lease for as long as it runs (including a Hrana cursor still producing rows), as does describe and each admin shell query. ATTACH of a namespace is a read of that namespace: the attaching connection keeps a lease on it for every later program until it is detached. A connection idle inside a transaction holds no lease; its next program is refused with MIGRATION_READ_FENCED and rolled back. /beta/listen is refused where reads are denied and ends when reads are fenced. At the deadline (--namespace-fence-default-read-drain-ms, 30s) running programs are cancelled through the connection's progress-handler cancel flag and report the read fence; the drain still waits for the actual releases and answers DRAINING if they do not come, and a replay of the same command resumes it. ClearSourceReadFence reopens reads with writes still fenced. Dump and replication stream leases follow. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Dump and replication now hold read leases on the namespace fence, so the source read fence drains them positively: - /dump is admitted by the fence gate before a connection is created (a failure to create one is an error, not a panic) and the export holds a dump lease. At the read drain's deadline the export is cancelled before its next row, and a write blocked on a peer that stopped reading fails at once, so the lease is released without the peer. A cancelled dump ends its body with the fence error and never reaches its final COMMIT;. - Replication hello, log_entries, batch_log_entries and snapshot are refused at their start with FAILED_PRECONDITION and x-libsql-fence-code while streams are denied; refusals are counted and logged at most once a minute per namespace. Streams (and the frames of a batch) are served through FencedStream, whose watcher ends the stream as soon as the gate closes or the drain cancels it, dropping the inner stream and the lease itself; the next poll yields the typed terminal status. - With --enable-namespace-fence, the RPC server and the user HTTP server send HTTP/2 keepalive pings (--namespace-fence-keepalive-interval-s, default 30 s, 20 s timeout) so dead peers are detected. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…mpaction timing `snapshot_stream_ends_typed` polled `get_snapshot_file(1)` and unwrapped its result while the compactor was still writing the first snapshot and the snapshot merger could be replacing merged files. The lookup lists the snapshot directory and then opens the chosen file, so under load it could fail with `NotFound` (directory not created yet, or a file removed by a merge between the listing and the open) and the test panicked. Open the stream through the `snapshot` RPC itself in a bounded loop that treats only "snapshot not found" and the vanished-file error as "not yet", and asserts that no read lease is held after a failed attempt. An opened stream holds its file open, so a later merge cannot affect it. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Every SQL program on every namespace took a registered read lease: the lease lock shared by the namespace's connections, an allocation for its cancel closure and a map insert and remove, with the fence off as much as on. Registration is only needed so that the read drain can cancel the work at its deadline. While the gate is at least two transitions away from a read fence (UNFENCED, RELEASED, TARGET_WRITABLE, nothing installing, indeterminate or closing reads), count normal SQL reads in an atomic counter instead. The drain's count includes it, so a read fence still waits for those programs; it cannot cancel them, so one still running after the grace answers DRAINING. Dumps, streams and every other read keep the registered lease. Adds an ignored micro-benchmark: 31 ns against 596 ns per read on a debug build.
tszymczyszyn-shopify
force-pushed
the
namespace-fence/2-write-gate
branch
from
October 13, 2026 20:51
8b1c0d9 to
0af6adc
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/3-read-fence
branch
from
October 13, 2026 20:51
da708e3 to
10d7857
Compare
Author
|
Closing. This stack is parked; the note at the top of the description explains why. The branch is kept at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds read leases on the fence controller and the source read fence: new SQL reads are refused with
MIGRATION_READ_FENCED, open reads are drained and cancelled at the deadline. Covers ATTACH (an attached namespace's fence is checked), the admin shell, and/beta/listen.Dump and replication streams hold read leases too: a fenced dump stops before the next row and before its final
COMMIT;, andlog_entries/snapshotstreams end with a typed status. Adds an optional HTTP/2 keepalive so a dead peer's stream can be detected.Review focus
The lease linearisation (gate checked under the lease lock) and the stream cancellation paths.
Since the first version, from review: the SQL read fast path (section 9.1). While a read fence is at least two transitions away, normal SQL reads are counted in an atomic counter instead of registering a lease; the read drain waits for them but cannot cancel them, so one still running after the grace answers
DRAINING. Dumps, streams and every other read keep the registered lease.read_lease_cost(an ignored test) measures 31 ns against 596 ns per read on a debug build, 139 ns against 1.53 µs across 8 threads.Commits
Stack
Part 4 of 9, based on
namespace-fence/2-write-gate. The stack sits on #53 (tszymczyszyn/atomic-namespace-create), which sits on #51: review and land #51 → #53 → #41 → … → #49 bottom-up, restacking after each squash or rebase merge.The first version of these PRs carried #35's fence diff unchanged on
v0.9.30-shopify-patches. On 2026-10-13 the stack was rebased onto #53 (how the conflicts were resolved is summarised in a comment on #41) and gained the commits marked above.