Repository navigation
[fence 9/9] libsql-server: fence operations: provenance, adoption, metrics - #49
Closed
tszymczyszyn-shopify wants to merge 6 commits into
Closed
tszymczyszyn-shopify wants to merge 6 commits into
tszymczyszyn-shopify wants to merge 6 commits into
Conversation
This was referenced Oct 5, 2026
tszymczyszyn-shopify
force-pushed
the
namespace-fence/8-operations
branch
from
October 13, 2026 19:41
dbb1e8e to
fe3a094
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/7-replicas
branch
from
October 13, 2026 19:41
993abd6 to
08bd3d2
Compare
Keep what the metastore's bottomless restore reports at startup (whether it recovered the database and the generation it restored from) instead of discarding it, record it on the MetaStore, and report it in the fence capability endpoint (`metastore`), in every fence view (`provenance`), in the `libsql_server_metastore_restored_from_backup` gauge and in a startup warning. When destroy_on_error rebuilds the metastore, the restore of the rebuilt metastore is what is reported. Tests cover a real bottomless restore against a local S3 endpoint, the admin rendering, and that `incarnation.current_log_id` names the rebuilt replication log after a dirty restart while the stored record keeps the log the source was acquired on. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Serve AdoptFence on the admin API (POST /v1/namespaces/:ns/fence/adopt). It needs the admin credential and the separate adoption key configured with --namespace-fence-adoption-key (SQLD_NAMESPACE_FENCE_ADOPTION_KEY), presented in the x-libsql-fence-adoption-key header. The key is kept as a SHA-256 digest and compared without an early exit; without a key, adoption is refused with adoption_not_authorised. The request names the current owner, two distinct approvers, an incident reference and a reason, which are stored in the record and receipt and written to one audit event (target libsql_server::fence::audit). Adoption moves the owner and the revision and nothing else: every admission stays as it was, capabilities of the old owner are revoked, and finished operations cannot be adopted. After a metastore rollback it re-establishes the record the marker holds under the new owner, settles the unavailable name and restores the namespace's own block_* values in memory. When the metastore holds no config row for the name, adoption is refused with namespace_config_missing instead of inventing a configuration. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
Every fence command the server answers (committed, replayed or refused) now emits one structured event under the libsql_server::fence::audit tracing target, with the namespace, operation and command id, outcome, revisions, state before and after, drain kind and duration, forced actions, replay/conflict and server instance; a committed adoption keeps its approvers, incident reference and reason. Metrics, all with bounded labels (docs/NAMESPACE_FENCE.md section 15): transitions by command and outcome, drain duration by kind, forced rollbacks and cancellations by kind, replays and conflicts, denials by code and surface (http, hrana, rpc, proxy, dump, replication, admin_shell, lifecycle), adoptions, and two gauges computed from the fence registry when /metrics is read: namespaces by role and state, and the age of the oldest active fence. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
…dule The fence module carried a module-wide `allow(dead_code)` while its consumers landed across the series. Everything is wired now, so remove it and deal with the five items it was hiding: - delete the `InBeginWriteTxnAfterCheck` and `AfterManagerRelease` hook points, which no code path reaches and no test arms; - delete the unused `AuditReport::drain` accessor and the `FenceController::cancel_read_leases` wrapper (its one test now uses `cancel_read_leases_by_kind`); - compile `DenialSurface::ALL`, `AuditReport::forced_kinds` and the `Fail`/`Indeterminate` hook outcomes only in the library's test build, which is the only place they are used or produced. Co-authored-by: Tomasz Szymczyszyn <tomasz.szymczyszyn@shopify.com>
The admin API's trace layer logged every request's headers at debug level, including the admin key in authorization and the fence adoption key, the second secret AdoptFence requires. An admin can raise the log level through POST /log-filter, so the admin key alone was enough to capture the adoption key. Log those two headers' values as <redacted>.
tszymczyszyn-shopify
force-pushed
the
namespace-fence/7-replicas
branch
from
October 13, 2026 20:51
08bd3d2 to
7291705
Compare
tszymczyszyn-shopify
force-pushed
the
namespace-fence/8-operations
branch
from
October 13, 2026 20:51
fe3a094 to
3a66d17
Compare
Author
|
Closing. This stack is parked; the note at the top of the description explains why. The branch is kept at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surfaces metastore restore provenance and the live log id in fence inspection, adds audited incident adoption (separate adoption key plus two recorded approvers), restart and eviction integration tests over the public protocol, and fence metrics plus a structured audit log (namespace and operation IDs only in logs, never metric labels). Removes the transitional dead-code allowances.
Review focus
Adoption's separate key and two distinct recorded approvers, plus metric label hygiene. The integration tests live here because their harness uses the adoption key.
Since the first version, from review: the admin request log redacts the
authorizationand adoption-key headers.Commits
Stack
Part 9 of 9, based on
namespace-fence/7-replicas. The stack sits on #53 (tszymczyszyn/atomic-namespace-create), which sits on #51: review and land #51 → #53 → #41 → … → #49 bottom-up, restacking after each squash or rebase merge.The first version of these PRs carried #35's fence diff unchanged on
v0.9.30-shopify-patches. On 2026-10-13 the stack was rebased onto #53 (how the conflicts were resolved is summarised in a comment on #41) and gained the commits marked above.