libsql-server: make namespace creation all-or-nothing (stacked on #51)

[tszymczyszyn-shopify]

Summary

Stacked on #51 (targets tszymczyszyn/streaming-dump-importer; to be retargeted to v0.9.30-shopify-patches once #51 merges). This PR is the last three commits: 6b469cfe4e code and tests, 626e079524 design doc and admin API, 28defca637 diagrams.

Makes POST /v1/namespaces/:ns/create all-or-nothing. Today NamespaceStore::create persists the namespace config before setting the namespace up and nothing undoes it: a dump that fails to import leaves a ghost namespace (config without data, name unusable, lazily turned into an empty database on first access); a request cancelled mid-import runs no cleanup at all; the schema configurator has no cleanup even on error.

Design: docs/ATOMIC_NAMESPACE_CREATE_DESIGN.md.

📖 Reviewer walkthrough: before/after and component diagrams, sequence diagrams for the happy path, failure/cancellation and eviction, the lifecycle state machine, a suggested reading order with pinned links, and where reviewer judgment is most valuable.

Contract (new, documented in ADMIN_API.md)

After a non-2xx or a lost connection the namespace is either absent (no config, no directory, name reusable) or complete, never partial. For callers: 2xx ⇒ complete; anything else ⇒ retry; already exists on the retry ⇒ the earlier attempt did complete.

How

The metastore write becomes the commit point; everything before it is reversible.

create(name, restore, config)
├─ Reservation::acquire(name)        lock the cache entry, pin it; reject if loaded or config exists
├─ handle(); store(cfg, flush=false) in memory only; setup can read it, nothing on disk
├─ configurator.discard_incomplete   rename dbs/<name> away iff it carries `.incomplete`
├─ make_namespace(...)               setup: FreshDir::begin → mkdir + marker (fsynced) … import … keep()
├─ handle.flush()                    ◄── COMMIT POINT
├─ reservation.publish(ns)           entry := Some(ns); unpin; lock released
└─ remove .incomplete                best effort; a stale marker is removed on the next load
     Err  ⇒ Reservation::abort (awaited): discard config → discard marked dir → unpin → unlock
     drop ⇒ the same, on the blocking pool (the creating request was cancelled)

• Reservation generalises the pattern fork already used (write-lock the cache entry, config in memory only, flush on success). The name stays locked until the attempt is published or fully undone, so waiters see the namespace or "doesn't exist", never a half-built one, and a retry issued the moment the error arrives neither races the cleanup nor finds leftovers. fork uses it too; its old guard used block_in_place (panics on a current-thread runtime) and published before flushing.
• Marker and tombstones (namespace/incomplete.rs): a new directory carries .incomplete until the config is persisted. Directories are never deleted in place; discard_dir renames them into <base>/.discarded/ (instant, frees the path) and deletes in the background; a startup sweep removes leftover tombstones and marked orphans; the metastore's filesystem recovery skips marked directories. Directories without the marker keep today's semantics.
• MetaStore::discard forgets a namespace unconditionally through the metastore's own change queue (so it is ordered after a config store that was still queued when the request was cancelled); remove's linked-DB guards would otherwise leave a ghost. existing_handle looks a namespace up without registering it; with() and the fork source use it.
• Cache eviction during an operation: the reservation is the entry's write lock, which moka knows nothing about. The locked entry is pinned in in_flight; load_namespace and destroy consult it first and wait on it; the eviction listener spawns its shutdown only for pinned entries (moka awaits async listeners inline from whichever cache operation runs maintenance); install publishes into a fresh entry when the locked one was evicted, or hands the namespace to destroy when that is why it is gone.
• Requests addressed to a namespace under creation wait (as during fork/reset). shutdown skips pinned entries; create/fork re-check has_shutdown before the commit point.
• Needs async-lock = "3" for RwLock::write_arc (already in the lockfile; the bump removes the duplicate 2.x).

Behaviour changes visible to clients

• Failed create ⇒ GET …/config is 404 and user requests get "namespace doesn't exist" (was: 200 / "no such table").
• Single-namespace mode: create default from a dump while it exists ⇒ already exists (was: 200 with the dump silently skipped). The config-upsert behaviour of a plain create default is unchanged.
• A <db-path>/.discarded/ directory appears; it holds renamed directories until the background delete or the startup sweep removes them.

Tests

• 9 lifecycle integration tests + 2 cancellation tests (both importers): failed creation leaves no trace (primary and schema configurators), cancelled creation (cancel fires once the marker exists), concurrent requests wait, crash remnant discarded, unmarked directory untouched, marker on a published namespace ignored (create before any lookup after a restart), single-namespace dump create rejected. 9 of the 11 fail on the unfixed code; the other two pin behaviour that must hold before and after.
• Store unit tests build a real NamespaceStore with a channel-fed streaming import: lookup waits across an eviction and sees the data, or "doesn't exist" when the import fails; destroy during creation; retry the instant the error returns (no polling); abort mid-import; discard vs remove's guards. Unit tests for the marker protocol, tombstones (with an open file), the startup sweep, and metastore recovery skipping marked directories.
• Full cargo nextest run -p libsql-server (270 tests): identical results to the base branch.

Staging soak (2026-10-15)

A pre-release build of this PR's code (bb446259f6; byte-identical to the current head, only docs moved since) was rolled out to a staging fleet of 192 database pods (two StatefulSets in parallel, zero restarts) and then used for the #51 validation: five namespaces created through the new path (0.45–7.27 GB imports, the longest 4.4 min) and five destroyed, with live write traffic to the pods' other namespaces throughout. A scan of all 192 pods after about 1.5 h found no libsql_server_namespace_create_aborted increments and no .discarded/ tombstones. The build stays on staging as an extended soak.

Sequencing and follow-ups

• Overlaps with Never report a namespace whose entry was emptied as missing #56 in NamespaceStore::checkpoint (both stop it from inserting an empty cache entry) and in how an empty, unlocked cache entry is treated in load_namespace/with. Whichever merges second gets rebased; this PR is not needed before Never report a namespace whose entry was emptied as missing #56.
• Accepted residues (design doc §9): the empty cache entry left after an abort (never served; healed on the next lookup or evicted), and a metastore rebuilt from the filesystem in the milliseconds between a namespace's flush and its unmark.
• Bottomless replication is not exercised by this change's tests.
• A cache entry state (Filling | Loaded | Gone) with the reservation kept outside the cache would remove the pin map and the pin-aware listener; deferred.

[tszymczyszyn-shopify]

Reviewer walkthrough

Pinned to bb446259f6 — the commit the staging pre-release was built from. The branch was since rebased onto #51's latest docs (current head 28defca637); the code is byte-identical, so every link below still shows exactly what runs on staging. Stacked on #51; this PR is the last three commits (code and tests, design doc and admin API, diagrams). Suggested reading order below; ~45 minutes, a little more with the primer.

TL;DR

POST /v1/namespaces/:ns/create used to persist the namespace's config first and set the namespace up afterwards; nothing undid the config on failure and nothing at all ran on cancellation, so a failed import left a ghost (name taken, next request lazily opened an empty database). This PR makes the metastore flush the commit point and makes everything before it reversible — with two drop guards (one per layer), an on-disk marker for crashes, and a pin that keeps the operation safe from cache eviction.

Contract for callers: 2xx ⇒ complete. Anything else, including a lost connection ⇒ retry; already exists on the retry ⇒ the earlier attempt did complete.

Primer: the pieces this PR touches (if you don't know libsql-server's namespace code)

A namespace is one SQLite database served under a name (http://<name>.<host>). Four things cooperate to make one exist:

Piece	File	What it is
NamespaceStore	namespace/store.rs	The registry of loaded namespaces: a moka cache Name → Arc<RwLock<Option<Namespace>>>, bounded by --max-active-namespaces (default 100). Evicted namespaces are shut down and lazily reloaded from disk on the next request. with(name, f) is how every user request reaches a namespace; create, fork, reset, destroy are the admin operations.
MetaStore	namespace/meta_store.rs	Which namespaces exist and their DatabaseConfig: an in-memory map (configs, what exists() reads) backed by a SQLite table (namespace_configs). Writes go through a FIFO change queue processed on the blocking pool. Gotcha this PR works around: handle(name) inserts the name into the in-memory map as a side effect.
Configurators	namespace/configurator/{primary,schema,replica,fork}.rs	setup(name, RestoreOption) builds a Namespace from dbs/<name>/: directory, .sentinel (dirty-shutdown flag), replication logger (wallog), SQLite connection maker, and — for RestoreOption::Dump — the dump import from #51. cleanup removes a namespace's files.
Admin API	http/admin/mod.rs	POST /v1/namespaces/:ns/create resolves dump_url into a stream and calls NamespaceStore::create.

Two moka facts matter: (1) try_get_with(key, init) runs init at most once per key and makes concurrent callers wait for it — the base code relies on this to avoid loading a namespace twice; (2) moka 0.12 has no background thread: cache maintenance (including awaiting async eviction listeners) runs inline inside whatever cache operation trips it.

Before this PR, create did: metastore.store(config) (durable) → load_namespace → setup. The order is what this PR inverts.

Before / after

Who owns what

The happy path, with a concurrent lookup

sequenceDiagram
    autonumber
    participant A as Admin API
    participant S as NamespaceStore
    participant M as MetaStore
    participant C as Configurator
    participant FS as Filesystem
    participant U as User request (with foo)

    A->>S: create(foo, Dump, cfg)
    S->>S: Reservation::acquire — get_with entry, write_arc(), reject if Some or exists(foo), pin in in_flight
    S->>M: handle(foo); store(cfg, flush=false)
    Note over M: in memory only — exists(foo) is now true, nothing on disk
    S->>C: discard_incomplete(foo)
    Note over C,FS: removes dbs/foo only if it still carries .incomplete (a crashed earlier attempt)
    S->>C: setup(foo)
    C->>FS: FreshDir::begin — mkdir dbs/foo + .incomplete (sync)
    C->>C: logger, WAL, connections, dump import…
    U-->>S: with(foo): existing_handle → load_namespace
    Note over S,U: in_flight has foo → wait on the pinned entry's read lock (never a lazy load)
    C-->>S: Ok(Namespace)  (FreshDir.keep())
    S->>M: handle.flush()
    Note over M: COMMIT POINT — row written; a row now implies a complete namespace
    S->>S: publish → install(ns) into the locked entry, unpin, unlock
    S->>FS: remove dbs/foo/.incomplete (best effort)
    S-->>A: 2xx
    S-->>U: read lock acquired → Some(ns) → serve

Loading

Failure or cancellation

sequenceDiagram
    autonumber
    participant A as Admin API
    participant S as NamespaceStore
    participant M as MetaStore
    participant C as Configurator
    participant FS as Filesystem
    participant U as User request (with foo)

    A->>S: create(foo, Dump, cfg)
    S->>S: Reservation::acquire (lock + pin)
    S->>M: store(cfg, flush=false)
    S->>C: setup(foo)
    C->>FS: mkdir dbs/foo + .incomplete
    U-->>S: with(foo) → waits on the pinned entry
    alt import fails
        C->>FS: FreshDir::drop — rename dbs/foo → .discarded/foo-<uuid> (sync), delete in background
        C-->>S: Err
        S->>S: Reservation::abort() — awaited
    else client disconnects (future dropped)
        Note over C,S: FreshDir and Reservation are dropped during teardown
        C->>FS: FreshDir::drop — rename, same as above
        S->>S: Reservation::drop — cleanup handed to the blocking pool
    end
    S->>M: Discard(foo) via the change queue (unconditional, ordered after any queued store)
    S->>C: discard_incomplete(foo) — no-op if FreshDir already renamed it
    S->>S: unpin, then unlock
    S-->>A: error (only after the cleanup, on the Err path)
    S-->>U: read lock acquired → None; exists(foo) false → NamespaceDoesntExist
    Note over A: a retry now finds a free name and a free path

Loading

Lifecycle of a name, including crashes

stateDiagram-v2
    [*] --> Absent
    Absent --> Reserved: acquire (entry locked + pinned, config in memory)
    Reserved --> SettingUp: FreshDir::begin — dbs/foo + .incomplete
    SettingUp --> Committed: flush — row written
    Committed --> Published: install, unpin, unlock
    Published --> Complete: .incomplete removed
    Complete --> [*]

    Reserved --> Discarding: Err / drop
    SettingUp --> Discarding: Err / drop — dir renamed to .discarded/
    Committed --> Discarding: drop before publish — Discard deletes the row again
    Discarding --> Absent: unpin, unlock

    SettingUp --> MarkedOrphan: process dies
    Committed --> Published: process dies here — row + marker; serves after restart, marker dropped on load
    MarkedOrphan --> Absent: sweep at startup, or next create of the name (discard_incomplete)
    Published --> Complete: process dies before unmark — stale marker removed on next load

    note right of Discarding
        Order matters: config → directory → unpin → unlock.
        The name is unobservable until it is clean.
    end note
    note right of MarkedOrphan
        No row, so exists(foo) is false: invisible to clients.
        The marker is what tells it apart from a real namespace.
    end note

Loading

Eviction during an operation (commit 3)

sequenceDiagram
    autonumber
    participant S as NamespaceStore (create foo)
    participant K as moka cache
    participant L as eviction listener
    participant U as with(foo)

    S->>K: get_with(foo) → entry E; write_arc(E); pin E in in_flight
    Note over S: import running for minutes…
    K->>K: capacity pressure — evicts E
    K->>L: notify(foo, E)
    alt foo is pinned
        L->>L: spawn the shutdown (don't await E's lock inline — moka awaits listeners from arbitrary cache ops)
    else not pinned (a normal eviction)
        L->>L: await E.write() and shut down inline (serialises with a reload of the same name)
    end
    U->>S: load_namespace(foo): in_flight has foo → wait on E (not on the cache)
    S->>K: install: store.get(foo) ≠ E and exists(foo) → insert fresh entry F with Some(ns)
    Note over S: E stays empty — the spawned listener finds nothing to shut down
    S->>S: unpin, unlock
    U->>S: E is None → exists(foo) true → try_get_with → F → serve

Loading

If foo was destroyed meanwhile instead of evicted, install sees exists(foo) == false and hands the namespace to E, where destroy is waiting for it ("created, then destroyed").

Suggested reading order

1. The on-disk protocol (small, self-contained)

• namespace/incomplete.rs: INCOMPLETE_MARKER, create_marked_dir/mark_incomplete (fsynced), discard_dir (rename into .discarded/, background delete), discard_if_incomplete, sweep — with unit tests at the bottom (tombstone with an open file, sweep, FS recovery skipping marked dirs).
• FreshDir: the configurator's guard; synchronous begin, Drop ⇒ discard_dir. Used by primary and schema setup; fork marks its temp dir.
• Startup sweep call in lib.rs.

2. The reservation (the heart of the change)

• Reservation: acquire (prefers a pinned entry; invalidates the empty entry it created when rejecting), publish, abort (awaited on error paths), Drop (cancellation only), take_cleanup with the config → directory → unpin → unlock order.
• create / create_reserved and fork / fork_reserved: same shape; has_shutdown re-checked before the commit; publish before unmark.
• MetaStore: ChangeMsg::{Update, Discard}; discard is unconditional (unlike remove, whose linked-DB guards would leave a ghost); existing_handle is the non-registering lookup; the flush branch of process no longer holds configs across try_process (lock order vs remove).

3. Eviction safety

• InFlight pin map + unpin, pin-aware eviction listener, install, load_namespace (pinned entry first; empty-and-unlocked entry treated as a miss), destroy (prefers the pinned entry), shutdown (skips pinned entries).

4. Tests

• Store unit tests (store.rs::test): a real NamespaceStore with a channel-fed streaming import — lookup waits across an eviction; lookup sees failure across an eviction; destroy during creation; retry the instant the error returns (no polling); abort mid-import (JoinHandle::abort); discard vs remove's guards.
• Integration (tests/namespaces/lifecycle.rs, dumps.rs cancel tests): failed create leaves no trace (both importers + schema configurator), cancelled create (cancel fires only once dbs/foo/.incomplete exists), concurrent requests wait, crash remnant discarded, unmarked dir untouched, marker on a published namespace ignored (creates before any lookup after a restart), single-namespace dump create rejected.
• 9 of the original 11 integration tests fail on the unfixed code; the two that pass pin behaviour that must hold before and after.

Behaviour changes visible to clients

• Failed create ⇒ GET …/config is 404 and users get "doesn't exist" (was: 200 / "no such table").
• Single-namespace mode: create default from a dump while it exists ⇒ already exists (was: 200, dump silently skipped). Plain create default stays a config upsert.
• <db-path>/.discarded/ appears; it holds renamed directories until the background delete or the startup sweep removes them.

Where reviewer judgment is most valuable

1. The cleanup order in Reservation::take_cleanup — is there any interleaving where the name becomes observable before it is clean?
2. install's three-way decision (same entry / destroyed / evicted) and load_namespace's empty-entry healing: do they cover every way an entry can go stale?
3. MetaStore::discard bypassing remove's linked-DB guards — correct for an unpublished namespace; is there any path where it runs for a published one? (It only runs from a Reservation, whose acquire rejects existing names.)
4. The pin-aware eviction listener: inline for normal evictions, spawned for pinned — any entry that is long-locked but not pinned?
5. Whether the tombstone directory and startup sweep are acceptable operationally (disk use is bounded by the background delete; sweep logs each leftover).

Explicitly out of scope

• Bottomless: not used in our deployment; this change is untested with it.
• Two accepted residues (design doc §9): an empty cache entry after an abort (never served; healed on the next lookup or evicted), and a metastore rebuilt from the filesystem in the milliseconds between a namespace's flush and its unmark.
• An import timeout; idempotency keys / async create jobs.

Full rationale, failure matrix and the review table: docs/ATOMIC_NAMESPACE_CREATE_DESIGN.md.

[tszymczyszyn-shopify]

Closing for now. Database moves are run by an operator, who deletes a namespace left behind by a failed import, so all-or-nothing creation is not needed yet. The branch is kept at bb446259f6. Reopen if partial namespaces from failed imports become a recurring problem. #51 continues on its own.

[tszymczyszyn-shopify]

Reopening. Beyond database moves, this fixes ghost namespaces after failed or cancelled creates, fork publishing before its config is saved, and the cache-eviction window during long operations. It is not planned for the next patch release; it will be reviewed on its own after that.