Never report a namespace whose entry was emptied as missing

[tszymczyszyn-shopify]

Problem

When libsql-server shuts down, or drops a namespace from its in-memory cache, a request that is already waiting for that namespace can be told that the namespace doesn't exist: HTTP 404, or gRPC FAILED_PRECONDITION "namespace doesn't exist". The namespace is fine. It is still on disk and in the metastore. The server only stopped holding it in memory while the request was waiting.

Why it happens. Each loaded namespace sits in a cache entry. Three things take the namespace out of its entry: shutdown, cache eviction and destroy. A request that picked up the entry just before that finds it empty, and NamespaceStore::with reports an empty entry as "doesn't exist". Only after destroy is that true.

When it happens. On every rolling restart, because the server tears the namespace store down while it is still accepting requests. Under cache pressure it can also happen at any time. In our production deployment it showed up as several hundred Namespace `<name>` doesn't exist errors in one week of rolling restarts and node drains, all for namespaces that existed. Against a local server, the harness in #58 reproduces it at will: with the cache capacity below the number of hot namespaces, 5.9% of answers were false 404s; a SIGTERM under load produced 13 to 41 of them each time.

Why it matters. A 404 tells the client the namespace is gone, and clients act on that, for example by creating it again or discarding local state. A temporary error would have been retried.

Two smaller problems in the same code:

• After shutdown had finished, an admin request could load the namespace from disk again.
• "Doesn't exist" also decides how gRPC requests are authenticated. For a missing namespace, the replication service falls back to the server-wide auth settings, and the proxy service falls back to the auth context sent with the request. A false "doesn't exist" sent requests for existing namespaces down those paths.

Upstream tracks the shutdown case as tursodatabase#1772.

Change

• with
  • Returns NamespaceStoreShutdown (503) once the store has shut down, on every path.
  • When the entry it got is empty, it looks the namespace up again, up to 3 times, so an evicted namespace is reloaded. An entry that stays cached but empty (a fork that failed before storing its record leaves one behind) is found on every attempt; see "Not in scope".
  • When it gives up, it answers NamespaceDoesntExist only if the metastore no longer has the namespace. Otherwise it answers NamespaceStoreShutdown, or the new NamespaceUnavailable (503).
• load_namespace checks has_shutdown again inside its init, which runs on a cache miss only, so one check covers with, create and fork. The error comes back wrapped as Error::Ref. reset checks the flag at its top. A load that was already opening the namespace when shutdown started can still finish; closing that is the shutdown-order follow-up.
• fork loads its source through with and uses the same error rule.
• create
  • Refuses after shutdown, like destroy and fork.
  • If shutdown starts between storing the record and loading the namespace, the load is refused before anything is opened or written, and create removes the record again. Left in place, every retry answered "already exists" for a namespace that was never loaded, so a dump was never applied.
• checkpoint no longer inserts an empty cache entry for a namespace that isn't loaded. Before, a checkpoint of an unloaded namespace made that namespace answer 404 until the entry was evicted.
• gRPC
  • rpc::namespace_error_status maps NamespaceDoesntExist to the existing FAILED_PRECONDITION, and NamespaceStoreShutdown and NamespaceUnavailable to UNAVAILABLE. Anything else stays INTERNAL. FAILED_PRECONDITION is only ever a missing namespace.
  • replication::auth::authenticate, ProxyService::extract_context and ReplicaProxyService::do_auth decide on the code it returns: only a missing namespace falls back to the user strategy (or skips authentication, in the proxy). So the error is classified, and Error::Ref unwrapped, in one place, and that place is tested with wrapped errors. Before, a Hello during shutdown got INTERNAL "Error fetching jwt key for a namespace: Namespace store has shutdown", and a replica shutting down answered INTERNAL from its proxy.
  • Error::as_ref unwraps nested Refs, as the HTTP mapping already did.
• Observability
  • libsql_server_namespace_emptied_entry: each time a request finds its entry emptied and looks again. Rises on every deploy; informational.
  • libsql_server_namespace_emptied_entry_gave_up: requests answered NamespaceUnavailable because every lookup found an emptied entry while the namespace still exists. Giving up because the store shut down, or because the namespace is really gone, is not counted. This is the only new user-visible failure; alert on it.
  • The retry is logged at debug, giving up at warn.

How to review

About 240 lines are production code, comments included; the rest is tests.

Reading order:

1. NamespaceStore::with and emptied_entry_error in namespace/store.rs. This is the fix.
2. The shutdown check inside load_namespace's init, the ones at the top of create and reset, and load_created.
3. The one-line changes in fork and checkpoint.
4. rpc::namespace_error_status and its call sites in proxy.rs, replica_proxy.rs, replication_log.rs and replication/auth.rs.
5. The tests. Skim them, apart from the comments on start_waiting and settle.

Invariants worth checking:

• Shutdown first. has_shutdown is set before shutdown() empties any entry. with checks it at the start of every attempt, and load_namespace checks it before opening a namespace. A request that wakes on an entry emptied by shutdown gets a 503. The ordering between the flag and the entries comes from the entry's RwLock: the flag store is sequenced before shutdown's write-unlock, and the request's read-lock acquires after it, so the Relaxed loads and stores are enough. The one remaining gap is a load that was already inside make_namespace when the flag was set.
• 404 only from the metastore. with returns NamespaceDoesntExist only when the metastore has no record. The remaining exception is the MetaStore::handle resurrection listed under follow-ups.
• Nothing fills an entry it found. A request that finds an emptied entry looks the namespace up again and gets whatever the cache holds then.
• No lock held across cache calls, in with. fork and reset hold an entry write lock across a cache call; that predates this change and is commented in the code, with the fix left to the follow-ups.
• Only a missing namespace changes how a request is authenticated. The three gRPC auth sites fall back (or skip authentication) on FAILED_PRECONDITION from namespace_error_status and on nothing else.
• Test synchronization. moka keeps an extra reference to a fresh entry until its housekeeping runs. The tests therefore settle the cache before they rely on reference counts.

Tests

namespace::store::test, rpc::test, rpc::proxy::test, rpc::replica_proxy::test and rpc::replication::replication_log::test add 19 tests. They drive each case deterministically: the test holds the entry's write lock, so the request waits on the same entry, then empties it the way each caller does. Covered:

• with after shutdown, and shutdown starting while a request waits: NamespaceStoreShutdown. Before, the first reloaded the namespace.
• An entry evicted while a request waits is reloaded into a fresh entry; the emptied entry is not filled. Before: NamespaceDoesntExist.
• An entry that stays cached but empty: NamespaceUnavailable after the three attempts, through the real with, and the entry is still the cached one.
• A namespace destroyed while a request waits, against the real destroy: still NamespaceDoesntExist.
• load_namespace refuses to open a namespace once the store has shut down; reset and create after shutdown are refused.
• create whose load shutdown refused removes its record; a retry against a fresh store on the same files creates the namespace.
• fork reloads a source evicted while it waited; a source emptied between its two loads answers NamespaceUnavailable.
• checkpoint of an unloaded namespace caches nothing.
• The emptied_entry_error outcomes.
• Hello (external and internal service), ProxyService::extract_context and ReplicaProxyService::do_auth after shutdown: UNAVAILABLE. The Hello test configures Basic auth and sends no credentials, and the proxy test sends an x-proxy-authorization header, so each fails if the fallback for the two 503 errors ever returns.
• The status mapping, including wrapped and doubly wrapped errors and the NAMESPACE_DOESNT_EXIST message.
• HTTP 503 and 404 for the respective errors, wrapped or not.

Mutation checks: putting the original NamespaceDoesntExist back on the give-up path, removing either shutdown check, dropping the record removal in create, letting the UNAVAILABLE case fall back to the user strategy, unwrapping only one Ref, swapping the two steps inside destroy, and restoring the proxy fallback each fail the test added for it. The concurrency tests passed 300 consecutive runs before the last restructuring and 30 after it.

cargo nextest run -p libsql-server: 225 tests, 224 passed, with the same results as the base branch.

• On macOS, embedded_replica::local::local_sync_with_writes and embedded_replica::replica_no_resync_on_restart fail on both the base and this branch under parallel load. replica_no_resync_on_restart passes when run alone.
• cluster::replication::apply_partial_snapshot and connection_core::test::test_many_concurrent are flaky under parallel load on both.

Verified against running servers

Debug sqld built from this branch and from the base, run locally with --enable-namespaces and driven over Hrana HTTP by the scripts in #58.

Scenario	Base	This branch
Eviction churn: --max-active-namespaces 2, 6 hot namespaces, 24 clients, 20 s	35,074 × 200, 2,197 × 404 (5.9%) for namespaces that exist	29,807 × 200, 0 non-200. 1,504 emptied entries retried, all succeeded; no give-ups.
SIGTERM under load: 20 namespaces, 32 clients, signal at 3 s, 3 runs	after the signal: 13 to 41 × 404, then connection errors; first 404 at +3 ms	after the signal: 0 × 404, 46 to 62 × 503 Namespace store has shutdown, then connection errors; first 503 at +1 ms

The 65-check regression matrix (regress.sh in #58) ran against both binaries. Only the intended lines differ:

• Checkpoint of an unloaded namespace, then query: base 404 (the namespace was unreachable until eviction), this branch 200.
• SIGTERM under load with namespaces disabled: base 1 × 404, this branch 0 (503 only).
• Replica SIGTERM under read load: base 12 × 404, this branch 0 (503 only).
• Delete under load (S6) is a race on both binaries. With requests in flight, destroy sometimes leaves data-wal behind (DirectoryNotEmpty, two 500 disk I/O error answers) and, on the base, requests sometimes still got 200 after the delete. In a reviewer's 10 runs per binary the directory stayed in 6 of 10 on the base and 2 of 10 here, where requests always got 404 after the delete; in the run above the base left it and this branch did not. The race is the MetaStore::handle resurrection under follow-ups; this change does not add to it and shows it less often.

Identical on both: namespace lifecycle (create, duplicate 400, query, config, stats, delete, real 404s, directory removed); fork with hot and cold source, unknown source 404, loaded destination 400; create from a file:// dump; shared-schema create with hot and cold schema source, linked-db delete guard; data intact after SIGTERM restarts with and without --snapshot-at-shutdown; replication catch-up, proxied writes through the replica, replication and proxied writes resuming after a primary restart; a replica request for an unknown namespace answering 404 through the NAMESPACE_DOESNT_EXIST string match; mixed versions in both directions catching up and recovering after a primary restart. The replica log showed the new mapping live: error connecting to primary. retrying. error: status: Unavailable, message: "Namespace store has shutdown", followed by recovery.

Not exercised live: the proxy-side UNAVAILABLE (the shutdown window is about 10 ms); it is covered by the unit test.

Rollout

• The rollout itself still runs the old binary on every pod it stops. Those pods answer the false 404 one last time while they shut down. Clients that act on a 404 should hold off on that during this rollout.
• What clients see during restarts afterwards:
  • HTTP: 503 instead of 404.
  • gRPC: UNAVAILABLE instead of FAILED_PRECONDITION or INTERNAL.
  • Embedded replicas (the libsql crate): sync() fails with a transport error carrying UNAVAILABLE instead of Error::NamespaceDoesntExist. An application that matched on "namespace doesn't exist" to reset its local replica no longer sees that during a primary restart, which is the point; one that only retried sees a different error text.
  • sqld replicas: snapshot maps UNAVAILABLE to SnapshotPending, so a replica waiting for a snapshot loops on 1 s sleeps logging "snapshot not ready yet" until the primary is back. RemoteConnection::connect in the write proxy retries UNAVAILABLE with uncapped exponential backoff, where it failed immediately on FAILED_PRECONDITION before.
• Fault metrics change shape. http/user/trace.rs counts any HTTP 5xx in libsql_server_user_http_fault{protcol="http"} and treats gRPC FAILED_PRECONDITION as success. Restart-time errors were therefore invisible there. After this change they appear as 503 and UNAVAILABLE on both protocols, and libsql_server_user_http_response{status="503"} rises during restarts. The first rollout of this build looks like today, because the stopping pods run the old binary; the next rolling restart is the first to show the burst. Check dashboards, alerts and deploy gates that read those series before then. Replicas log a non-fatal replication error instead of namespace ... doesn't exist, trying to replicate again while their primary restarts.
• The 503 window lasts the whole namespace teardown. The server shuts the namespace store down before it stops accepting requests, and /health keeps answering 200, so from SIGTERM until every namespace is checkpointed the pod stays in rotation and answers 503 to every request, with no Retry-After. Clients must retry 503 with backoff. Before this change the same window produced the false 404s. Locally it was 10 to 16 ms for 20 small namespaces; measure it in staging on a pod with thousands of loaded namespaces before relying on any client's retry budget.
• Replicas with lazy creation. A replica request waiting while its namespace is taken out of the cache now retries. Replicas create namespaces lazily, so the retry passes the metastore check and reopens the namespace. For a name that was destroyed in the meantime, that opens a fresh replica namespace, as any request for an unknown name does on a replica today; the replicator then gets FAILED_PRECONDITION from the primary.
• Rollback is a revert. Nothing on disk or on the wire changes.

Not in scope (follow-ups)

Ordered by payoff.

1. Shutdown order with a readiness signal. Fail readiness once has_shutdown is set (a separate endpoint, since failing /health would end the pod mid-flush), stop accepting connections, drain in-flight requests with a timeout, then take the namespaces out. This turns the 503 window into zero errors and closes the last shutdown gap, a load already inside make_namespace when the flag is set.
2. MetaStore::handle resurrects records. handle inserts a default in-memory record when the namespace is missing. A request that passes exists just before destroy removes the record re-creates it, and can then open an empty database over the directory being deleted; that is the delete-under-load race above. This predates this change; the retry loop repeats the exists/handle pair per attempt, so a request that retried has up to three such windows instead of one. A non-inserting handle_if_exists closes it. It also removes the last reason ProxyService::extract_context skips authentication for a missing namespace.
3. Entries left empty. A fork that fails before storing its record leaves an empty entry cached. with then gives up with NamespaceUnavailable on every request for that name, and create of that name returns Ok without loading (so a dump is dropped), until the entry is evicted or the server restarts. On the base the same sequence answers 404. libsql-server: drop a namespace cache entry that was left empty #59 (draft) drops such an entry under moka's key lock; it was split out of this change because it is the most delicate part and libsql-server: make namespace creation all-or-nothing (stacked on #51) #53's entry redesign removes the class. Until then: a sustained gave_up rate for one namespace means a restart.
4. Fork onto an existing but unloaded namespace destroys it. Found by the matrix, identical on the base. fork tests to_lock.is_some() instead of the metastore, so an unloaded destination passes. The rename fails with Directory not empty, ForkTask::fork runs remove_dir_all on the destination directory on any error, and the store's cleanup removes the record. Observed: 500, then the destination answers 404 and its directory is gone. Admin API only, but data loss. Fix: check metadata.exists(&to) before get_with(to), and make ForkTask::fork remove only a directory it created. Deserves its own PR soon.
5. Cache entry state. Arc<RwLock<Option<Namespace>>> lets "empty" be a normal state that every reader must handle; with, fork, create and reset each handle it differently. A Filling | Loaded | Gone state with the reservation held outside the cache removes the class, including the entry lock fork and reset hold across a cache call. libsql-server: make namespace creation all-or-nothing (stacked on #51) #53 explored this.
6. Replicator backoff. After a namespace is destroyed on the primary, the replica retries NamespaceDoesntExist in a tight loop: about 5,500 log lines in 2 s per namespace (the existing TODO in configurator/replica.rs).
7. with on the hot path. exists and handle both lock the global metastore mutex, and handle builds a MetaStoreHandle that a cache hit discards. Check the cache first and call handle only on a miss. Benchmark before and after.
8. Smaller items. checkpoint of an unknown or unloaded namespace returns Ok (pre-existing). Under eviction churn, Namespace::shutdown logs unable to remove .sentinel file when a namespace was reopened while the evicted instance was still shutting down (pre-existing, both binaries). A request that holds a connection maker when destroy removes the directory fails with 500 unable to open database file. A proxied write in flight when the primary exits answers 500 PrimaryStreamDisconnect, and new ones wait in the write proxy's unbounded Unavailable retry loop until the primary is back. format_err logs every 503 at error level, as it did the 404s.
9. Test debt. A lazy-creation (replica) fixture: under lazy creation with skips the metastore check and never answers DoesntExist, and a request waiting while its namespace is destroyed reopens a fresh replica namespace for the name; the reload, give-up and destroy tests should run with it. Counters asserted through a DebuggingRecorder. A positive control for checkpoint. start_waiting/settle depend on moka's reference counts; a #[cfg(test)] hook in with would decouple them. One shared make_store for the store and scheduler tests.