libsql-server: memory-bounded streaming dump importer (opt-in, benchmarkable against the buffered one)

[tszymczyszyn-shopify]

Summary

Adds an opt-in, memory-bounded streaming importer for POST /v1/namespaces/:ns/create + dump_url, selectable per request or server-wide, alongside the existing (now called buffered) importer, which is kept byte-for-byte so both can be benchmarked and validated on the same libsql-server instance.

Design: docs/STREAMING_DUMP_IMPORT_DESIGN.md.

Stacked on #56 (the false-404 fix), which is meant to land first; #53 is stacked on this PR. Rebased onto #56 on 2026-10-16. The only change that rebase needed in this PR is the new dump_import field in #56's store test helper, in commit 1. The walkthrough's commit links predate the rebase.

📖 Reviewer walkthrough — request flow, suggested reading order, invariants, failure paths, compatibility differences, test anchors, and review focus.

Why

The current importer does read_to_string on the whole dump (plus a lowercase copy for the attach check), so import RSS is proportional to dump size. Everything downstream (SQLite page cache, replication log) is already bounded or flushed to disk; the importer's String was the only size-proportional allocation.

The buffered importer also rejects any dump whose data contains the substring attach (the check is contains("attach") over the whole file). Ordinary text often contains the word (attached, attachment), and a database with such text cannot be imported with the shipped code at all.

What

• Runtime selection
  • request body: "dump_importer": "buffered" | "streaming" (400 if given without dump_url, 422 for unknown values)
  • server default: --dump-importer / SQLD_DUMP_IMPORTER (default buffered: no behavior change until opted in)
  • streaming knobs: --dump-import-max-statement-size (64 MiB → HTTP 413), --dump-import-queue-bytes (16 MiB), --dump-import-queue-depth (256)
• Streaming pipeline: async reader → StatementFramer (single-pass resumable port of SQLite's complete.c state machine, absolute line/column tracking) → bounded mpsc + byte-budget Semaphore → one BLOCKING_RT executor thread owning the connection. The queue has a message limit for small rows and a byte limit for large statements; design doc §4 explains why both are needed. Explicit End message; a channel closed without End (stream error, framing error, cancelled admin request) ⇒ ROLLBACK on the executor thread.
• Per statement: UTF-8 check → sqlite3_parser for policy (empty frames, libsql_wasm_func_table skip, ATTACH/DETACH → 400 with the existing message, same n_stmt > 2 && autocommit ⇒ NoTxn rule) → execute the original statement text. Parser errors are remapped to absolute dump positions, so the existing error snapshots apply to both importers unchanged.
• Metrics libsql_server_dump_import_{duration_seconds,bytes,statements,max_statement_bytes}{importer} plus failures{importer,kind}; start/progress/success logs, and failure logs with the failure category and partial progress.
• file: dumps are read in 64 KiB chunks (was 4 KiB); benefits both importers.

Behavior differences (streaming vs buffered), see §10 of the design

• The word attach inside data no longer rejects the dump (the streaming check is statement-level, with the authorizer as backstop). A standalone DETACH is rejected with 400 (buffered: fails at execution with 500).
• Schema SQL stored in sqlite_schema is the dump's DDL verbatim; buffered stores the parser's re-serialization (e.g. ON plain (a), multi-line trigger bodies). Data is identical.
• Invalid UTF-8 / NUL → 400 (was 500); oversized statement → 413; row-returning statements (EXPLAIN, stray SELECT) execute with rows discarded (was 500).

Validation

Validated on a pre-production deployment with large real databases. Peak importer memory stayed flat as dump size grew, every imported database dumped back byte-identical to its input, and PRAGMA integrity_check returned ok. That build predates the rebase onto #56, which did not change this PR's runtime code.

Local run (debug build, 36 MB dump / 300 007 statements, scripts/bench-dump-import.sh): streaming 15.2 s / +15 MB, buffered 19.7 s / +135 MB, identical data.

Tests

• 20 unit tests (dump_import::{complete,framer,mod}): framer cases run for every chunk size 1..=len; a differential test checks the framer against SQLite's own sqlite3_complete on 2 000 random token soups × 5 chunkings; linear framing of a semicolon-dense 1 MiB statement; zero-copy hand-over of large frames.
• 39 dump integration tests: every existing dump test also runs with _streaming (existing snapshots reused by name, including the absolute-position one: line 7, column 11); new: 1/7-byte HTTP chunking, truncated body → 500 + rollback, 413, NUL/invalid UTF-8 → 400, dump_importer field validation, server-wide default, WASM-table skip, empty dump, EOF without ;, executor failure under a saturated queue, abandoned admin request (executor rolls back and releases the connection), a 50k-statement dump under a 64 KiB queue budget, and an equivalence test comparing both importers' exported data.
• Full cargo nextest run -p libsql-server: same result as the base branch (embedded_replica::local::local_sync_with_writes fails identically on the base commit on macOS).

Commits

1. extract dump loading into a selectable importer: pure refactor + plumbing; buffered importer moved verbatim, streaming stubbed; existing tests/snapshots untouched
2. add incremental SQL statement framer
3. add memory-bounded streaming dump importer + tests
4. docs + benchmark script
5. address adversarial review of the streaming dump importer: resumable complete.c port replacing a per-semicolon sqlite3_complete() call that was O(n²) in interior semicolons; zero-copy large frames; config hardening; progress/partial-stats logging; tests
6. docs: describe the linear dump statement scanner
7. docs: drop internal references from the streaming importer design
8. docs: explain the import queue's two limits; also notes that libsql-server: make namespace creation all-or-nothing (stacked on #51) #53 is stacked on this PR

Follow-ups

• Flip --dump-importer to streaming by default, then remove the buffered importer after a soak. The §15 acceptance criteria are met by the validation above.
• The per-statement policy (WASM-table skip, transaction rule) is duplicated between the two importers until the buffered one is removed; the parameterised tests pin both.
• No admission control for concurrent streaming imports: each occupies one BLOCKING_RT thread and its own queue budget for its duration.
• Enormous single statements/BLOBs are out of scope; the size cap is a safety valve, not support.
• Namespace lifecycle on a failed or cancelled create (config kept, directory not cleaned) is pre-existing and not changed here; libsql-server: make namespace creation all-or-nothing (stacked on #51) #53, stacked on this PR, makes creation all-or-nothing. Until it lands, an operator deletes a namespace left behind by a failed import.
• Local macOS build: libsql-ffi/build.rs passes a header to cc as a source, which current Xcode ar silently drops, so sqld fails to link; see libsql-ffi: do not compile pcre2_internal.h on Apple targets #57. Not part of this PR.

[tszymczyszyn-shopify]

Reviewer walkthrough

Pinned to ecb94314aa. This is a large diff, but the runtime path is concentrated in five new files under namespace/dump_import. Suggested review order below.

TL;DR

This adds an opt-in streaming implementation for namespace creation from dump_url. The historical importer remains the default and the control arm. The new path incrementally frames SQL, puts complete statements behind count + byte backpressure, and executes them in order on one blocking thread owning the connection. Its importer memory is bounded by the configured queue plus the largest allowed statement rather than the whole dump.

No default behavior changes: buffered remains the server default. Selection is per request ("dump_importer":"streaming") or server-wide (--dump-importer / SQLD_DUMP_IMPORTER).

End-to-end flow

flowchart LR
    A[POST namespace create] --> B[CreateNamespaceReq]
    B --> C[DumpSource<br/>stream + optional importer]
    C --> D[resolve request override<br/>or server default]
    D --> E{importer}
    E -->|buffered| F[legacy whole-dump importer]
    E -->|streaming| G[async DumpStream reader]
    G --> H[StatementFramer<br/>linear complete.c state machine]
    H --> I[mpsc queue<br/>depth bound]
    I --> J[Semaphore permit travels<br/>with frame: byte bound]
    J --> K[one BLOCKING_RT executor<br/>owns PrimaryConnection]
    K --> L[UTF-8 + parse/policy check]
    L --> M[execute original SQL text]
    M --> N[COMMIT or best-effort ROLLBACK]

Loading

The explicit Msg::End distinguishes clean EOF from a dropped reader. A closed channel without End means stream/framing failure or request cancellation and forces rollback.

Suggested reading order

1. API and selection plumbing (small)

• CreateNamespaceReq and request validation: dump_importer requires dump_url; the URL becomes a DumpSource carrying the stream and optional override.
• DumpSource / RestoreOption: carries selection through namespace setup without changing other restore modes.
• Importer resolution at the fresh-database boundary: request override first, then server default.
• CLI/env flags and defaults, with validation when building DbConfig.

2. Dispatcher and unchanged control arm

• DumpImporterKind, config and defaults: buffered is deliberately still the default; request/CLI parsing is case-insensitive.
• load_dump dispatcher, logs and metrics.
• buffered.rs: the old algorithm extracted as the A/B control. Review commit cf6fab3 separately if you want to verify the behavior-preserving move.

3. Statement boundary detection (most correctness-sensitive)

• complete.rs: resumable Rust port of SQLite complete.c's tokenizer and 8×8 state machine. This handles quotes, comments, bracket identifiers and CREATE [TEMP] TRIGGER … END; across arbitrary chunk boundaries.
• StatementFramer::push / finish: absolute positions, NUL/size checks, one linear scan, and zero-copy hand-over for frames ≥1 MiB.
• Differential and pathological-shape tests: 2,000 deterministic token soups × five chunkings are compared with the real sqlite3_complete(); separate tests cover semicolon density and allocation identity.

Why the local port: calling non-resumable sqlite3_complete() for each candidate semicolon was quadratic. It measured 20 s CPU for one 1 MiB/50k-semicolon statement on a tokio worker. The port scans every dump byte once and keeps the FFI only as its test oracle.

4. Bounded reader/executor pipeline

• load_dump_streaming: drives the stream and framer, joins the blocking executor, and resolves source-vs-executor error precedence.
• send_frame: an owned semaphore permit travels inside each message. Frames larger than the queue budget acquire the whole budget and therefore travel alone.
• run_executor: one thread owns the connection, installs the ATTACH/DETACH authorizer, consumes until explicit End, and rolls back on every error/channel-close path.
• handle_statement and execution: UTF-8 → parse one command → legacy WASM-table skip → ATTACH/DETACH policy → legacy transaction rule → execute the original SQL while draining returned rows.

Memory model: one stream chunk + one unfinished frame + at most queue_bytes of queued frames + the statement being executed. max_statement_bytes is a separate safety cap (64 MiB default); huge individual BLOB literals are intentionally not optimized beyond that cap.

5. Failure semantics and tests

Event	Result
Source/framing failure	sender drops; executor sees close, rolls back; specific source error wins
Executor parse/execute/policy failure	receiver drops; blocked reader wakes; executor error wins
Clean EOF	final tail is sent, then explicit End; executor requires autocommit
EOF with open transaction	explicit rollback + NoCommit
Admin request cancellation	sender/stream drop; uncancelled blocking executor sees close and rolls back
Executor panic	connection is dropped during unwind; SQLite rolls back an open transaction

The main integration anchors are chunking + truncated body, cross-importer data equivalence and large-dump backpressure, and EOF, saturated-queue failure, and cancellation.

Intentional compatibility differences

The full matrix is in design §10. The important ones:

• streaming stores the original DDL text; buffered executes the parser's normalized rendering (data is equivalent);
• the word attach inside data is accepted by streaming instead of triggering buffered's substring false positive;
• invalid UTF-8/NUL is 400 instead of 500; oversized statements are 413;
• standalone DETACH is 400 instead of buffered's execution-time 500;
• row-returning statements execute with rows discarded instead of failing with ExecuteReturnedResults.

These differences are opt-in because buffered remains the default.

Evidence

• 20 dump_import unit tests.
• 39 dump integration tests, including cancellation and saturated backpressure.
• CI was fully green on code head 967f8f637e; this pinned head only updates documentation to match that code and has CI re-running.
• 36 MB / 300,007 statements: streaming 15.2 s / +15 MB RSS, buffered 19.7 s / +135 MB RSS; identical 300,300 data rows; integrity_check=ok.
• Pathological 10 MB CSS-shaped dump (~5,000 interior semicolons per 200 KB row): streaming 0.56 s / +21 MB, buffered 0.53 s / +32 MB, identical data. The pre-fix framer took ~0.36 s per row.

Where reviewer judgment is most valuable

1. Does CompletionScanner faithfully match the bundled SQLite completion rules? The differential test is the main guard.
2. Do the channel-close/error-precedence paths always terminate and rollback, including cancellation?
3. Is the stated memory bound convincing, especially the permit lifetime inside Msg::Stmt and oversized-frame behavior?
4. Are the documented opt-in behavior differences acceptable for rollout?
5. Is keeping buffered as default until production-shaped validation the right migration boundary?

Explicitly out of scope / deferred

• Metastore cleanup when namespace creation fails or the admin request is cancelled (pre-existing lifecycle behavior).
• A server-side import timeout.
• Bottomless's own buffering.
• Admission control for many concurrent long-lived imports on BLOCKING_RT.
• Removing the buffered importer or flipping the default; both require production soak first.

The full rationale, memory model, behavior matrix, benchmark procedure and adversarial-review resolutions are in STREAMING_DUMP_IMPORT_DESIGN.md.