Rindle docs and package mapSkip to main content

Module writeplane

Module writeplane 

Expand description

The shared write-plane machinery for the public one-shot SQL surface (303-PUBLIC-SQL-COORDINATOR-EXTRACTION-PLAN.md). PR 0 landed two layers, PRs 1–4 the coordinator’s batch, DDL, script, and migration slices:

Nothing in this module reads a clock — functions that need time take it as a parameter and the hosts pass their own now_millis().

Modules§

bookkeeping
The public-SQL bookkeeping layer every one-shot write host shares (303-PUBLIC-SQL-COORDINATOR-EXTRACTION-PLAN.md, PR 0 — decision C2): the exact-replay outcome cache, the migrations journal, the idempotency key/identity family, and the result-limit family, moved wholesale from the replicator. Conn-bound functions included — this crate links SQLite — so the coordinator’s control flow never reaches back across the crate boundary for a helper.
budget
Shared limits for one public statement holding a write connection.
oneshot
The shared one-shot coordinator (303-PUBLIC-SQL-COORDINATOR-EXTRACTION-PLAN.md, PRs 1–4): the public-SQL batch, DDL, script, and migration orchestrations — validation, replay decisions, the statement loops with their byte accounting, the lost-race re-reads, and the migration replay/adopt matrix inside the host-owned barrier — written once over the OneShotBackend trait so a second write host implements the trait instead of re-deriving the control flow.
public_http
Transport-neutral policy and JSON bodies shared by the two /v1/sql/* HTTP hosts.
request
Pure request shaping for the write plane’s two intake surfaces.
storage
Connection-only write-plane storage, schema upgrades, and DDL safety guards.

Structs§

DeployMigrationRequest
One private deploy migration after transport parsing. This is deliberately distinct from the public /v1/sql/migrate request (PublicSqlRequest::Migration): deploy migrations also admit data-only files and an operator-reviewed checksum override.
MigrationRecord
One row of the migrations journal — the durable identity a replayed migration tag is validated against. Every decision over it belongs to the caller; the readers here only decode.
ProducerCensus
How many distinct producers hold durable watermark state — the growth signal design 306 §4 names as the mitigation for the one contract it cannot enforce server-side.
PublicMigrateAck
The typed outcome of one public migration: applied is false on an exact replay. The cursor is required — a fresh apply always commits one, and a replay either resolves the exact stored cursor or refuses MIGRATION_OUTCOME_UNAVAILABLE (extraction plan C6).
PublicOperationCommit
The typed outcome of one committed — or exactly replayed — public one-shot operation. results are wire-encoded statement results (the dedicated tagged SQL codec), so a stored replay is byte-equivalent to the fresh commit; the hosts wrap this in their own success envelope (decision C7 — routing metadata and the HTTP layer stay per-host).
PublicSqlRejection
A transport-neutral public SQL parse/validation failure. The host supplies only its concrete HTTP response wrapper; status, typed code, retry scope, and transaction-state semantics are fixed here.
ScriptOutcome
The typed outcome of one public script (execute-multiple). A script never fails at the top level — failure is data: the completed prefix’s wire-encoded results, the last committed statement’s cursor, and the failing statement’s index with its pre-classified error. The hosts render failure through render_public_script_error and wrap success per-host (decision C7).
SqlAuthorizationError
StoredPublicOutcome
One stored exact-replay outcome of the public SQL surface (the replicator’s SqlOutcomeRecord, renamed on the move — extraction plan C2).

Enums§

BookkeepingError
The narrow error currency of the moved bookkeeping helpers (extraction plan, decision C2). It converts losslessly into BOTH MasterError (the replicator’s remaining call sites, via its From impl there) and WritePlaneError (the coordinator) — four 1:1 arms each, no catch-all, so a fifth variant is a compile error on both conversions.
MigrationFileKind
The classification of one migration file: only-DDL or only-data (mixed files are refused).
PublicReadConsistency
The read routing posture selected by the public request’s explicit or client-wide default.
PublicSqlRequest
A fully shaped request from the public SQL surface. Hosts map this value into their own dispatch currency; no transport-specific response or command type crosses this boundary.
SqlIngressAuth
Authentication posture for the versioned public SQL surface.
SqlOutcomeNamespace
The two durable retention-floor namespaces of the outcome cache: one-shot operations (sql-operation: keys) and public transactions (sql-transaction: keys).
WritePlaneError
The one-shot write plane’s error currency (extraction plan, decision C3). Classification into this enum happens exactly once per host — the replicator’s exhaustive From<MasterError> — and rendering happens exactly once, in render_public_sql_error / render_public_script_error, so the typed public codes and the SQLITE_ERROR/sqlite_code classification can never be collapsed into strings and reconstructed. There is deliberately NO conversion out of this type: helpers that both hosts and the coordinator call return BookkeepingError instead, so the one-way promise is enforced by the type graph.

Constants§

CLIENT_MUTATIONS_TABLE
The replicated bookkeeping table carrying each client’s high-water mutation id. Single-_ prefixed: captured by CDC (unlike __replica_meta) and hosted by the engine like any base table — each client’s lmid flows to it through its own one-row system query, transactionally with the data (§8.2).
DATA_MIGRATIONS_TABLE
Replicated apply-once journal for pure-DML migration files. Unlike the DDL journal this table is part of the capture registry: its marker row is the final change in every data-migration run, making zero-user-effect files durable and carrying identity through restore/promotion.
LOG_META_TABLE
A tiny single-row-per-key bookkeeping table. Holds gc_floor — the highest cursor the GC has reclaimed, persisted so the fan-out’s cursor-too-old check survives restart and never regresses (CHANGE-SOURCE-DESIGN.md §10.2) — and the durable SQL-outcome retention floors (SqlOutcomeNamespace::floor_key). Not registered for capture.
MIGRATIONS_JOURNAL_TABLE
The producer-side migration journal (MIGRATIONS-VIA-CHANGELOG-DESIGN.md §4): a master’s record of “have I already minted a ddl journal frame for this migration id.” It is the idempotency key for the migrate surface (a re-POST of a known tag is a no-op), NOT the ordered transport — that is the journal’s ddl frame. Host-local, NOT registered for capture, so it never ships downstream (Drizzle-compatible tag, §3.2).
OUTCOME_RETENTION_LMIDS
Retention bound for ROOM_MUTATION_OUTCOMES_TABLE rows, by lmid distance — never by time (Slice I-ii). When a room flush advances a (doc, client) ledger row to lmid, rows with mid ≤ lmid − K prune in the same transaction. K = 512 mirrors the room shell’s per-client recorded-outcome FIFO cap (MAX_RECORDED_OUTCOMES_PER_CLIENT = 512, packages/room/src/shell.ts) — the two ends of the outcome-resolution surface degrade at the same depth. The accepted loss class is the H-v one: a pruned mid reads as applied through the daemon, exactly as an evicted map entry re-answers with silence on the room socket; a client is only ever that far behind its own ledger with a backlog ≥ K in flight.
PRODUCER_OFFSETS_TABLE
The foreign-write dedup watermark (design 306): one row per producer, overwritten in place.
RINDLE_REQUEST_ID_HEADER
ROOM_CLIENT_MUTATIONS_TABLE
The domain-scoped ledger for room-flush lmid co-commits (RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md §7.1). Keyed by (doc, client_id) — one gapless mid stream per domain, one ledger row per domain — so a room flush and a slow-path daemon mutation for the SAME client never collide on a shared row (the Rev 1 data-loss bug, §8.5’s “ledger isolation” invariant). CLIENT_MUTATIONS_TABLE stays exclusively the slow-path stream; room flushes retarget here.
ROOM_MUTATION_OUTCOMES_TABLE
The durable twin of the H-iv-b mutationOutcome frame ({mid, kind, reason?, name?, args?}), keyed (doc, client_id, mid) — the §4 lifecycle’s outcome-resolution surface for THE NAMED INVARIANT: never retire a room-domain entry off a daemon-carried lmid without outcome resolution (§3.3/§7.5). After downgrade the room socket that ordered outcome-before-ack is gone, so non-applied verdicts must be readable through the daemon subscription plane like the §7.1 ledger row. Rows are written by Slice I-ii’s flush split — this slice only creates + registers the table; an absent row for a covered mid reads as applied (only non-applied outcomes are recorded, matching the room shell’s recorded-outcome map).
ROOM_PLACEMENT_TABLE
The room placement-fence table (RINDLE-REALTIME-DESIGN.md §2.5): one row per doc, bumped by every claim. A flush carrying an epoch below the current claim is fenced — validated with the apply on the single-threaded engine, so the check and the commit are atomic. Unregistered bookkeeping like SOURCE_OFFSETS_TABLE.
ROOM_WATERMARK_TABLE
The §4.2 downgrade fence: (doc, flush_seq), co-committed monotonically in every room flush’s transaction (ClusterConsumer::commit_room_flush). Cross-authority cvs are incomparable, so the fence is data that RIDES THE ECHO: a downgraded client keeps its frozen ghost source until its daemon subscription delivers flush_seq ≥ finalFlushSeq — proof the store it fell back to holds the room’s final flush, whatever the authority shape (single daemon / lagging read-follower / PG).
SCOPE_SESSIONS_TABLE
The §4.1 occupancy table (RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md): one row per (scope, session), upserted by the api-server on every labeled lease mint/renewal and aged out lazily by expires_at (mark/refresh/age-out — lease expiry needs no hook anywhere). The row delta IS the upgrade doorbell: a solo client’s only live connection is its daemon subscription, so the 1→2 wake signal must materialize as a row in the store it is subscribed to — which is why Db::enable_realtime_lifecycle registers this table rather than just creating it.
SOURCE_OFFSETS_TABLE
The durable per-change-source cursor table (CHANGE-SOURCE-DESIGN.md §4). One row per source; the offset upsert is co-transactional with the batch it covers (ClusterConsumer::commit_normalized_with_offset). Daemon bookkeeping — NOT registered for capture (same posture as _rindle_sql_outcomes; the consumer’s own resume position is meaningless on any other host).
SOURCE_OFFSET_WHOLE_RUN
The chunk_seq sentinel meaning “the whole run at this offset is durably applied” — the common case (every pure-row run and every run-boundary commit). A genuine value < this is a mid-run checkpoint (offset, chunk_seq) left by the commit-at-DDL-boundary follower (RELAY-DDL-DESIGN.md §6.6): chunks 0..=chunk_seq of offset are applied, the tail is not. The resume/dedup compare is the keyset (offset, chunk_seq), with the incoming begin(R) treated as (R, WHOLE_RUN) — so a whole run sorts at/above any of its mid-run positions. i64::MAX is safe as a sentinel: chunk_seq is a 0-based within-run ordinal (one per spilled ≤CHUNK_ROWS chunk), so a real value reaching i64::MAX is physically impossible. Mirrors the relay fan-out’s ScanPos “past the end of this run’s chunks” sentinel (rindle-replicator).
SQL_IDEMPOTENCY_MAX_FUTURE_SKEW_MS
Public idempotency keys carry their mint time so an evicted outcome can fail closed without retaining one tombstone per operation. A modest future-skew allowance accommodates ordinary client clock drift while bounding how far one forged key can ratchet the durable floor.
SQL_IDEMPOTENCY_PREFIX
The canonical public idempotency-key prefix: sql1.<13-digit-unix-ms>.<32-lowercase-hex>.
SQL_OUTCOMES_TABLE
Exact replay records for the public SQL surface. The result is stored before commit; run_id is known before the journal assigns a cid and resolves back to that journal point on replay. A zero-effect mutation uses a local metadata-only commit and therefore stores no run id/cursor.
SQL_OUTCOME_MAX_RECORDS
Retained-outcome count ceiling; the sweep evicts oldest-first beyond it.
SQL_OUTCOME_RETENTION_MS
How long one stored public outcome is retained before the sweep may evict it.
WRITE_DDL_TIME_BUDGET
DDL’s longer wall-clock fence while every other writer is quiesced.
WRITE_STATEMENT_TIME_BUDGET
Default public writer-statement wall-clock fence.

Traits§

MigrationSection
The migration-store view OneShotBackend::with_migration_barrier passes to its closure. Reads carry NO decisions — the replay/adopt matrix over them is execute_public_migration’s, the same compiled code on every host; every op executes under the barrier’s exclusion span.
OneShotBackend
The backend contract of the one-shot public SQL surface. The master implements it over its session-transaction primitives; a standalone daemon implements it over its own store (303 S5). Every method returns pre-classified WritePlaneError — classification happens once, host-side, and there is no conversion back out (decision C3).
PublicReplayReads
The three reads behind the shared replay spine, generic over the host’s error currency. The coordinator instantiates it over OneShotBackend (with WritePlaneError); the replicator’s transaction-outcome cursor path instantiates it over a raw connection with MasterError — ONE decision matrix and ONE resolver, no twin, which is what decision C4 means by “generic by construction”.

Functions§

adopt_migration_checksums
Backfill a migration row’s optional identities without overwriting values already made durable. Hosts own the surrounding transaction because producer metadata commits differently on HCTree and WAL2; the column contract is shared.
append_bounded_script_results
Append one batch’s results to a script’s partial-result set, charging the shared aggregate byte allowance (separators included) and failing with the public result-cap error on overflow.
authorize_sql_ingress
Apply the two-token authority rule after a host has supplied its header matcher.
classify_migration_file
Classify one migration file’s statement vector as DDL or data, refusing mixed and unsupported statement classes.
cursor_history_lost
data_migration_checksum
The recorded content checksum of a DATA migration tag, if one was applied (the kind-collision probe for the DDL surface’s MIGRATION_KIND_MISMATCH refusal).
ddl_only_violation_message
Enrich the DDL-only guard’s rejection when a DROP TABLE caused SQLite’s implicit foreign-key delete.
encoded_statement_result_len
The encoded wire length of one statement result — the unit the aggregate result cap accounts in.
enforce_encoded_result_array_limit
Enforce the aggregate cap over an already-encoded result array (the replay path’s check).
ensure_data_migrations_table
Create the captured apply-once data-migration journal.
ensure_log_meta_table
Create the host-local outcome-floor table.
ensure_master_migrations_table
Create the producer/master shape of the DDL migration journal. A daemon with its existing surrogate-key shape calls ensure_migration_identity_columns instead.
ensure_master_run_id_indexes
Index the two run_id columns the master’s post-commit path seeks on — master only, and deliberately not part of ensure_sql_outcomes_table.
ensure_migration_identity_columns
Add the common identity columns to either supported _rindle_migrations table shape.
ensure_outcome_matches_request
Refuse a replay whose stored request identity differs from the retried request (OPERATION_ID_MISMATCH).
ensure_producer_offsets_table
Create the foreign-write producer watermark table.
ensure_sql_outcome_identity_column
Add exact-request and resolved-cursor columns to a legacy outcome cache.
ensure_sql_outcomes_table
Create the exact public SQL outcome cache in its current shape.
ensure_writeplane_tables
Create every storage table owned by the producer/master write plane.
execute_public_batch
Execute one public SQL batch: validate the idempotency key against the caller’s now_ms (future skew bounded — the coordinator reads no clock), check for an exact replay, then run every statement inside one backend transaction while bounding the encoded result array — the JSON array delimiters (2 bytes) and inter-result separators count toward aggregate_result_byte_limit — and commit with the outcome row. Any statement or accounting error rolls the unit back. A commit error is re-read: a concurrent request carrying the same identity may have won while this attempt was open, and its co-transactional outcome is authoritative — the exact stored bytes are returned instead of a duplicate-key/conflict error.
execute_public_ddl
Execute one public DDL operation: validate the idempotency key against the caller’s now_ms, refuse non-DDL statements before any backend call, check for an exact replay, then hand the fresh apply to the backend’s critical section as one atomic unit (OneShotBackend::apply_public_ddl).
execute_public_migration
Execute one public migration (extraction plan C6): apply a permanent, checksum-guarded migration identity, where reusing an id with identical bytes is an exact replay and reusing it for different DDL is refused instead of silently accepting schema drift.
execute_public_read
Execute the shared public read path on a host-owned read connection. The host supplies its current committed cursor lookup; connection ownership stays outside.
execute_public_script
Execute one public script (execute-multiple): an ORDERED sequence of autocommit units, each dispatched by statement class into execute_public_batch (reads/writes) or execute_public_ddl (DDL) under a per-index derived idempotency key — sql-script:<key>:<index>. Replay identity across retries depends on that exact format: every completed statement’s outcome is stored under its derived key, so a retry of the whole script replays the durable prefix instead of re-applying it, then resumes at the statement that failed. Transaction-control statements and classes outside the v1 surface are refused without reaching the backend.
insert_sql_outcome
Insert one outcome row inside the caller’s open transaction (co-transactional with the effects it records). created_at is the host’s now_millis() — this module reads no clock.
insert_sql_outcome_with_cursor
insert_sql_outcome with an exact cursor written atomically. Standalone WAL2 commits use this form because their next TxId is known inside the open transaction; the HCTree master continues to insert a run id and backfill its engine-assigned cursor after commit.
journal_data_migration
Record a pure-data migration inside the caller’s captured transaction. The marker is the final captured row in that migration, so a zero-user-effect file is still durable and replay identity survives restore.
journal_migration
Record that tag’s ddl entry has been minted, inside the caller’s open transaction (so the journal can never disagree with the log entry it guards across a crash, §4).
journal_migration_with_cursor
journal_migration with the host’s exact commit cursor written in the same transaction. WAL2 authorities know their next TxId before COMMIT and use this form; HCTree keeps using the run-id form and resolves/backfills its engine-assigned cursor after commit.
migration_content_checksum_from_record
A record’s canonical content checksum: the stored one when present, else derived from the normalized statements (the legacy pre-content_checksum row shape).
migration_normalized_statements_from_record
Decode and re-normalize the statement vector a migration record stored, if any.
migration_record
Whether migration tag has already minted a ddl log entry (the producer idempotency check, MIGRATIONS-VIA-CHANGELOG-DESIGN.md §4).
parse_public_idempotency_key_at
Parse one canonical public idempotency key (sql1.<13-digit-unix-ms>.<32-lowercase-hex>) against the caller’s now, returning its mint time. Future skew beyond SQL_IDEMPOTENCY_MAX_FUTURE_SKEW_MS is refused.
parse_public_or_derived_idempotency_key_at
Like parse_public_idempotency_key_at, additionally admitting the server-derived script form (sql-script:<canonical-parent>:<index>).
pre_ddl_table_names
Snapshot the non-internal table names before a DDL run.
producer_offsets_ddl
The CREATE TABLE for the foreign-write producer watermark. Shared verbatim by every host that mints it — the connection-only bootstrap here, rindled’s cluster DDL, the write-master’s open batch, and the restore’s runtime initialization — so the four cannot drift into two shapes.
producer_offsets_upsert_sql
The producer watermark’s upsert, shared with the capture-aware writers that must run it through their own exec (the row is replicated data, not connection-local bookkeeping) rather than through upsert_producer_seq.
public_commit_outcome_on_conn
Resolve the terminal outcome of a public transaction from committed metadata only.
public_idempotency_minted_at_ms
Derived execute-multiple keys retain their parent public identity and therefore its floor timestamp. The suffix is server-created and never accepted directly at public intake.
public_transaction_minted_at_ms
The mint time encoded in one public transaction id (…-<hex-nanos>[-ro]), if parseable. A pure parser — the transaction-namespace seams stay host-side, but the outcome sweep consults this for sql-transaction: keys.
read_lmid
Read the durable last mutation id for client_id, or zero when absent.
read_producer_seq
Read the durable last sequence a producer wrote, or zero when the producer is new.
read_sql_outcome
Read one stored outcome row — a record read only, no decisions.
refresh_generation_bound
Replace a generation-bound resource only after its successor opens successfully. Failed refreshes retain a healthy predecessor and stay retryable.
refuse_broken_foreign_keys
Refuse a DDL run that leaves a dangling foreign key or recreates a referenced table without its referenced key.
render_public_script_error
Render one script (execute-multiple) failure: the same classification ladder as render_public_sql_error with the script’s statement_index + partial_results keys, preserving the completed prefix of the ordered autocommit script (the replicator’s old script_sql_error, arm for arm).
render_public_sql_error
Render one one-shot public SQL error — the single classification-to-bytes truth every host shares (the replicator’s old master_sql_error match, arm for arm). A host classifies its own error type into WritePlaneError once, renders here, and wraps the (status, body) pair in its HTTP layer.
resolve_public_outcome_cursor
Resolve the exact cursor carried by one retained outcome row — resolve_replay_cursor wired to the outcome surface: the run_id scan, and the durable outcome row as the GC-race re-read.
resolve_replay_cursor
The ONE exact-cursor resolution algorithm (decision C4), generic over the durable row it resolves: the stored cursor short-circuits, then the legacy run_id scan, then the GC-race re-read of the durable row filtered on the same run_id — journal GC may have backfilled the row and removed the run frame between the first read and the scan, so the re-read is what carries correctness. The outcome surface wires it to the outcome row (resolve_public_outcome_cursor); the migration surface wires it to the section’s scan with record() as the re-read leg, inside the barrier.
shape_deploy_migrations
Shape the private /migrate body into its migrations plus whether the caller used the batch form.
shape_public_sql_request
Shape one public SQL path/body pair. None means the path is not a recognized public SQL route and lets the host preserve its ordinary 404 behavior.
sql_error_response
Build one public SQL error body: {code, message, retry_scope} plus the optional sqlite_code and transaction_state keys. Returns (status, body); the hosts wrap the pair in their own HTTP layer (C7 — response wrapping stays per-host).
sql_operation_key
The outcome-cache key of one public one-shot operation.
sql_outcome_floor_ms
The retention floor for namespace: the durable ratcheted floor, never below the TTL horizon derived from the caller’s now_ms.
sql_outcome_minted_at_ms
The mint time carried by one stored sql-operation: outcome key, if canonical.
sql_request_identity
The canonical request identity of a statement vector — the compact JSON encoding an outcome row stores so a replayed key can prove it carries the same request.
sql_result_limit_error
The public result-cap error, worded once.
sql_success_body
Wrap statement results in the common write-authority success envelope. served_by names the authority role that actually answered — "master" on the replicator, "standalone" on a source-less daemon — so observability keyed on routing never invents a replication plane that isn’t there.
statement_class_name
The wire-visible human name of one statement class — embedded in stable public error messages, so it moves with the classifier rather than being re-derived per host.
statement_is_drop_table
Whether sql is a comment-tolerant DROP TABLE statement.
statement_sql_error
Render one typed public-surface statement failure.
statement_sql_error_status
The public statement surface’s status ladder: result caps are 413, an expired exact outcome is 410 (gone), an identity/checksum conflict is 409, everything else 400.
stored_public_operation
The replay spine at the coordinator surface: stored_public_operation_over driven through the backend trait. Used by both the batch and DDL flows.
stored_public_operation_over
The shared replay decision matrix (decision C4), generic over the host error: a miss whose key minted at or below the retention floor fails closed (OPERATION_OUTCOME_EXPIRED); a stored request identity that differs from the retry refuses (OPERATION_ID_MISMATCH); a run_id-bearing outcome whose exact cursor cannot be resolved refuses (OPERATION_OUTCOME_EXPIRED) rather than answer with a fabricated cursor.
sweep_sql_outcomes
Evict expired and over-quota outcome rows and ratchet the durable per-namespace retention floors. now is the host’s clock; runs inside the host’s own bookkeeping transaction.
take_changes
Move a validated changes array out of a parsed request body without cloning its Value tree. Non-array and absent fields preserve the existing empty-vector fallback.
upsert_lmid
Upsert the durable last mutation id inside the caller’s open transaction.
upsert_producer_seq
Upsert the producer watermark inside the caller’s open transaction. MUST run in the same transaction as the effects it deduplicates, so a crash can never commit an effect without its receipt (design 306 §3.2).
validate_public_ddl_poststate
Validate the producer-side post-state shared by standalone deploy migrations and public SQL DDL. Call this from the schema transaction’s final validation hook, before any outcome/journal tail and before COMMIT, so every refusal rolls the schema and its host bookkeeping back as one unit.
validate_public_request_id
Validate the optional public request-id header and build the shared typed rejection body.
validated_request_id
Return the one canonical request id, rejecting absent, duplicate, and malformed values alike. Callers use this only for response echo; absence therefore maps to None.
with_writer_statement_budget
Run one public writer statement under the standard DML or DDL limits.
with_writer_statement_limits
Run one public writer statement under caller-supplied limits. Exposed for deterministic boundary tests; production hosts normally use with_writer_statement_budget.