Module bookkeeping
Expand description
The public-SQL bookkeeping layer every one-shot write host shares
(303-PUBLIC-SQL-COORDINATOR-EXTRACTION-PLAN.md, PR 0 — decision C2): the exact-replay
outcome cache, the migrations journal, the idempotency key/identity family, and the
result-limit family, moved wholesale from the replicator. Conn-bound functions included —
this crate links SQLite — so the coordinator’s control flow never reaches back across the
crate boundary for a helper.
Two rules hold module-wide:
- No clock, anywhere. Every function that needs time takes it as a parameter
(
created_at,now,now_ms); the hosts pass their ownnow_millis()and tests script the values. - Every fallible function returns
BookkeepingError— the narrow currency that converts losslessly into both the replicator’sMasterErrorand the coordinator’sWritePlaneError.
Structs§
- Migration
Record - One row of the migrations journal — the durable identity a replayed migration tag is validated against. Every decision over it belongs to the caller; the readers here only decode.
- Stored
Public Outcome - One stored exact-replay outcome of the public SQL surface (the replicator’s
SqlOutcomeRecord, renamed on the move — extraction plan C2).
Enums§
- Migration
File Kind - The classification of one migration file: only-DDL or only-data (mixed files are refused).
- SqlOutcome
Namespace - The two durable retention-floor namespaces of the outcome cache: one-shot operations
(
sql-operation:keys) and public transactions (sql-transaction:keys).
Constants§
- CLIENT_
MUTATIONS_ TABLE - The replicated bookkeeping table carrying each client’s high-water mutation id.
Single-
_prefixed: captured by CDC (unlike__replica_meta) and hosted by the engine like any base table — each client’s lmid flows to it through its own one-row system query, transactionally with the data (§8.2). - DATA_
MIGRATIONS_ TABLE - Replicated apply-once journal for pure-DML migration files. Unlike the DDL journal this table is part of the capture registry: its marker row is the final change in every data-migration run, making zero-user-effect files durable and carrying identity through restore/promotion.
- LOG_
META_ TABLE - A tiny single-row-per-key bookkeeping table. Holds
gc_floor— the highest cursor the GC has reclaimed, persisted so the fan-out’scursor-too-oldcheck survives restart and never regresses (CHANGE-SOURCE-DESIGN.md§10.2) — and the durable SQL-outcome retention floors (SqlOutcomeNamespace::floor_key). Not registered for capture. - MIGRATIONS_
JOURNAL_ TABLE - The producer-side migration journal (
MIGRATIONS-VIA-CHANGELOG-DESIGN.md§4): a master’s record of “have I already minted addljournal frame for this migration id.” It is the idempotency key for the migrate surface (a re-POST of a knowntagis a no-op), NOT the ordered transport — that is the journal’sddlframe. Host-local, NOT registered for capture, so it never ships downstream (Drizzle-compatibletag, §3.2). - OUTCOME_
RETENTION_ LMIDS - Retention bound for
ROOM_MUTATION_OUTCOMES_TABLErows, by lmid distance — never by time (Slice I-ii). When a room flush advances a(doc, client)ledger row tolmid, rows withmid ≤ lmid − Kprune in the same transaction.K = 512mirrors the room shell’s per-client recorded-outcome FIFO cap (MAX_RECORDED_OUTCOMES_PER_CLIENT = 512,packages/room/src/shell.ts) — the two ends of the outcome-resolution surface degrade at the same depth. The accepted loss class is the H-v one: a pruned mid reads asappliedthrough the daemon, exactly as an evicted map entry re-answers with silence on the room socket; a client is only ever that far behind its own ledger with a backlog ≥ K in flight. - ROOM_
CLIENT_ MUTATIONS_ TABLE - The domain-scoped ledger for room-flush lmid co-commits
(RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md §7.1). Keyed by
(doc, client_id)— one gapless mid stream per domain, one ledger row per domain — so a room flush and a slow-path daemon mutation for the SAME client never collide on a shared row (the Rev 1 data-loss bug, §8.5’s “ledger isolation” invariant).CLIENT_MUTATIONS_TABLEstays exclusively the slow-path stream; room flushes retarget here. - ROOM_
MUTATION_ OUTCOMES_ TABLE - The durable twin of the H-iv-b
mutationOutcomeframe ({mid, kind, reason?, name?, args?}), keyed(doc, client_id, mid)— the §4 lifecycle’s outcome-resolution surface for THE NAMED INVARIANT: never retire a room-domain entry off a daemon-carried lmid without outcome resolution (§3.3/§7.5). After downgrade the room socket that ordered outcome-before-ack is gone, so non-applied verdicts must be readable through the daemon subscription plane like the §7.1 ledger row. Rows are written by Slice I-ii’s flush split — this slice only creates + registers the table; an absent row for a covered mid reads asapplied(only non-applied outcomes are recorded, matching the room shell’s recorded-outcome map). - ROOM_
PLACEMENT_ TABLE - The room placement-fence table (
RINDLE-REALTIME-DESIGN.md§2.5): one row per doc, bumped by every claim. A flush carrying an epoch below the current claim is fenced — validated with the apply on the single-threaded engine, so the check and the commit are atomic. Unregistered bookkeeping likeSOURCE_OFFSETS_TABLE. - ROOM_
WATERMARK_ TABLE - The §4.2 downgrade fence:
(doc, flush_seq), co-committed monotonically in every room flush’s transaction (ClusterConsumer::commit_room_flush). Cross-authoritycvs are incomparable, so the fence is data that RIDES THE ECHO: a downgraded client keeps its frozen ghost source until its daemon subscription deliversflush_seq ≥ finalFlushSeq— proof the store it fell back to holds the room’s final flush, whatever the authority shape (single daemon / lagging read-follower / PG). - SCOPE_
SESSIONS_ TABLE - The §4.1 occupancy table (RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md): one row per
(scope, session), upserted by the api-server on every labeled lease mint/renewal and aged out lazily byexpires_at(mark/refresh/age-out — lease expiry needs no hook anywhere). The row delta IS the upgrade doorbell: a solo client’s only live connection is its daemon subscription, so the 1→2 wake signal must materialize as a row in the store it is subscribed to — which is whyDb::enable_realtime_lifecycleregisters this table rather than just creating it. - SOURCE_
OFFSETS_ TABLE - The durable per-change-source cursor table (CHANGE-SOURCE-DESIGN.md §4). One row per
source; the offset upsert is co-transactional with the batch it covers
(
ClusterConsumer::commit_normalized_with_offset). Daemon bookkeeping — NOT registered for capture (same posture as_rindle_sql_outcomes; the consumer’s own resume position is meaningless on any other host). - SOURCE_
OFFSET_ WHOLE_ RUN - The
chunk_seqsentinel meaning “the whole run at thisoffsetis durably applied” — the common case (every pure-row run and every run-boundary commit). A genuine value< thisis a mid-run checkpoint(offset, chunk_seq)left by the commit-at-DDL-boundary follower (RELAY-DDL-DESIGN.md §6.6): chunks0..=chunk_seqofoffsetare applied, the tail is not. The resume/dedup compare is the keyset(offset, chunk_seq), with the incomingbegin(R)treated as(R, WHOLE_RUN)— so a whole run sorts at/above any of its mid-run positions.i64::MAXis safe as a sentinel:chunk_seqis a 0-based within-run ordinal (one per spilled ≤CHUNK_ROWSchunk), so a real value reachingi64::MAXis physically impossible. Mirrors the relay fan-out’sScanPos“past the end of this run’s chunks” sentinel (rindle-replicator). - SQL_
IDEMPOTENCY_ MAX_ FUTURE_ SKEW_ MS - Public idempotency keys carry their mint time so an evicted outcome can fail closed without retaining one tombstone per operation. A modest future-skew allowance accommodates ordinary client clock drift while bounding how far one forged key can ratchet the durable floor.
- SQL_
IDEMPOTENCY_ PREFIX - The canonical public idempotency-key prefix:
sql1.<13-digit-unix-ms>.<32-lowercase-hex>. - SQL_
OUTCOMES_ TABLE - Exact replay records for the public SQL surface. The result is stored before commit;
run_idis known before the journal assigns a cid and resolves back to that journal point on replay. A zero-effect mutation uses a local metadata-only commit and therefore stores no run id/cursor. - SQL_
OUTCOME_ MAX_ RECORDS - Retained-outcome count ceiling; the sweep evicts oldest-first beyond it.
- SQL_
OUTCOME_ RETENTION_ MS - How long one stored public outcome is retained before the sweep may evict it.
Functions§
- adopt_
migration_ checksums - Backfill a migration row’s optional identities without overwriting values already made durable. Hosts own the surrounding transaction because producer metadata commits differently on HCTree and WAL2; the column contract is shared.
- append_
bounded_ script_ results - Append one batch’s results to a script’s partial-result set, charging the shared aggregate byte allowance (separators included) and failing with the public result-cap error on overflow.
- classify_
migration_ file - Classify one migration file’s statement vector as DDL or data, refusing mixed and unsupported statement classes.
- data_
migration_ checksum - The recorded content checksum of a DATA migration
tag, if one was applied (the kind-collision probe for the DDL surface’sMIGRATION_KIND_MISMATCHrefusal). - encoded_
statement_ result_ len - The encoded wire length of one statement result — the unit the aggregate result cap accounts in.
- enforce_
encoded_ result_ array_ limit - Enforce the aggregate cap over an already-encoded result array (the replay path’s check).
- ensure_
outcome_ matches_ request - Refuse a replay whose stored request identity differs from the retried request
(
OPERATION_ID_MISMATCH). - insert_
sql_ outcome - Insert one outcome row inside the caller’s open transaction (co-transactional with the
effects it records).
created_atis the host’snow_millis()— this module reads no clock. - insert_
sql_ outcome_ with_ cursor insert_sql_outcomewith an exact cursor written atomically. Standalone WAL2 commits use this form because their nextTxIdis known inside the open transaction; the HCTree master continues to insert a run id and backfill its engine-assigned cursor after commit.- journal_
data_ migration - Record a pure-data migration inside the caller’s captured transaction. The marker is the final captured row in that migration, so a zero-user-effect file is still durable and replay identity survives restore.
- journal_
migration - Record that
tag’sddlentry has been minted, inside the caller’s open transaction (so the journal can never disagree with the log entry it guards across a crash, §4). - journal_
migration_ with_ cursor journal_migrationwith the host’s exact commit cursor written in the same transaction. WAL2 authorities know their nextTxIdbefore COMMIT and use this form; HCTree keeps using the run-id form and resolves/backfills its engine-assigned cursor after commit.- migration_
content_ checksum_ from_ record - A record’s canonical content checksum: the stored one when present, else derived from the
normalized statements (the legacy pre-
content_checksumrow shape). - migration_
normalized_ statements_ from_ record - Decode and re-normalize the statement vector a migration record stored, if any.
- migration_
record - Whether migration
taghas already minted addllog entry (the producer idempotency check,MIGRATIONS-VIA-CHANGELOG-DESIGN.md§4). - parse_
public_ idempotency_ key_ at - Parse one canonical public idempotency key (
sql1.<13-digit-unix-ms>.<32-lowercase-hex>) against the caller’snow, returning its mint time. Future skew beyondSQL_IDEMPOTENCY_MAX_FUTURE_SKEW_MSis refused. - parse_
public_ or_ derived_ idempotency_ key_ at - Like
parse_public_idempotency_key_at, additionally admitting the server-derived script form (sql-script:<canonical-parent>:<index>). - public_
commit_ outcome_ on_ conn - Resolve the terminal outcome of a public transaction from committed metadata only.
- public_
idempotency_ minted_ at_ ms - Derived execute-multiple keys retain their parent public identity and therefore its floor timestamp. The suffix is server-created and never accepted directly at public intake.
- public_
transaction_ minted_ at_ ms - The mint time encoded in one public transaction id (
…-<hex-nanos>[-ro]), if parseable. A pure parser — the transaction-namespace seams stay host-side, but the outcome sweep consults this forsql-transaction:keys. - read_
sql_ outcome - Read one stored outcome row — a record read only, no decisions.
- sql_
operation_ key - The outcome-cache key of one public one-shot operation.
- sql_
outcome_ floor_ ms - The retention floor for
namespace: the durable ratcheted floor, never below the TTL horizon derived from the caller’snow_ms. - sql_
outcome_ minted_ at_ ms - The mint time carried by one stored
sql-operation:outcome key, if canonical. - sql_
request_ identity - The canonical request identity of a statement vector — the compact JSON encoding an outcome row stores so a replayed key can prove it carries the same request.
- sql_
result_ limit_ error - The public result-cap error, worded once.
- statement_
class_ name - The wire-visible human name of one statement class — embedded in stable public error messages, so it moves with the classifier rather than being re-derived per host.
- sweep_
sql_ outcomes - Evict expired and over-quota outcome rows and ratchet the durable per-namespace retention
floors.
nowis the host’s clock; runs inside the host’s own bookkeeping transaction.