Rindle docs and package mapSkip to main content

Module bookkeeping

Module bookkeeping 

Expand description

The public-SQL bookkeeping layer every one-shot write host shares (303-PUBLIC-SQL-COORDINATOR-EXTRACTION-PLAN.md, PR 0 — decision C2): the exact-replay outcome cache, the migrations journal, the idempotency key/identity family, and the result-limit family, moved wholesale from the replicator. Conn-bound functions included — this crate links SQLite — so the coordinator’s control flow never reaches back across the crate boundary for a helper.

Two rules hold module-wide:

  • No clock, anywhere. Every function that needs time takes it as a parameter (created_at, now, now_ms); the hosts pass their own now_millis() and tests script the values.
  • Every fallible function returns BookkeepingError — the narrow currency that converts losslessly into both the replicator’s MasterError and the coordinator’s WritePlaneError.

Structs§

MigrationRecord
One row of the migrations journal — the durable identity a replayed migration tag is validated against. Every decision over it belongs to the caller; the readers here only decode.
StoredPublicOutcome
One stored exact-replay outcome of the public SQL surface (the replicator’s SqlOutcomeRecord, renamed on the move — extraction plan C2).

Enums§

MigrationFileKind
The classification of one migration file: only-DDL or only-data (mixed files are refused).
SqlOutcomeNamespace
The two durable retention-floor namespaces of the outcome cache: one-shot operations (sql-operation: keys) and public transactions (sql-transaction: keys).

Constants§

CLIENT_MUTATIONS_TABLE
The replicated bookkeeping table carrying each client’s high-water mutation id. Single-_ prefixed: captured by CDC (unlike __replica_meta) and hosted by the engine like any base table — each client’s lmid flows to it through its own one-row system query, transactionally with the data (§8.2).
DATA_MIGRATIONS_TABLE
Replicated apply-once journal for pure-DML migration files. Unlike the DDL journal this table is part of the capture registry: its marker row is the final change in every data-migration run, making zero-user-effect files durable and carrying identity through restore/promotion.
LOG_META_TABLE
A tiny single-row-per-key bookkeeping table. Holds gc_floor — the highest cursor the GC has reclaimed, persisted so the fan-out’s cursor-too-old check survives restart and never regresses (CHANGE-SOURCE-DESIGN.md §10.2) — and the durable SQL-outcome retention floors (SqlOutcomeNamespace::floor_key). Not registered for capture.
MIGRATIONS_JOURNAL_TABLE
The producer-side migration journal (MIGRATIONS-VIA-CHANGELOG-DESIGN.md §4): a master’s record of “have I already minted a ddl journal frame for this migration id.” It is the idempotency key for the migrate surface (a re-POST of a known tag is a no-op), NOT the ordered transport — that is the journal’s ddl frame. Host-local, NOT registered for capture, so it never ships downstream (Drizzle-compatible tag, §3.2).
OUTCOME_RETENTION_LMIDS
Retention bound for ROOM_MUTATION_OUTCOMES_TABLE rows, by lmid distance — never by time (Slice I-ii). When a room flush advances a (doc, client) ledger row to lmid, rows with mid ≤ lmid − K prune in the same transaction. K = 512 mirrors the room shell’s per-client recorded-outcome FIFO cap (MAX_RECORDED_OUTCOMES_PER_CLIENT = 512, packages/room/src/shell.ts) — the two ends of the outcome-resolution surface degrade at the same depth. The accepted loss class is the H-v one: a pruned mid reads as applied through the daemon, exactly as an evicted map entry re-answers with silence on the room socket; a client is only ever that far behind its own ledger with a backlog ≥ K in flight.
ROOM_CLIENT_MUTATIONS_TABLE
The domain-scoped ledger for room-flush lmid co-commits (RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md §7.1). Keyed by (doc, client_id) — one gapless mid stream per domain, one ledger row per domain — so a room flush and a slow-path daemon mutation for the SAME client never collide on a shared row (the Rev 1 data-loss bug, §8.5’s “ledger isolation” invariant). CLIENT_MUTATIONS_TABLE stays exclusively the slow-path stream; room flushes retarget here.
ROOM_MUTATION_OUTCOMES_TABLE
The durable twin of the H-iv-b mutationOutcome frame ({mid, kind, reason?, name?, args?}), keyed (doc, client_id, mid) — the §4 lifecycle’s outcome-resolution surface for THE NAMED INVARIANT: never retire a room-domain entry off a daemon-carried lmid without outcome resolution (§3.3/§7.5). After downgrade the room socket that ordered outcome-before-ack is gone, so non-applied verdicts must be readable through the daemon subscription plane like the §7.1 ledger row. Rows are written by Slice I-ii’s flush split — this slice only creates + registers the table; an absent row for a covered mid reads as applied (only non-applied outcomes are recorded, matching the room shell’s recorded-outcome map).
ROOM_PLACEMENT_TABLE
The room placement-fence table (RINDLE-REALTIME-DESIGN.md §2.5): one row per doc, bumped by every claim. A flush carrying an epoch below the current claim is fenced — validated with the apply on the single-threaded engine, so the check and the commit are atomic. Unregistered bookkeeping like SOURCE_OFFSETS_TABLE.
ROOM_WATERMARK_TABLE
The §4.2 downgrade fence: (doc, flush_seq), co-committed monotonically in every room flush’s transaction (ClusterConsumer::commit_room_flush). Cross-authority cvs are incomparable, so the fence is data that RIDES THE ECHO: a downgraded client keeps its frozen ghost source until its daemon subscription delivers flush_seq ≥ finalFlushSeq — proof the store it fell back to holds the room’s final flush, whatever the authority shape (single daemon / lagging read-follower / PG).
SCOPE_SESSIONS_TABLE
The §4.1 occupancy table (RINDLE-REALTIME-QUERY-ENABLEMENT-DESIGN.md): one row per (scope, session), upserted by the api-server on every labeled lease mint/renewal and aged out lazily by expires_at (mark/refresh/age-out — lease expiry needs no hook anywhere). The row delta IS the upgrade doorbell: a solo client’s only live connection is its daemon subscription, so the 1→2 wake signal must materialize as a row in the store it is subscribed to — which is why Db::enable_realtime_lifecycle registers this table rather than just creating it.
SOURCE_OFFSETS_TABLE
The durable per-change-source cursor table (CHANGE-SOURCE-DESIGN.md §4). One row per source; the offset upsert is co-transactional with the batch it covers (ClusterConsumer::commit_normalized_with_offset). Daemon bookkeeping — NOT registered for capture (same posture as _rindle_sql_outcomes; the consumer’s own resume position is meaningless on any other host).
SOURCE_OFFSET_WHOLE_RUN
The chunk_seq sentinel meaning “the whole run at this offset is durably applied” — the common case (every pure-row run and every run-boundary commit). A genuine value < this is a mid-run checkpoint (offset, chunk_seq) left by the commit-at-DDL-boundary follower (RELAY-DDL-DESIGN.md §6.6): chunks 0..=chunk_seq of offset are applied, the tail is not. The resume/dedup compare is the keyset (offset, chunk_seq), with the incoming begin(R) treated as (R, WHOLE_RUN) — so a whole run sorts at/above any of its mid-run positions. i64::MAX is safe as a sentinel: chunk_seq is a 0-based within-run ordinal (one per spilled ≤CHUNK_ROWS chunk), so a real value reaching i64::MAX is physically impossible. Mirrors the relay fan-out’s ScanPos “past the end of this run’s chunks” sentinel (rindle-replicator).
SQL_IDEMPOTENCY_MAX_FUTURE_SKEW_MS
Public idempotency keys carry their mint time so an evicted outcome can fail closed without retaining one tombstone per operation. A modest future-skew allowance accommodates ordinary client clock drift while bounding how far one forged key can ratchet the durable floor.
SQL_IDEMPOTENCY_PREFIX
The canonical public idempotency-key prefix: sql1.<13-digit-unix-ms>.<32-lowercase-hex>.
SQL_OUTCOMES_TABLE
Exact replay records for the public SQL surface. The result is stored before commit; run_id is known before the journal assigns a cid and resolves back to that journal point on replay. A zero-effect mutation uses a local metadata-only commit and therefore stores no run id/cursor.
SQL_OUTCOME_MAX_RECORDS
Retained-outcome count ceiling; the sweep evicts oldest-first beyond it.
SQL_OUTCOME_RETENTION_MS
How long one stored public outcome is retained before the sweep may evict it.

Functions§

adopt_migration_checksums
Backfill a migration row’s optional identities without overwriting values already made durable. Hosts own the surrounding transaction because producer metadata commits differently on HCTree and WAL2; the column contract is shared.
append_bounded_script_results
Append one batch’s results to a script’s partial-result set, charging the shared aggregate byte allowance (separators included) and failing with the public result-cap error on overflow.
classify_migration_file
Classify one migration file’s statement vector as DDL or data, refusing mixed and unsupported statement classes.
data_migration_checksum
The recorded content checksum of a DATA migration tag, if one was applied (the kind-collision probe for the DDL surface’s MIGRATION_KIND_MISMATCH refusal).
encoded_statement_result_len
The encoded wire length of one statement result — the unit the aggregate result cap accounts in.
enforce_encoded_result_array_limit
Enforce the aggregate cap over an already-encoded result array (the replay path’s check).
ensure_outcome_matches_request
Refuse a replay whose stored request identity differs from the retried request (OPERATION_ID_MISMATCH).
insert_sql_outcome
Insert one outcome row inside the caller’s open transaction (co-transactional with the effects it records). created_at is the host’s now_millis() — this module reads no clock.
insert_sql_outcome_with_cursor
insert_sql_outcome with an exact cursor written atomically. Standalone WAL2 commits use this form because their next TxId is known inside the open transaction; the HCTree master continues to insert a run id and backfill its engine-assigned cursor after commit.
journal_data_migration
Record a pure-data migration inside the caller’s captured transaction. The marker is the final captured row in that migration, so a zero-user-effect file is still durable and replay identity survives restore.
journal_migration
Record that tag’s ddl entry has been minted, inside the caller’s open transaction (so the journal can never disagree with the log entry it guards across a crash, §4).
journal_migration_with_cursor
journal_migration with the host’s exact commit cursor written in the same transaction. WAL2 authorities know their next TxId before COMMIT and use this form; HCTree keeps using the run-id form and resolves/backfills its engine-assigned cursor after commit.
migration_content_checksum_from_record
A record’s canonical content checksum: the stored one when present, else derived from the normalized statements (the legacy pre-content_checksum row shape).
migration_normalized_statements_from_record
Decode and re-normalize the statement vector a migration record stored, if any.
migration_record
Whether migration tag has already minted a ddl log entry (the producer idempotency check, MIGRATIONS-VIA-CHANGELOG-DESIGN.md §4).
parse_public_idempotency_key_at
Parse one canonical public idempotency key (sql1.<13-digit-unix-ms>.<32-lowercase-hex>) against the caller’s now, returning its mint time. Future skew beyond SQL_IDEMPOTENCY_MAX_FUTURE_SKEW_MS is refused.
parse_public_or_derived_idempotency_key_at
Like parse_public_idempotency_key_at, additionally admitting the server-derived script form (sql-script:<canonical-parent>:<index>).
public_commit_outcome_on_conn
Resolve the terminal outcome of a public transaction from committed metadata only.
public_idempotency_minted_at_ms
Derived execute-multiple keys retain their parent public identity and therefore its floor timestamp. The suffix is server-created and never accepted directly at public intake.
public_transaction_minted_at_ms
The mint time encoded in one public transaction id (…-<hex-nanos>[-ro]), if parseable. A pure parser — the transaction-namespace seams stay host-side, but the outcome sweep consults this for sql-transaction: keys.
read_sql_outcome
Read one stored outcome row — a record read only, no decisions.
sql_operation_key
The outcome-cache key of one public one-shot operation.
sql_outcome_floor_ms
The retention floor for namespace: the durable ratcheted floor, never below the TTL horizon derived from the caller’s now_ms.
sql_outcome_minted_at_ms
The mint time carried by one stored sql-operation: outcome key, if canonical.
sql_request_identity
The canonical request identity of a statement vector — the compact JSON encoding an outcome row stores so a replayed key can prove it carries the same request.
sql_result_limit_error
The public result-cap error, worded once.
statement_class_name
The wire-visible human name of one statement class — embedded in stable public error messages, so it moves with the classifier rather than being re-derived per host.
sweep_sql_outcomes
Evict expired and over-quota outcome rows and ratchet the durable per-namespace retention floors. now is the host’s clock; runs inside the host’s own bookkeeping transaction.