Crate rindle_cdc_apply 

rindle-cdc-apply — the engine-free CDC apply plane (design 309)

“Apply a rindle CDC stream to a SQLite database” as a complete, closed capability — a follower without IVM. The CDC family has capture (rindle-cdc), transport (rindle-stream/rindle-replicator), and image distribution (rindle-cdc-image); this crate is the apply quarter:

• table registration — declared (ApplyConsumer::register_table) or introspected (ApplyConsumer::register_existing_table, with the _rindle_columns PG-mirror sidecar override);
• row apply — decode a normalized op, build its SQL, execute it on the single observed writer connection (ApplyConsumer::apply_muts);
• the cursor discipline — the _rindle_source_offsets upsert riding the SAME transaction as the effects (a crash can never land data without its cursor or vice-versa), run-fencing (run_id), mid-run chunk_seq, and the durable head stamps (ApplyConsumer::commit_follower_txn_with_head);
• replicated DDL — statements + idempotency marker + caller bookkeeping in ONE ordinary transaction, marker present ⇔ DDL applied exactly (ApplyStore::exec_ddl_with_marker);
• resume + portability — ApplyConsumer::source_checkpoint, ApplyStore::checkpoint_truncate, and (in rindle-writeplane) the restore bootstrap.

The two hosts

A headless consumer — rindle-backup-sqlite’s portable journal replay, or any external process mirroring a rindle stream into its own SQLite — opens with ApplyConsumer::open and composes nothing else: no worker threads, no engines, no drain. The live replica (rindle_replica::Cluster/ClusterConsumer) recomposes on top: its worker pool implements CommitFanout, and every apply method there DELEGATES here — one implementation, moved, never forked — so the live follower and a headless replay stay byte-identical by construction.

The seam is CommitFanout: the write transaction hands its captured chunks to whatever observes the commit, and absence of a derivation pool is a type (NoFanout), not a zero-worker parameter. Control flow is identical either way — the capture buffer still drains at PUSH_CHUNK_ROWS, the guards still gate every commit, the barriers just have no one to hold.

Capture (the rindle-cdc preupdate hook) is mandatory in both hosts: its divergence guards (uncaptured_user_event_count, capture-error poisoning) protect the headless paths too — a DDL bounce that misses a registration surfaces as uncaptured rows instead of silent drift.

The derivation (query) surface deliberately does not exist here — an ApplyConsumer has no query/read_snapshot/view_schema, so a host that cannot derive cannot express the call (the type system replaces the refusal rule, design 309 §3).

Structs

ApplyConsumer

The engine-free CDC apply consumer. !Send — lives on one thread, like the store under it. See the module docs; construction is open (headless) or from_parts (a derivation host composing over a shared store).

ApplyStore

The apply plane’s connection pair + capture context. !Send (the preupdate hook and the transaction cells are single-thread state) — one per thread, like the replica handles built over it.

ApplyTxn

An open write transaction on the store’s single writer connection. See the module docs; the derivation host’s ClusterWriteTxn is a thin public wrapper over this.

CommitInfo

What one committed transaction did, beyond its data effects: the new global tx id (the commit version cv the optimistic protocol stamps on outgoing batches). Client lmid advances are NOT reported here — _rindle_client_mutations rows ride the capture like any data and reach each client through its own system query (§8.2).

NoFanout

The headless fan-out: accepts pushes by dropping them, commits trivially. What a consumer that applies a foreign CDC stream with no derivation composes with — rindle-backup-sqlite’s portable replay, an external SQLite-mirroring consumer. tx_begin never fails, so the Failed arms of the transaction machinery are structurally unreachable headless.

SourceHead

Persisted source accounting at the applied cursor. Portable bases carry this so lag/change-count stamps resume from restored history instead of zero.

WalCheckpoint

One PRAGMA wal_checkpoint(TRUNCATE) outcome (ApplyStore::checkpoint_truncate).

Enums

DdlMigrationError

A standalone authority’s checked DDL apply can fail either in the cluster/schema machinery or in the shared write-plane policy checks. Keeping the latter typed preserves stable migration error codes all the way to the host’s HTTP renderer.

StreamBegin

The result of CommitFanout::tx_begin. Either way a stream handle is returned so whatever the observer pinned at the begin-barrier is always released.

Constants

ANALYSIS_LIMIT

Cap how many rows ANALYZE / PRAGMA optimize samples per index, so a maintenance tick’s stats refresh on the live writer stays sub-millisecond regardless of table size (see rindle-replica’s maintenance module, which restores this bound after its full-ANALYZE escape hatch). Applied to every connection opened here.

APPLIED_DDL_TABLE

The follower’s DDL idempotency journal: one row per applied ddl entry, keyed by the migration id (or the entry offset when a source ships none). Unregistered bookkeeping like the offsets table — never captured/fanned. A crash-window replay (the ddl re-delivered before its cursor advanced) dedups against this BEFORE re-applying, so an already-applied reshape is skipped exactly rather than re-run-and-inferred-from-the-error (see ApplyStore::exec_ddl_with_marker).

DEFAULT_WRITER_BEGIN_SQL

The BEGIN flavor a headless apply store’s writer transaction opens with — the same flavor the live rindle_replica::Cluster passes (306 S5 left BEGIN IMMEDIATE as the one writer flavor), so the headless applier and the live follower run byte-identical transactions.

PUSH_CHUNK_ROWS

Upper bound on one streamed fan-out chunk, in captured rows (CLUSTER-INCREMENTAL-FANOUT-DESIGN.md §4.2). Bounds the writer’s transient capture buffer — and, on the live replica, the per-worker transient output and the cross-thread message — to O(chunk) rather than O(txn); a txn ≤ this size streams as a single chunk. Owned by the apply plane because the between-statement drain (ApplyTxn::maybe_pump) is what enforces the writer-side bound, with or without a derivation pool on the other end.

Traits

CommitFanout

The observer of an apply store’s commits. Implemented by the live replica’s worker pool (fanning captured chunks out to the IVM workers behind the snapshot/commit handshake) and by NoFanout (a headless applier — backup replay, an external stream consumer — with no derivation to feed).

FanoutGate

The commit verdict for an in-flight fan-out, released after the writer’s durable COMMIT (or its failure).

FanoutStream

One in-flight transaction’s fan-out: bounded push chunks, then a terminal finish handing back the commit-verdict gate. Dropping it without finish is an abort — the implementation must release anything it pinned.

Functions

client_mutations_table_meta

The registered-table metadata for [CLIENT_MUTATIONS_TABLE] — one literal, shared by the headless enable and the live ClusterConsumer’s so the two can never drift.

ensure_source_offsets_table_on_conn

source_offsets_table_ddl applied to a caller-owned connection (the initial-snapshot store writes it before any consumer exists).

ensure_unique_pk_index_for

Ensure table has a UNIQUE index covering exactly the PK columns, which row-identity point lookups (the engine’s TableSource, the follower’s apply keys) require. Idempotent. column_names is the table’s columns in cid order; primary_key holds indexes into it.

open_journal

Open one read-write connection (mirrors Db::open’s per-connection setup). Used for the apply store’s writer, the cluster coordinator’s writer, and each IVM worker.

open_journal_read_only

Open a physically read-only connection for the public read surface. PRAGMA query_only is only a mutable connection setting; opening with SQLITE_OPEN_READ_ONLY is what prevents a callback from turning it off and mutating the durable database behind CDC.

set_wal_autocheckpoint

Set PRAGMA wal_autocheckpoint = pages on one connection — the WAL-growth knob a host that wants to own its checkpoint schedule passes at open (OpenOptions::wal_autocheckpoint; design 410 §3.5).

source_offsets_table_ddl

The _rindle_source_offsets DDL — one string, shared with the snapshot/restore stores that mint the table on a bare connection so it cannot drift from the consumer’s own ensure_source_offsets_table.

upsert_source_offset

Upsert the durable (offset, chunk_seq) checkpoint for source into txn (co-transactional with the effects that txn carries — a crash can never land data without its cursor, §4). Shared by the whole-batch (ApplyConsumer::commit_normalized_with_offset) and streaming-follower (ApplyConsumer::commit_follower_txn) commit paths. A whole-run commit passes [SOURCE_OFFSET_WHOLE_RUN]; a mid-run segment passes the boundary chunk’s chunk_seq. run_id is the run’s identity token from its begin frame (the fencing proof echoed on the next subscribe — RELAY-CURSOR-EPOCH-FENCING-DESIGN.md §2); None (a pre-fence upstream or a non-streaming source) stores NULL, which subscribes without a fence as before.

upsert_source_offset_hashed

upsert_source_offset carrying the §8.3 batch identity: the emitter-computed batch_hash of the run’s exact bytes, stored beside the cursor so a same-offset resubmission with a different body is a loud error, never a silent dedup (T6). The hash-less paths (follower stream, plain change sources) write NULL — once the cursor advances, the previous offset’s identity is dead anyway.