High-concurrency runtime

Run the daemon (rindled)

Run rindled as a source-less standalone authority or a read-only fleet follower: config, network planes, recovery contract, and the multi-threaded Cluster underneath.

View as Markdown

rindled is the always-up IVM server you run like Postgres. It has two explicit postures:

  • Standalone authority — one source-less daemon owns a wal2 SQLite file and serves reads, writes, migrations, SQL, materializations, and subscriptions at one origin.
  • Fleet follower — the daemon holds a read-only wal2 replica, derives deltas from a rindle-replicator master’s ordered stream, and serves reads and subscriptions. Every write endpoint — and the whole public /v1/sql surface, reads included — remains fenced with a fail-closed 409 that names the master.

The browser subscribes to the daemon’s public WebSocket. Your API server authorizes named operations and drives its private HTTP control plane. In standalone, its read and write transports point at the same origin; in a fleet, a split client sends writes to the master and reads to a follower.

See the architecture for how the three tiers fit and deploying & scaling for the durability and scale tradeoffs of each posture. This page covers the daemon itself.

The binary

rindled lives in the rindle-server crate and ships as a prebuilt, per-platform binary with @rindle/cli. For local dev you rarely invoke it directly — rindle dev renders your rindle.ncl, supervises the selected profile, and runs your application with topology-derived bindings. Use rindle up when you deliberately want only the data tier. It’s the first thing the synced-app quickstart does. This page is the daemon in full — the config, the two planes, restart recovery, and the engine underneath — for when you run it yourself.

To run a daemon directly (in production, or under your own supervisor), point a release, container, or otherwise supervised rindled binary at a JSON config:

rindled --config follower.json
# …or, with a Rindle source license (rindle-server is a commercial crate):
cargo build -p rindle-server --bin rindled --release
./target/release/rindled --config follower.json

The config declares its file, ports, auth, worker count, and posture. A follower has one change source — the write-master’s fan-out stream:

{
  "db": "follower.db",
  "httpPort": 7600,
  "wsPort": 7601,
  "authToken": "dev-daemon-token",
  "nWorkers": 4,
  "sources": [
    { "kind": "replicator", "name": "rindle-master", "url": "ws://127.0.0.1:7610/subscribe" }
  ]
}

A standalone config is deliberately source-less and opts in explicitly. It needs a second, distinct bearer for the public SQL surface when exposed beyond loopback:

{
  "db": "standalone.db",
  "httpPort": 7600,
  "wsPort": 7601,
  "bindHost": "127.0.0.1",
  "authToken": "private-control-token",
  "sqlAuthToken": "server-sql-token",
  "nWorkers": 4,
  "standalone": true,
  "sources": [],
  "tables": []
}

Create application tables through migrations. The tables field is primarily useful to bootstrap a directly spawned daemon in a test or embedding.

  • db — the file-backed wal2 SQLite database (defaults to rindle.db). It is the authority in standalone and a replica in follower posture.
  • httpPort / wsPort — the control and subscription ports. 0 binds an ephemeral port (handy in tests — the chosen port comes back in the readiness signal).
  • authToken — the private control-plane bearer. A non-loopback bind refuses to start without it.
  • sqlAuthToken — the public /v1/sql/* bearer. Standalone requires this surface for @rindle/sql-client; it must differ from authToken when both are present.
  • nWorkers — IVM worker threads in the underlying Cluster (defaults to 2).
  • defaultLeaseTtlMs — how long a materialization lease lives without renewal.
  • standalone / sources — choose exactly one posture. standalone: true requires zero sources. Follower posture requires exactly one kind:"replicator" source. Omitting the standalone flag never guesses from an empty list; it stays a follower and fails closed. A follower receives schema DDL from its master, while a standalone daemon applies it locally through /migrate.

On a successful start rindled prints exactly one line of JSON to stdout, so a supervisor or test runner can wait on it:

{"ready":true,"httpPort":7600,"wsPort":7601}

Two listeners and trust surfaces

The daemon exposes two network listeners, kept separate so the untrusted browser plane and trusted server-to-server traffic never share a door:

Plane Port Who connects Carries
Public WebSocket wsPort browser clients the normalized protocol — init, subscribe / unsubscribe, and cv-stamped snapshot + delta frames out
Private HTTP control httpPort your API server / operator all postures: /materialize, /execute-sql-read, /dematerialize, /schema, /version; standalone also: /execute-sql-txn, /mutate-session/*, /reject-mutation, /migrate
Public SQL over HTTP httpPort trusted SQL clients / API server /v1/sql/*, protected independently by sqlAuthToken; standalone owns reads and writes here — a follower refuses the entire surface

In follower posture, the write endpoints live on the rindle-replicator master. If you point a write at the follower, it refuses it with a fail-closed error that names the master. A misrouted write cannot silently vanish. The same 409 fences the entire public /v1/sql surface, reads included: session-consistency cursors are minted in the master’s journal sequence, which a follower’s local commit counter cannot fence, so honoring them would return silently stale rows or spurious CURSOR_HISTORY_LOST. Send all /v1/sql traffic to the write-master — in a fleet, the unified edge does this for you. Standalone enables its mutation, migration, session, and public SQL write surfaces, but not fleet change-source ingress such as /apply-row-change-txn.

The control routes require authToken when one is set; /v1/sql/* requires the separate sqlAuthToken. These are operator-level topology details. A fleet edge can expose one RINDLE_URL and one server-only RINDLE_DATABASE_TOKEN, routing each call internally. A loopback standalone rindle dev connection also exports one application origin; a directly networked standalone API server configures its private daemon client and public SQL/database transport with their respective bearers. Browsers never speak either trusted surface directly. The low-level @rindle/daemon-client package remains the typed client for custom self-hosted routing and supervisor integrations.

Standalone durability contract

Standalone deliberately has no HCTree journal, follower fan-out, rindle-backup plane, PITR, or automatic failover. Desktop/local software owns an ordinary SQLite snapshot (VACUUM INTO, the backup API, or an OS backup) and its RPO. A hosted micro deployment keeps one persistent volume and restores a completed provider volume snapshot into a replacement instance; recovery has downtime and may lose writes back to the advertised snapshot interval.

A product-initiated snapshot before a destructive migration or risky upgrade must stop intake, drain or roll back open sessions, and quiesce/close the writer first. Provider-scheduled snapshots should be advertised only after a restore drill has captured an active wal2 database, restored it to a new volume, passed SQLite integrity checks, and re-proved view-after-write equals a fresh query.

A standalone wal2 file cannot be promoted in place to an HCTree master. Moving to a fleet is an export/import into a new store followed by a routing cutover. Choose the fleet profile from the start when continuous journal recovery, followers, or read fan-out are requirements.

Restart recovery: the boot id

rindled keeps no durable materialization state — on restart it has the data (it’s file-backed) but no live queries or pins. So it stamps every control-plane response with a boot id header that changes when it restarts. The HttpRindleDaemonClient surfaces it via onBootId, and your API server re-asserts its pinned queries when it fires:

const daemonToken = process.env.RINDLE_DAEMON_TOKEN ?? process.env.RINDLE_DATABASE_TOKEN!;
const daemon = new HttpRindleDaemonClient({
  baseUrl: process.env.RINDLE_URL!,
  headers: { authorization: `Bearer ${daemonToken}` },
  onBootId: () => api.assertPins().catch(console.error), // re-warm after a restart
});

The hook rides responses you already make, so there’s no polling — the next control-plane call after a restart re-establishes the warm set.

Under the hood: the Cluster

rindled runs the multi-threaded Cluster engine from rindle-replica. Where the single-thread Db advances every query on one thread, Cluster shards queries across a pool of IVM worker threads behind a single writer/coordinator. In standalone, writes serialize on that wal2 writer, while each read-only public SQL transaction runs beside it on its own wal2 reader snapshot — see Rindle SQL. On a follower, pre-formed transactions arrive over the master’s stream. The same per-transaction handshake keeps IVM derivation correct:

  1. The coordinator opens the write transaction. In standalone the preupdate hook captures row deltas as authoritative SQL runs; a follower applies its incoming row changes.
  2. commit fans the captured batch to every worker and waits for each to pin its own pre-commit snapshot (the barrier).
  3. The writer commits durably while the workers derive their queries’ deltas concurrently under snapshot isolation.
  4. Each worker emits its affected queries’ deltas, then a progress marker so the drain layer knows the transaction is fully delivered.

A query lives on exactly one worker, so per-query event order is preserved. If a worker faults during derivation, that query is torn down (and the pool respawns the worker) rather than corrupting the stream. The contract is unchanged from the single-thread path: view-after-write == fresh-query. See crates for the Cluster API.

The query planner

The daemon runs the cost-based join-flip planner — it annotates each flippable EXISTS with a flip decision before lowering, and picks the cheaper drive side from a real-SQLite cost model. It is result-preserving (only the work changes, never the rows) and is on by default (Cluster::open enables it). Opting out is in-process only today, via Cluster::open_with_planning(path, n, false), not yet through the config file.

Scope

rindled is the productionizing read server, but it is young. The replica’s schema constraints apply — plain tables (no triggers / generated columns), numbers within ±(2⁵³−1). See replica and views. The database bearer is server-wide. Finer-grained authz lives in your API tier.

This page runs one daemon. Standalone is intentionally one authority and one file; starting a second writable copy would create two authorities, not a replica. To add continuous journal recovery or scale reads across affinity-placed followers, move to the fleet profile described in deploying & scaling. That move is an export/import into a new HCTree store followed by a routing cutover, not an in-place promotion of the standalone wal2 file.

Next steps