Rindle

Concepts

How it works

The five-step lifecycle every crate shares — build a query, lower it into a graph, add a view, hydrate, push and flush — and why incremental beats recompute.

Rindle keeps a query result current by incremental view maintenance (IVM): you build a query, hydrate a materialized view from it, push the changes that happen to the underlying tables, and the view updates by the difference — never by re-running the query. The contract is view-after-push == fresh-query: applying the change stream leaves the view exactly equal to running the query from scratch.

Under the hood, Rindle is the rindle engine (with optional rindle_replica and rindle_sqlite crates). This page walks the data path end to end. For the shapes a query may take, see Supported queries; for the change stream itself, see Change model.

The data path

A query travels through four stages:

  1. Build an Ast — either with Deltic, the fluent rindle::table query builder, or by deserializing the engine’s JSON wire format.
  2. Lower that Ast into a wired dataflow Graph of operators with rindle::build_pipeline.
  3. Hydrate a View over the graph to materialize the initial result set.
  4. Push source changes and read the incrementally-maintained ViewData.

1. Build a query

The fluent builder produces an Ast. Each method returns the builder, and build finishes it:

use rindle::table;

let ast = table("issue")
    .r#where("open", true)
    .build();

r#where takes a field name and a value. build consumes the builder and returns the Ast. (r#where is spelled with the raw-identifier escape because where is a Rust keyword.)

You can also deserialize an Ast directly from its JSON wire format via serde, which is how a JS client hands a query to the engine.

2. Lower it into a graph

rindle::build_pipeline walks the Ast and allocates the operator graph. Its signature is:

pub fn build_pipeline(
    graph: &mut rindle::Graph,
    ast: &rindle::Ast,
    resolve: &impl Fn(&str) -> Option<(rindle::NodeId, rindle::Schema)>,
) -> Result<rindle::NodeId, rindle::BuildError>;

You first register each base table as a source on the Graph, then pass a resolve closure that maps a table name to its source NodeId and Schema. build_pipeline returns the NodeId of the pipeline’s top operator (or a BuildError if the Ast is outside the supported subset):

use std::collections::HashMap;
use rindle::{build_pipeline, table, Graph, NodeId, Schema};

let mut graph = Graph::new();

// Register the base table as an in-memory source. `Schema::new` takes the
// columns, the primary-key column indices, and the sort.
let schema = Schema::new(vec!["id", "open"], vec![0], vec![(0, true)]);
let issue_src: NodeId = graph.add_source(schema.clone(), Vec::new());

// `resolve` maps each table name to (source NodeId, schema).
let mut sources: HashMap<&str, (NodeId, Schema)> = HashMap::new();
sources.insert("issue", (issue_src, schema.clone()));
let resolve = |name: &str| sources.get(name).cloned();

let ast = table("issue").r#where("open", true).build();
let top: NodeId = build_pipeline(&mut graph, &ast, &resolve)
    .expect("build the pipeline");

In production prefer the fallible Graph::try_add_source, which validates ingest row widths and returns a RindleError instead of panicking on a malformed row.

3. Hydrate a view

A View is the materialized sink. Add it over the pipeline’s top operator with Graph::add_view, wire the final edge with Graph::set_sink_edge, then hydrate:

let view = graph.add_view(top, schema.clone());
graph.set_sink_edge(top, view);
graph.hydrate(view);

// Read the materialized tree. `ViewData` is `List(..)` or `One(..)`.
let data = graph.view_data(view);

Graph::hydrate drains the input pipeline once to build the initial result. As with push, there is a fallible peer, Graph::try_hydrate, that returns Result<(), RindleError> — use it in a server.

4. Push changes

A mutation to a base table is a rindle::SourceChange. There are exactly three:

pub enum SourceChange {
    Add(OwnedRow),
    Remove(OwnedRow),
    Edit { row: OwnedRow, old: OwnedRow },
}

Push one with Graph::source_push, then flush_view to close the transaction and fire the view’s listeners:

use rindle::{value::owned_row, OwnedValue, SourceChange};

graph.source_push(
    issue_src,
    SourceChange::Add(owned_row(vec![OwnedValue::Int(7), OwnedValue::Bool(true)])),
);
graph.flush_view(view);

let updated = graph.view_data(view); // reflects the new row

The cost of the push is proportional to the change, not to the table size. In a server, drive mutations through the fallible Graph::try_source_push (returns RindleError); the infallible source_push / hydrate shown above .expect the result and are for tests and prototyping. See the crate’s fault model for the full survivability contract.

The engine is single-threaded by design: one Graph per thread (it is !Send). Scale with N independent graphs and message passing, never a shared Arc<Mutex<Graph>>.

Query planner (opt-in)

Some queries can be lowered more than one way. A correlated EXISTS can be driven from the parent (filter the parent, probe the child) or flipped to be driven from the child (stream matching children up to their parents) — and which is cheaper depends on the data. Rindle ships a cost-based planner (rindle-planner) that picks.

Planning is a pure Ast → Ast step that runs before lowering (step 2 above): it annotates each flippable EXISTS with a flip decision and changes nothing else. It is result-preserving — the flipped and unflipped plans return the same rows; only the work differs. The replica wrappers expose it as an opt-in at open time, and leave it off by default:

// single-threaded:
let db = rindle_replica::Db::open_with_planning("app.db", true)?;
// multi-threaded (the engine rindled runs):
let (cluster, events) = rindle_replica::Cluster::open_with_planning("app.db", 4, true)?;

The cost model behind it is a real-SQLite model in rindle-sqlite — it reads SQLite’s own scan-status and table statistics rather than guessing. See run the daemon for where planning fits in a deployment.

Driving it from a database: rindle_replica

Wiring sources, resolves, and views by hand is the low-level path. The rindle_replica crate packages the whole lifecycle over a SQLite database: open a file, register the tables a query touches, register the query, and subscribe to its change stream.

use rindle_replica::{Db, QueryId, Update};
use rindle::table;

let db = Db::open("app.db")?;
db.register_table("issue")?;

// `Db::query` takes a caller-chosen `QueryId` tag and an `Ast` (build it with
// `rindle::table(..)`), not a SQL string.
let ast = table("issue").r#where("open", true).build();
let query = db.query(QueryId(1), ast)?;

// `subscribe` takes a callback. It fires once immediately with `Update::Hydrated`
// (the initial set), then `Update::Changed` after every committed write that
// affects this query.
query.subscribe(|update: &Update| match update {
    Update::Hydrated { tx_id, changes } => {
        println!("hydrated @ {tx_id:?}: {} change(s)", changes.len());
    }
    Update::Changed { tx_id, changes } => {
        println!("changed @ {tx_id:?}: {} change(s)", changes.len());
    }
});

Each changes element is a rindle_replica::ChangeEvent (the engine’s owned CaughtChange) — the incremental delta, not a re-fetch. Its variants mirror the engine’s change model: Add, Remove, Edit, and Child (for a change to a nested relationship). The variants are not Insert/Delete/Update.

Writes go through a single controlled writer. Db::write opens a transaction and returns a WriteTxn; its commit derives each query’s incremental events against the pre-commit snapshot and delivers them to subscribers:

let mut tx = db.write()?;
tx.exec_batch("INSERT INTO issue (id, open) VALUES (8, 1)")?;
tx.commit()?; // derives + delivers `Update::Changed` to affected subscribers

This one-writer, write-then-abort model is what makes a delivered event exactly equal a freshly-hydrated query. For the subscription shapes and how a view stays in sync, see Replica and views.

Where to go next