Rindle

The engine

How it works

The five-step lifecycle every crate shares — build a query, lower it into a graph, add a view, hydrate, push and flush — and why incremental beats recompute.

View as Markdown

Rindle keeps a query result current by incremental view maintenance (IVM): you build a query, hydrate a materialized view from it, push the changes that happen to the underlying tables, and the view updates by the difference — never by re-running the query. The contract is view-after-push == fresh-query: applying the change stream leaves the view exactly equal to running the query from scratch.

Under the hood this is the open-source rindle engine (Apache-2.0), with the rindle-sqlite backend when your sources are SQLite tables. This page walks the data path end to end. For the shapes a query may take, see Supported queries; for the change stream itself, see Change model.

The data path

A query travels through four stages:

  1. Build an Ast — either with Deltic, the fluent rindle::table query builder, or by deserializing the engine’s JSON wire format.
  2. Lower that Ast into a wired dataflow Graph of operators with rindle::build_pipeline.
  3. Hydrate a View over the graph to materialize the initial result set.
  4. Push source changes and read the incrementally-maintained ViewData.

1. Build a query

The fluent builder produces an Ast. Each method returns the builder, and build finishes it:

use rindle::table;

let ast = table("issue")
    .r#where("open", true)
    .build();

r#where takes a field name and a value. build consumes the builder and returns the Ast. (r#where is spelled with the raw-identifier escape because where is a Rust keyword.)

You can also deserialize an Ast directly from its JSON wire format via serde, which is how a JS client hands a query to the engine.

2. Lower it into a graph

rindle::build_pipeline walks the Ast and allocates the operator graph. Its signature is:

pub fn build_pipeline(
    graph: &mut rindle::Graph,
    ast: &rindle::Ast,
    resolve: &impl Fn(&str) -> Option<(rindle::NodeId, rindle::SourceSchema)>,
) -> Result<rindle::NodeId, rindle::BuildError>;

You first register each base table as a source on the Graph, then pass a resolve closure that maps a table name to its source NodeId and SourceSchema — the table-registration type (columns, primary key, default sort; deliberately no relationships, which are a property of a query). build_pipeline returns the NodeId of the pipeline’s top operator (or a BuildError if the Ast is outside the supported subset):

use std::collections::HashMap;
use rindle::{build_pipeline, table, Graph, NodeId, SourceSchema};

let mut graph = Graph::new();

// Register the base table as an in-memory source. `SourceSchema::new` takes the
// columns, the primary-key column indices, and the table's default sort.
let schema = SourceSchema::new(vec!["id", "open"], vec![0], vec![(0, true)]);
let issue_src: NodeId = graph.add_source(schema.clone(), Vec::new());

// `resolve` maps each table name to (source NodeId, SourceSchema).
let mut sources: HashMap<&str, (NodeId, SourceSchema)> = HashMap::new();
sources.insert("issue", (issue_src, schema));
let resolve = |name: &str| sources.get(name).cloned();

let ast = table("issue").r#where("open", true).build();
let top: NodeId = build_pipeline(&mut graph, &ast, &resolve)
    .expect("build the pipeline");

In production prefer the fallible Graph::try_add_source, which validates ingest row widths and returns a RindleError instead of panicking on a malformed row.

3. Hydrate a view

A View is the materialized sink; it takes the richer Schema type (sort, relationship slots, singularity), which rindle::view_schema derives from the Ast and the same resolve — one derivation, shared with the wasm client. Add the view over the pipeline’s top operator with Graph::add_view, wire the final edge with Graph::set_sink_edge, then hydrate:

use rindle::view_schema;

let vschema = view_schema(&ast, &resolve).expect("derive the view schema");
let view = graph.add_view(top, vschema);
graph.set_sink_edge(top, view);
graph.hydrate(view);

// Read the materialized result: a reference-stable list of `Entry`s — each an
// owned `row` plus, for nested queries, child `rels`. Iterate `data.items`.
let data = graph.view_data(view);

Graph::hydrate drains the input pipeline once to build the initial result. As with push, there is a fallible peer, Graph::try_hydrate, that returns Result<(), RindleError> — use it in a server.

4. Push changes

A mutation to a base table is a rindle::SourceChange. There are exactly three:

pub enum SourceChange {
    Add(OwnedRow),
    Remove(OwnedRow),
    Edit { row: OwnedRow, old: OwnedRow },
}

Push one with Graph::source_push, then flush_view to close the transaction and fire the view’s listeners:

use rindle::{owned_row, OwnedValue, SourceChange};

graph.source_push(
    issue_src,
    SourceChange::Add(owned_row(vec![OwnedValue::Int(7), OwnedValue::Bool(true)])),
);
graph.flush_view(view);

let updated = graph.view_data(view); // reflects the new row

The cost of the push is proportional to the change, not to the table size. In a server, drive mutations through the fallible Graph::try_source_push (returns RindleError); the infallible source_push / hydrate shown above .expect the result and are for tests and prototyping. The rindle crate-level rustdoc documents the full survivability contract.

The engine is single-threaded by design: one Graph per thread (it is !Send). Scale with N independent graphs and message passing, never a shared Arc<Mutex<Graph>>.

Query planner

Some queries can be lowered more than one way. A correlated EXISTS can be driven from the parent (filter the parent, probe the child) or flipped to be driven from the child (stream matching children up to their parents) — and which is cheaper depends on the data. Rindle ships a cost-based planner (rindle-planner) that picks.

Planning is a pure Ast → Ast step that runs before lowering (step 2 above): it annotates each flippable EXISTS with a flip decision and changes nothing else. It is result-preserving — the flipped and unflipped plans return the same rows; only the work differs. The public seam is rindle_planner::plan_ast — lower the planned AST instead of the raw one:

use rindle_planner::plan_ast;

// `cost` is a `ConnectionCostModel`; `rindle-sqlite` provides a real-SQLite one.
let top = build_pipeline(&mut graph, &plan_ast(&ast, cost), &resolve)?;

The cost model is a real-SQLite model in rindle-sqlite — it reads SQLite’s own scan-status and table statistics rather than guessing. (The live-replica runtime runs planning by default.) See run the daemon for where planning fits in a deployment.

Driving it from a database

Wiring the source, resolve, and view by hand is the low-level path — and it is the whole open engine. The Rust quickstart does exactly this over a real SQLite table (via rindle-sqlite’s write-through TableSource), and fold the delta stream yourself swaps the View for a raw change sink.

Turning ordinary SQL writes into the change stream automatically (a preupdate hook behind one controlled writer), plus multi-threaded scale-out across readers, is the live-replica runtime — Rindle’s commercial high-concurrency layer on top of this same engine.

Next steps