Rindle

The engine

How it works

The five-step lifecycle every crate shares — build a query, lower it into a graph, add a view, hydrate, push and flush — and why incremental beats recompute.

View as Markdown

Rindle keeps a query result current by incremental view maintenance (IVM): you build a query, hydrate a materialized view from it, and push the changes that happen to the underlying tables. The view updates by the difference — never by re-running the query. The contract is view-after-push == fresh-query: applying the change stream leaves the view exactly equal to running the query from scratch.

Internally this is the open-source rindle engine (Apache-2.0), with the rindle-sqlite backend when your sources are SQLite tables. This page walks the data path end to end. For the shapes a query can take, see Supported queries. For the change stream itself, see Change model.

The data path

A query travels through four stages:

  1. Build an Ast — either with Deltic, the fluent rindle::table query builder, or by deserializing the engine’s JSON wire format.
  2. Lower that Ast into a wired dataflow Graph of operators with rindle::build_pipeline.
  3. Hydrate a View over the graph to materialize the initial result set.
  4. Push source changes and read the incrementally-maintained ViewData.

1. Build a query

The fluent builder produces an Ast. Each method returns the builder, and build finishes it:

use rindle::table;

let ast = table("issue")
    .r#where("open", true)
    .build();

r#where takes a field name and a value. build consumes the builder and returns the Ast. (r#where is spelled with the raw-identifier escape because where is a Rust keyword.)

You can also deserialize an Ast directly from its JSON wire format via serde, which is how a JS client hands a query to the engine.

2. Lower it into a graph

rindle::build_pipeline walks the Ast and allocates the operator graph. Its signature is:

pub fn build_pipeline(
    graph: &mut rindle::Graph,
    ast: &rindle::Ast,
    resolve: &impl Fn(&str) -> Option<(rindle::NodeId, rindle::SourceSchema)>,
) -> Result<rindle::NodeId, rindle::BuildError>;

You first register each base table as a source on the Graph, then pass a resolve closure that maps a table name to its source NodeId and SourceSchema. SourceSchema is the table-registration type (columns, primary key, default sort — deliberately no relationships, which are a property of a query). build_pipeline returns the NodeId of the pipeline’s top operator (or a BuildError if the Ast is outside the supported subset):

use std::collections::HashMap;
use rindle::{build_pipeline, table, Graph, NodeId, SourceSchema};

let mut graph = Graph::new();

// Register the base table as an in-memory source. `SourceSchema::new` takes the
// columns, the primary-key column indices, and the table's default sort.
let schema = SourceSchema::new(vec!["id", "open"], vec![0], vec![(0, true)]);
let issue_src: NodeId = graph.add_source(schema.clone(), Vec::new());

// `resolve` maps each table name to (source NodeId, SourceSchema).
let mut sources: HashMap<&str, (NodeId, SourceSchema)> = HashMap::new();
sources.insert("issue", (issue_src, schema));
let resolve = |name: &str| sources.get(name).cloned();

let ast = table("issue").r#where("open", true).build();
let top: NodeId = build_pipeline(&mut graph, &ast, &resolve)
    .expect("build the pipeline");

In production prefer the fallible Graph::try_add_source, which validates ingest row widths and returns a RindleError instead of panicking on a malformed row.

3. Hydrate a view

A View is the materialized sink. It takes the richer Schema type (sort, relationship slots, singularity), which rindle::view_schema derives from the Ast and the same resolve — one derivation, shared with the wasm client. Add the view over the pipeline’s top operator with Graph::add_view, wire the final edge with Graph::set_sink_edge, then hydrate:

use rindle::view_schema;

let vschema = view_schema(&ast, &resolve).expect("derive the view schema");
let view = graph.add_view(top, vschema);
graph.set_sink_edge(top, view);
graph.hydrate(view);

// Read the materialized result: a reference-stable list of `Entry`s — each an
// owned `row` plus, for nested queries, child `rels`. Iterate `data.items`.
let data = graph.view_data(view);

Graph::hydrate drains the input pipeline once to build the initial result. As with push, there is a fallible peer, Graph::try_hydrate, that returns Result<(), RindleError> — use it in a server.

4. Push changes

A mutation to a base table is a rindle::SourceChange. There are exactly three:

pub enum SourceChange {
    Add(OwnedRow),
    Remove(OwnedRow),
    Edit { row: OwnedRow, old: OwnedRow },
}

Push one with Graph::source_push, then flush_view to close the transaction and fire the view’s listeners:

use rindle::{owned_row, OwnedValue, SourceChange};

graph.source_push(
    issue_src,
    SourceChange::Add(owned_row(vec![OwnedValue::Int(7), OwnedValue::Bool(true)])),
);
graph.flush_view(view);

let updated = graph.view_data(view); // reflects the new row

The cost of the push is proportional to the change, not to the table size. In a server, drive mutations through the fallible Graph::try_source_push (returns RindleError). The infallible source_push / hydrate shown above .expect the result and are for tests and prototyping. The rindle crate-level rustdoc documents the full survivability contract.

The engine is single-threaded by design: one Graph per thread (it is !Send). Scale with N independent graphs and message passing, never a shared Arc<Mutex<Graph>>.

Query planner

Some queries can be lowered more than one way. A correlated EXISTS can be driven from the parent (filter the parent, probe the child) or flipped to be driven from the child (stream matching children up to their parents). Which is cheaper depends on the data. Rindle ships a cost-based planner (rindle-planner) that picks.

Planning is a pure Ast → Ast step that runs before lowering (step 2 above): it annotates each flippable EXISTS with a flip decision and changes nothing else. It is result-preserving — the flipped and unflipped plans return the same rows. Only the work differs. The public seam is rindle_planner::plan_ast — lower the planned AST instead of the raw one:

use rindle_planner::plan_ast;

// `cost` is a `ConnectionCostModel`; `rindle-sqlite` provides a real-SQLite one.
let top = build_pipeline(&mut graph, &plan_ast(&ast, cost), &resolve)?;

The cost model is a real-SQLite model in rindle-sqlite — it reads SQLite’s own scan-status and table statistics rather than guessing. (The live-replica runtime runs planning by default.) See run the daemon for where planning fits in a deployment.

Driving it from a database

Wiring the source, resolve, and view by hand is the low-level path — and it is the whole open engine. The Rust quickstart does exactly this over a real SQLite table (via rindle-sqlite’s write-through TableSource), and fold the delta stream yourself swaps the View for a raw change sink.

The live-replica runtime turns ordinary SQL writes into the change stream automatically (a preupdate hook behind one controlled writer) and adds multi-threaded scale-out across readers. It is Rindle’s commercial high-concurrency layer on top of this same engine.

Next steps