# How it works

The five-step lifecycle every crate shares — build a query, lower it into a graph, add a view, hydrate, push and flush — and why incremental beats recompute.

Rindle keeps a query result current by **incremental view maintenance (IVM)**: you
build a query, hydrate a materialized view from it, push the changes that happen to
the underlying tables, and the view updates by the *difference* — never by re-running
the query. The contract is *view-after-push == fresh-query*: applying the change
stream leaves the view exactly equal to running the query from scratch.

Under the hood this is the open-source `rindle` engine (Apache-2.0), with the
`rindle-sqlite` backend when your sources are SQLite tables. This page walks the
data path end to end. For the shapes a query may take, see
[Supported queries](/docs/supported-queries); for the change stream itself, see
[Change model](/docs/change-model).

## The data path

A query travels through four stages:

1. **Build** an `Ast` — either with **Deltic**, the fluent `rindle::table` query
   builder, or by deserializing the engine's JSON wire format.
2. **Lower** that `Ast` into a wired dataflow `Graph` of operators with
   `rindle::build_pipeline`.
3. **Hydrate** a `View` over the graph to materialize the initial result set.
4. **Push** source changes and read the incrementally-maintained `ViewData`.

## 1. Build a query

The fluent builder produces an `Ast`. Each method returns the builder, and
`build` finishes it:

```rust
use rindle::table;

let ast = table("issue")
    .r#where("open", true)
    .build();
```

`r#where` takes a field name and a value. `build` consumes the builder and returns
the `Ast`. (`r#where` is spelled with the raw-identifier escape because `where` is a
Rust keyword.)

You can also deserialize an `Ast` directly from its JSON wire format via `serde`, which is
how a JS client hands a query to the engine.

## 2. Lower it into a graph

`rindle::build_pipeline` walks the `Ast` and allocates the operator graph. Its
signature is:

```rust
pub fn build_pipeline(
    graph: &mut rindle::Graph,
    ast: &rindle::Ast,
    resolve: &impl Fn(&str) -> Option<(rindle::NodeId, rindle::SourceSchema)>,
) -> Result<rindle::NodeId, rindle::BuildError>;
```

You first register each base table as a source on the `Graph`, then pass a `resolve`
closure that maps a table name to its source `NodeId` and `SourceSchema` — the
table-registration type (columns, primary key, default sort; deliberately no
relationships, which are a property of a *query*). `build_pipeline` returns the
`NodeId` of the pipeline's top operator (or a `BuildError` if the `Ast` is outside
the supported subset):

```rust
use std::collections::HashMap;
use rindle::{build_pipeline, table, Graph, NodeId, SourceSchema};

let mut graph = Graph::new();

// Register the base table as an in-memory source. `SourceSchema::new` takes the
// columns, the primary-key column indices, and the table's default sort.
let schema = SourceSchema::new(vec!["id", "open"], vec![0], vec![(0, true)]);
let issue_src: NodeId = graph.add_source(schema.clone(), Vec::new());

// `resolve` maps each table name to (source NodeId, SourceSchema).
let mut sources: HashMap<&str, (NodeId, SourceSchema)> = HashMap::new();
sources.insert("issue", (issue_src, schema));
let resolve = |name: &str| sources.get(name).cloned();

let ast = table("issue").r#where("open", true).build();
let top: NodeId = build_pipeline(&mut graph, &ast, &resolve)
    .expect("build the pipeline");
```

In production prefer the fallible `Graph::try_add_source`, which validates ingest row
widths and returns a `RindleError` instead of panicking on a malformed row.

## 3. Hydrate a view

A `View` is the materialized sink; it takes the richer `Schema` type (sort,
relationship slots, singularity), which `rindle::view_schema` derives from the
`Ast` and the same `resolve` — one derivation, shared with the wasm client. Add
the view over the pipeline's top operator with `Graph::add_view`, wire the final
edge with `Graph::set_sink_edge`, then `hydrate`:

```rust
use rindle::view_schema;

let vschema = view_schema(&ast, &resolve).expect("derive the view schema");
let view = graph.add_view(top, vschema);
graph.set_sink_edge(top, view);
graph.hydrate(view);

// Read the materialized result: a reference-stable list of `Entry`s — each an
// owned `row` plus, for nested queries, child `rels`. Iterate `data.items`.
let data = graph.view_data(view);
```

`Graph::hydrate` drains the input pipeline once to build the initial result. As with
push, there is a fallible peer, `Graph::try_hydrate`, that returns `Result<(),
RindleError>` — use it in a server.

## 4. Push changes

A mutation to a base table is a `rindle::SourceChange`. There are exactly three:

```rust
pub enum SourceChange {
    Add(OwnedRow),
    Remove(OwnedRow),
    Edit { row: OwnedRow, old: OwnedRow },
}
```

Push one with `Graph::source_push`, then `flush_view` to close the transaction and
fire the view's listeners:

```rust
use rindle::{owned_row, OwnedValue, SourceChange};

graph.source_push(
    issue_src,
    SourceChange::Add(owned_row(vec![OwnedValue::Int(7), OwnedValue::Bool(true)])),
);
graph.flush_view(view);

let updated = graph.view_data(view); // reflects the new row
```

The cost of the push is proportional to the change, not to the table size. In a
server, drive mutations through the fallible `Graph::try_source_push` (returns
`RindleError`); the infallible `source_push` / `hydrate` shown above `.expect` the result
and are for tests and prototyping. The `rindle` crate-level rustdoc documents the
full survivability contract.

> The engine is single-threaded by design: one `Graph` per thread (it is `!Send`).
> Scale with N independent graphs and message passing, never a shared
> `Arc<Mutex<Graph>>`.

## Query planner

Some queries can be lowered more than one way. A correlated `EXISTS` can be driven
from the parent (filter the parent, probe the child) or **flipped** to be driven
from the child (stream matching children up to their parents) — and which is cheaper
depends on the data. Rindle ships a cost-based planner (`rindle-planner`) that picks.

Planning is a pure **`Ast` → `Ast`** step that runs **before lowering** (step 2
above): it annotates each flippable `EXISTS` with a `flip` decision and changes
nothing else. It is **result-preserving** — the flipped and unflipped plans return
the same rows; only the work differs. The public seam is `rindle_planner::plan_ast`
— lower the *planned* AST instead of the raw one:

```rust
use rindle_planner::plan_ast;

// `cost` is a `ConnectionCostModel`; `rindle-sqlite` provides a real-SQLite one.
let top = build_pipeline(&mut graph, &plan_ast(&ast, cost), &resolve)?;
```

The cost model is a real-SQLite model in `rindle-sqlite` — it reads SQLite's own
scan-status and table statistics rather than guessing. (The
[live-replica runtime](/docs/replica-and-views) runs planning by default.) See
[run the daemon](/docs/daemon#the-query-planner) for where planning fits in a
deployment.

## Driving it from a database

Wiring the source, resolve, and view by hand is the low-level path — and it *is* the
whole open engine. The [Rust quickstart](/docs/quickstart) does exactly this over a
real SQLite table (via `rindle-sqlite`'s write-through `TableSource`), and
[fold the delta stream yourself](/docs/example-rust) swaps the `View` for a raw
change sink.

Turning *ordinary SQL writes* into the change stream automatically (a `preupdate`
hook behind one controlled writer), plus multi-threaded scale-out across readers, is
the [live-replica runtime](/docs/replica-and-views) — Rindle's commercial
high-concurrency layer on top of this same engine.

## Next steps

- [Overview](/docs/overview) — what Rindle is and when to reach for it.
- [Quickstart](/docs/quickstart) — stand up a live query end to end.
- [Change model](/docs/change-model) — the `Add` / `Remove` / `Edit` / `Child` deltas
  in depth.
- [Fold the delta stream yourself](/docs/example-rust) — the raw change sink.
- [Supported queries](/docs/supported-queries) — the matrix of what builds and pushes.
- [The live-replica runtime](/docs/replica-and-views) — SQL writes in, deltas out, readers scaled.

---

[View this page on Rindle](https://rindle.sh/docs/how-it-works)