rindle/

lib.rs

//! # rindle — an incremental view maintenance (IVM) engine
//!
//! Build a query, hydrate its result, and push source changes to update that result
//! incrementally. This crate provides the raw graph, query builder, memory source,
//! and optional WebAssembly bindings. It is based on Zero's ZQL dataflow engine.
//!
//! Applications that need SQLite persistence and automatic SQL change capture should
//! start with `rindle-replica`. Custom SQLite sources live in the separate
//! `rindle-sqlite` crate. This core crate does not link SQLite or require a C toolchain
//! with its default features.
//!
//! ## Public entry points
//!
//! - **Build a query** — [`Query`] / [`table`] (fluent) or an [`Ast`] (Zero's wire
//!   JSON, via `serde`), then [`build_pipeline`] to lower it into a wired arena
//!   [`Graph`](crate::graph::Graph).
//! - **Run it** — hydrate the [`Graph`](crate::graph::Graph) over a [`MemorySource`],
//!   push [`SourceChange`](crate::change::SourceChange)s, and read the maintained
//!   [`View`] / [`ViewData`]. A `rindle-sqlite::TableSource` can supply a SQLite leaf.
//! - **Import core types** — use `rindle::graph::{Graph, NodeId}`,
//!   `rindle::change::SourceChange`, and `rindle::value::{OwnedRow, OwnedValue}`.
//!   The fluent [`table`] builder is re-exported at the crate root.
//! - **Errors** — fallible paths return [`RindleError`] (see *Fault model* below).
//!
//! ## Feature flags
//!
//! `memory` is enabled by default and required by this crate. There is no `sqlite`
//! feature; use the `rindle-sqlite` crate for that backend. `serde` enables AST
//! serialization, and `wasm` adds the JavaScript bindings plus `memory` and `serde`.
//! `observe` enables tracing, `metrics` enables process counters, and `testkit`
//! exposes the test oracle. `fast-alloc` opts into mimalloc and its native toolchain.
//!
//! ## Design
//!
//! The arena + `NodeId` operator graph, RAII (`Drop`) cursor cleanup, the COW
//! `Arc<Node>` B+tree, and the reentrant-fetch-during-push thesis that motivates them
//! are written up in `docs/DESIGN.md`.
//!
//! # Fault model (productionization WS02)
//!
//! The engine is built to be **survivable in a server**: no data-reachable input or
//! transient backend error should abort the process or silently corrupt view state.
//!
//! - **Use the `try_*` entry points in production** — [`Graph::try_source_push`](crate::graph::Graph::try_source_push),
//!   [`Graph::try_hydrate`](crate::graph::Graph::try_hydrate), [`Graph::try_add_source`](crate::graph::Graph::try_add_source) — which return [`RindleError`].
//!   Their infallible peers ([`Graph::source_push`](crate::graph::Graph::source_push) / [`Graph::hydrate`](crate::graph::Graph::hydrate) /
//!   [`Graph::add_source`](crate::graph::Graph::add_source)) `.expect` the result and are for tests/prototyping.
//! - **Strict change validation** ([`Graph::set_validate_changes`](crate::graph::Graph::set_validate_changes), WS02.2): off by
//!   default (hot-path parity). When on, a malformed change stream (ADD of an
//!   existing row, REMOVE/EDIT of an absent row) returns
//!   [`RindleError::ConsistencyViolation`] — in release, not a stripped `debug_assert!`.
//! - **Ingest validation** (WS02.4): [`Graph::try_add_source`](crate::graph::Graph::try_add_source) rejects a wrong-width
//!   row with [`RindleError::SchemaViolation`] before it reaches an unchecked index.
//! - **Predicates never crash** on a column/literal type mismatch (WS02.4).
//! - **SQLite operator-storage errors** (the spill backend) park a [`RindleError`] on
//!   the graph's runtime-error sink and surface via `take_runtime_error` instead of
//!   `.expect`-aborting; spilled blobs carry a format-version byte and a mismatch is
//!   a typed error (rebuild-from-source), not a panic (WS02.3 / WS09.3).
//! - **Panic taxonomy** (the `invariant` module, WS02.1): every remaining panic is
//!   a build-time / internal invariant; data-reachable failures are [`RindleError`].
//! - **Server fault isolation** (WS02.5): build the server with
//!   `--profile release-server` (`panic = "unwind"`) and drive mutations through
//!   [`Graph::source_push_isolated`](crate::graph::Graph::source_push_isolated) to contain a residual panic to one mutation
//!   (then discard + rebuild the view — never reuse a torn one). The wasm client
//!   keeps `panic = "abort"` plus a console panic hook.

//! # Client (wasm/JS)
//!
//! With the `wasm` feature, `wasm::RindleView` drives the engine from JavaScript over a
//! 5-call lifecycle (the `wasm` module has the full rustdoc):
//!
//! 1. `RindleView.build(astJson, schemas, data)` — lower an AST (Zero's wire JSON) + a
//!    `{ table: SchemaSpec }` map (+ optional `{ table: row[][] }` initial data),
//!    hydrate, return a handle. `BuildError`/`RindleError` throw as a JS `Error` with a
//!    `.kind` tag — never a wasm abort.
//! 2. `view.data()` — the materialized tree as a JS value: `{ <col>: v, …, <relName>:
//!    [child…] | child | null }`, in-view relationships only.
//! 3. `view.push(table, { type: 'add'|'remove'|'edit', row, old? })` — one source
//!    change (does **not** flush).
//! 4. `view.flush()` — notify subscribers if the view changed. This is a notification
//!    boundary, not a durable SQL transaction.
//! 5. `view.subscribe((data, resultType) => …)` — fired once immediately, then when
//!    a changed view flushes or its result type changes. Release the generated JS
//!    handle with `.free()` when finished.
//!
//! JavaScript numbers enter the number domain; the boundary rejects `bigint` and
//! outgoing exact integers outside the JS safe-integer range. Strings stay strings;
//! objects and arrays become JSON cells. Build the artifact with
//! `pnpm run build:wasm` from the repository root. **Concurrency:** one `Graph`/`RindleView` per thread
//! (it is `!Send`); scale with N independent graphs and message passing, never a
//! shared `Arc<Mutex<Graph>>`.

// Broken intra-doc links fail the build (WS08.6): the published rustdoc must not ship
// dangling references. CI also runs `RUSTDOCFLAGS="-D warnings" cargo doc` on both
// feature axes to catch the rest (e.g. redundant link targets).
#![deny(rustdoc::broken_intra_doc_links)]

#[cfg(not(feature = "memory"))]
compile_error!("enable the `memory` feature (the only built-in source backend; the SQLite backend lives in the `rindle-sqlite` crate)");

pub(crate) mod ast;
pub mod btree;
pub(crate) mod builder;
pub mod change;
/// Owned, fully-materialized change events off the dataflow pipeline
/// ([`CaughtChange`](changes::CaughtChange) / [`CaughtNode`](changes::CaughtNode) +
/// [`expand_change`](changes::expand_change)). Graduated out of the test oracle so the
/// production change-stream sink ([`Graph::add_change_sink`](graph::Graph::add_change_sink))
/// and out-of-crate consumers can use them. See `src/changes.rs`.
pub(crate) mod changes;
// Moved to `rindle-value`; re-exported so `crate::error::…` and the public
// `rindle::RindleError` path below are both unchanged.
pub(crate) use rindle_value::error;
/// Parameterized query families (design 310 §4): [`BindingSet`](family::BindingSet) —
/// the binding-set handle a family root connection's membership predicate reads — and
/// [`FamilyPipeline`](family::FamilyPipeline), what [`build_family_pipeline`] returns
/// and what `Graph::bind_family_partition` / `unbind_family_partition` /
/// `hydrate_family` drive. See `src/family.rs`.
pub mod family;
/// Flat change events for cross-process / cross-language view reconstruction
/// ([`FlatChange`](flat::FlatChange) + [`flatten`](flat::flatten)): the nested
/// [`CaughtChange`](changes::CaughtChange) tree linearized into root→leaf paths so a
/// remote receiver can rebuild the exact `ArrayView`. See `src/flat.rs` and
/// `FLAT-CHANGES-DESIGN.md` (the wire format + receiver `apply_change` contract).
pub(crate) mod flat;
/// The flat-change **subscription protocol** ([`Hello`](flat_protocol::Hello) /
/// [`Batch`](flat_protocol::Batch) / [`Publisher`](flat_protocol::Publisher) /
/// [`Subscriber`](flat_protocol::Subscriber)): the batch envelope, epoch/seq framing,
/// and the receiver-side safety rules (in-order apply, gap → re-hydrate, at-most-once,
/// epoch/schema/comparator validation). See `src/flat_protocol.rs` and
/// `FLAT-CHANGES-DESIGN.md` §5.4/§5.5/§2.3.
pub(crate) mod flat_protocol;
/// A reference [`Receiver`](flat_receiver::Receiver): a host-agnostic port of the
/// View's `apply_change`, fed by [`FlatChange`](flat::FlatChange)s, that reconstructs
/// the exact `ArrayView` tree (rc multi-path + edit-move, no COW). The executable form
/// of `FLAT-CHANGES-DESIGN.md` §6 and the conformance oracle. See `src/flat_receiver.rs`.
pub(crate) mod flat_receiver;
pub mod graph;
/// Panic taxonomy + the `invariant!` macro (WS02.1): the one rule separating
/// build-time/internal invariants (keep as panic) from data-reachable failures
/// (convert to [`error::RindleError`]). See `src/invariant.rs`.
pub(crate) mod invariant;
/// The cross-process **journal frame envelope**
/// ([`FrameHeader`](journal_frame::FrameHeader) + [`FrameKind`](journal_frame::FrameKind)):
/// the typed header (kind / committed_at / run identity + totals) in front of an opaque
/// change payload wherever journal entries live — the hctree master's `hct_journal`
/// payload column and the S3 archival segments (`rindle-backup`). Pure bytes, no deps.
/// See `src/journal_frame.rs`, `designs-implemented/211-HCTREE-LEADER-CDC-DESIGN.md` §4.1, and
/// `designs-implemented/212-JOURNAL-S3-SHIPPING-DESIGN.md` §2.2.
pub mod journal_frame;
/// JS-boundary safe-integer walkers (productionization 09.8, design 226 Stage A): the
/// opt-in `strict_i64` check that refuses an out-of-`Number.MAX_SAFE_INTEGER` `Int`
/// crossing to JS with a typed error instead of silently rounding it.
pub mod js_safe;
pub(crate) mod memory_source;
/// Build-gated process metrics (WS03, the `metrics` feature): `metric_inc!`/… fold
/// each seam into a relaxed atomic add on a process-global registry the daemon's
/// Prometheus endpoint reads, and to argument-consuming no-ops (no global linked)
/// otherwise. The scrape-path sibling of `observe`; see `src/metrics.rs`.
pub mod metrics;
/// Feature-gated observability shim (WS03.1): `obs_span!`/`obs_event!`/… expand to
/// `tracing` when `--features observe`, and to argument-consuming no-ops (no
/// `tracing` dependency) otherwise — keeping the default/wasm build lean.
pub(crate) mod observe;
pub mod op;
/// The optimistic-writes fork/rebase loop ([`OptimisticTables`](optimistic::OptimisticTables)
/// — per-table `sync` forks + the §1.3 rewind over the one live pipeline). The cycle's
/// buffered event stream is forwarded raw to the consumer (the §3 `Coalescer` was removed —
/// see the module docs). See `src/optimistic.rs` and `OPTIMISTIC-WRITES-DESIGN.md`.
// Only the wasm `Db` drives this. While the module was `pub` that was invisible;
// closing it makes the non-wasm build see an unused module, so scope the allow to
// exactly the builds where it is genuinely unreachable.
#[cfg_attr(not(feature = "wasm"), allow(dead_code))]
pub(crate) mod optimistic;
pub(crate) mod predicate;
/// The guarded push fan-out reverse index (`designs/205-GUARDED-PUSH-FANOUT-DESIGN.md`):
/// prunes a source write's per-connection fan-out to the connections whose
/// equality-shaped `where` guard could match. A conservative superset index —
/// never under-approximates — so `filter_push` stays the exact gate.
pub mod push_index;
pub(crate) mod query;
pub(crate) mod scalar;
pub mod source_common;
pub mod storage;
/// The spec-`11` operator test harness (`Catch`/`Snitch`/runners/fingerprint).
/// Exported only for crate tests or the `testkit` cargo feature so production
/// library builds do not expose harness helpers. Use via `rindle::testkit::*`.
#[cfg(any(test, feature = "testkit"))]
pub mod testkit;
/// The canonical value-equivalence key ([`CanonVal`](canon::CanonVal) /
/// [`CanonKey`](canon::CanonKey)) — the join family's grouping key, the limiters'
/// partition identity, and a parameterized query family's binding (design 310, D1).
/// Lives in `rindle-value` (below the engine, so `rindle-wire` can name it) and is
/// re-exported here as the `rindle::canon` namespace.
pub use rindle_value::canon;
/// The value vocabulary — moved to the dependency-free `rindle-value` crate and
/// re-exported here, so `rindle::value::…` is unchanged for every consumer. Crates that
/// need only cells (the write plane, the CDC apply plane) depend on `rindle-value`
/// directly and therefore cannot reach the engine at all.
pub use rindle_value::value;
/// The production materialization sink (`09`): the `Arc`-shared, reference-stable
/// `ArrayView` + immutable tree differ `apply_change`. See `src/view.rs`.
pub(crate) mod view;
/// The wasm/JS client boundary (WS01): `RindleView` + value/error/schema/view-tree
/// marshalling. Only compiled for the `wasm` feature. See `src/wasm/`.
#[cfg(feature = "wasm")]
pub mod wasm;
/// The view schema on the wire + content fingerprint
/// ([`WireSchema`](wire_schema::WireSchema) / [`schema_fp`](wire_schema::schema_fp) +
/// [`COMPARATOR_VERSION`](wire_schema::COMPARATOR_VERSION)): the hierarchical view
/// [`Schema`](value::Schema) reduced to what a receiver needs (names + resolved sort +
/// nested relationships), shipped once. See `src/wire_schema.rs` and
/// `FLAT-CHANGES-DESIGN.md` §5.2/§5.5.
pub(crate) mod wire_schema;

#[cfg(kani)]
mod kani_proofs;

// Crate-root re-export of the observability shim macros (WS03.1 step) so the
// instrumented seams (WS03.2–03.5) can call `obs_span!`/… without a module path.
// Kept `pub(crate)` — these are internal instrumentation, not public API (WS01.7
// audits the public surface). Unreferenced until the seams land.
#[allow(unused_imports)]
pub(crate) use observe::{obs_counter_inc, obs_event, obs_gauge_set, obs_span};

// Crate-root re-export of the metrics instrumentation macros so the seams can call
// `metric_inc!`/… unqualified. `pub(crate)` — internal instrumentation, not public API.
#[allow(unused_imports)]
pub(crate) use metrics::{
    metric_add, metric_build_err, metric_change_kind, metric_changes_inc, metric_inc, metric_timer,
};

// ===========================================================================
// Public API — ONE CANONICAL PATH PER ITEM.
//
// The query you build (`Ast`/`Query`/`table`), the engine you drive
// (`graph::Graph` + `try_*`), the values you push (`value::OwnedValue` /
// `change::SourceChange` / `value::Schema`), the view you read
// (`View`/`ViewData`/`Entry`), and the error you handle (`RindleError`).
//
// The rule: every public item is reachable at EXACTLY ONE path. A module is
// therefore either
//   * `pub(crate) mod` — implementation; selected items re-exported here, so the
//     canonical path is `rindle::Item`; or
//   * `pub mod` — a namespace (`value`, `change`, `graph`, `source_common`,
//     `storage`, `btree`, `journal_frame`, `op`, `metrics`, `push_index`,
//     `js_safe`, `testkit`, `wasm`) — and then NOTHING from it is re-exported
//     here, so the canonical path is `rindle::module::Item`.
// Re-exporting from a `pub mod` puts an item at two addresses and is the one
// thing not to do below.
//
// NOTE: `#[doc(hidden)]` does NOT reduce the surface — it only hides an item from
// the rendered docs. `cargo public-api` still reports every hidden item, and a
// `pub` item in a `pub` module stays reachable regardless of the attribute. Only
// module/item visibility shrinks the surface. `scripts/public-api.sh --check`
// diffs the result against the committed snapshot in `rust/public-api/`.
// ===========================================================================

pub use ast::{
    canon_of_lit, Aggregate, Ast, Bound, Condition, CorrelatedSubquery,
    CorrelatedSubqueryCondition, Correlation, Dir, ExistsOp, Lit, Op, OrderPart, SimpleCondition,
    System, ValuePosition,
};
pub use builder::{build_family_pipeline, build_pipeline, view_schema, BuildError};
pub use changes::{CaughtChange, CaughtNode};
pub use error::RindleError;
pub use flat::{flatten, flatten_all, FlatChange, FlatOp, PathSeg, WireNode, WireRow};
pub use flat_protocol::{
    Applied, Batch, Hello, ProtocolError, Publisher, SnapStatus, SnapshotChunk, Subscriber,
};
pub use flat_receiver::{Receiver, RecvNode};
pub use memory_source::MemorySource;
pub use predicate::LikeMatcher;
pub use query::{table, Cond, ExistsOpts, Parent, ParentRow, Query};
pub use scalar::{has_scalar_subquery, resolve_scalars, ScalarCatalog, ScalarSource};
pub use view::{Entry, EntryList, Listener, ResultType, View, ViewData};
pub use wire_schema::{schema_fp, to_schema, to_wire, WireRel, WireSchema, COMPARATOR_VERSION};

// --- internal plumbing: reachable because a public signature or an out-of-crate
//     test needs it. `#[doc(hidden)]` keeps it out of the RENDERED DOCS only — it is
//     still public surface and still appears in the `cargo public-api` snapshot. ---
#[doc(hidden)]
pub use builder::{
    complete_ordering, create_predicate, normalize_pipeline_ast, query_local_slot_names,
    schema_primary_key_names, transform_filters,
};
#[doc(hidden)]
pub use memory_source::{build_scan_start, constrained_index_sort};
#[doc(hidden)]
pub use predicate::{CmpOp, CompiledPredicate, ValueSet};
// Reachable ONLY from this crate's own `tests/` (which compile as external consumers,
// so `pub(crate)` is not enough). They are not API; the right fix is to move those two
// tests inward as unit tests, which would let both of these become `pub(crate)`.
#[doc(hidden)]
pub use ast::canonicalize_wire_number_lits;
#[doc(hidden)]
pub use view::Col0Node;
#[doc(hidden)]
pub use view::{apply_change, EntryId, EntryListInner, Mutate, TxnDirty, TxnGen, REL_ROOT};