Module storage 

Spec 10 — operator scratch state: the small, sorted, string-keyed side-table a stateful operator (Take, Cap) keeps bookkeeping in across fetch/push calls. Ports zql/src/ivm/operator.ts:132 Storage and zql/src/ivm/memory-storage.ts.

It is not a row store, not the source index (crate::btree), and not the view tree. It is the IVM analogue of a per-operator hash-map, hoisted behind a trait so the server can later spill it to SQLite while the client keeps it in RAM (spec 10 §1.1).

What’s here vs. the SQLite backend

This module is the backend-agnostic core: the Storage trait, the tight StorageValue enum, MemoryStorage for the client/test path, the StorageProvider seam, and the StorageFactory the builder calls. The server spill-to-SQLite backend (DatabaseStorage/OpStorage over rusqlite, spec 10 §4.4) lives in the rindle-sqlite crate, plugged in via StorageFactory::custom(Rc::new(database_storage)) — so the core crate carries no rusqlite dependency.

Backing structure — BTreeMap for now (a measured decision)

MemoryStorage is backed by a std::collections::BTreeMap, kept private behind the Storage trait. This is the spec 10 §4.3 choice. The crate’s COW B+tree (crate::btree) was considered to avoid BTreeMap’s wasm codegen, but it is an OwnedRow set keyed by a Sort, not a key→value map — adapting it costs an encode/decode adapter for a bundle win that is currently unmeasured. Since the backing is invisible behind the trait, we ship the simplest correct thing now and revisit only if twiggy (M9) shows the BTreeMap delta actually matters. Two properties make BTreeMap a good fit: it stores StorageValue directly (so get of a Take{bound} is an Arc refcount bump, not a row copy — spec 10 §3.2/§8.1), and it gives the sorted-key invariant + a native range for the prefix Storage::scan.

Structs

MemoryStorage

In-RAM Storage. A BTreeMap<Box<str>, StorageValue> (kept private — see the module docs) gives the sorted-key invariant, O(log n) point ops, and a native range for the prefix scan. RefCell provides the interior mutability the &self trait methods need (single-threaded per pipeline — foundations §1.3, so no Sync required).

ReduceAcc

One per-column running accumulator for a Sum/Avg aggregate inside a StorageValue::Reduce (REDUCE-DESIGN.md §5). The integer and float sums are kept apart so the emitted sum matches SQLite’s typing — integer iff every summed value was an integer, real once any float contributes — and stays fully invertible (a later Remove of a float value demotes the result back to an integer).

StorageFactory

Enums

StorageValue

The value stored in operator scratch state. A tight enum, not generic JSON: the only writers are Take and Cap (spec 10 §1.3), and their payloads are known. This (a) keeps serde_json out of the wasm client, (b) makes every value a flat, cheaply-clonable struct instead of a heap JSON tree, and (c) gives operators type-safe state instead of the JS storage as TakeStorage casts (take.ts:78).

Traits

Storage

Per-operator scratch state: a sorted string-keyed map with a prefix range scan. Ports zql/src/ivm/operator.ts:132 Storage.

StorageProvider

A backend’s per-operator store factory, erased behind a trait object so the core graph’s StorageFactory can vend SQLite-backed operator storage (the rindle-sqlite DatabaseStorage) without the core crate naming a rusqlite type. The in-memory path bypasses this (it builds MemoryStorage directly).

Functions

create_storage

What the builder (08) calls to give a stateful operator its own store. On the client/test path a fresh object is the namespace (spec 10 §3.3, §4.5): each Take/Cap gets a disjoint keyspace because it gets a distinct MemoryStorage. The JS name argument is not needed by the memory backend (uniqueness is object identity — spec 10 §6 deviation D2), so it is not taken here; the builder keeps name at its own boundary for debug/logging.