One Hostname, Many Followers: How Rindle Routes Reads

Rindle keeps registered queries up to date incrementally instead of re-running them. In Rindle Cloud, that work happens on read followers, and every browser holds a live connection to one of them. This post explains how a browser finds its follower, how it stays there, and what happens when that follower dies or falls behind.

# Why reads live on followers

Rindle Cloud deploys one shape. A write-master (rindle-replicator) commits every transaction into one ordered change log. It fans that log out to read followers (rindled). Each follower applies the stream into its own database and maintains the live queries. Browsers subscribe to followers.

A follower has no write ingress at all. Its only intake is the master’s ordered stream. If a write reaches a follower, the follower answers with a 409 that names the master.

Why so strict? Incremental view maintenance is defined over one totally ordered stream of changes. When the node that runs views only applies that stream, the hardest concurrency problem disappears. There is no local writer interleaved with view maintenance. The follower hot path is one path, and we harden that one path.

# The two legs of a read

A live query reaches the server on two connections.

The first leg is HTTP. The client asks the api-server to resolve a named query. The api-server authorizes the request, builds the approved query AST, and asks a follower to materialize it. The follower builds the pipeline in its own memory and mints a lease. The api-server returns the lease token to the client.

The second leg is a WebSocket. The client subscribes with the lease token, and the follower streams row deltas from then on.

Here is the constraint. The lease and the materialization live in one follower’s RAM. A lease token minted by follower A is not valid on follower B. Both legs must land on the same follower. That is the entire routing problem.

# One hostname, one edge in front

The client configuration holds one URL. One process — the edge — owns that hostname. It terminates the browser’s WebSocket and the read-plane HTTP, and proxies each to a follower. The api-server sends its read calls to the same hostname. A larger fleet changes nothing in the application: no new endpoints, no client changes.

# Connect first, then carry a ticket

The client connects the WebSocket before it sends its first lease request. The edge picks a live follower and proxies the socket to it. Then the edge mints a signed affinity ticket that names that follower. The first frame down the socket carries the ticket. The client stores the ticket in sessionStorage, so stickiness is per tab.

From then on, the ticket travels with both legs. When the socket reconnects, the client offers the ticket in the WebSocket subprotocol list. When the client sends a lease request, it includes the ticket. The api-server forwards it untouched, and the edge reads it from the Rindle-Affinity header and routes the call to the follower that the ticket names.

One sequencing rule ties the legs together. On a fresh connection, the client waits for the ticket frame before it sends the first lease request. The wait is bounded at four seconds. After that, the request proceeds without a ticket, and the edge places it on its own. Without this rule, the two legs can race to two different followers.

browser ──── ws connect ────────▶ edge ── places ──▶ follower F
browser ◀── {t:"affinity", ticket:T} ── edge   (first frame, T names F)
browser ── lease request {name, args, T} ──▶ api-server
api-server ── /materialize, Rindle-Affinity: T ──▶ edge ──▶ F
browser ── {t:"subscribe", leaseToken} ── ws ──▶ F ◀── row deltas

# What is in a ticket

The payload names the app, the follower, the region, the issue time, the expiry time, and a fleet generation number. The edge signs the encoded payload with HMAC-SHA256 under a fleet key that only the edge holds. A ticket lives for one hour. Verification rejects a ticket that is malformed, forged, expired, aimed at the wrong app, or from a stale generation. Key rotation works through a two-key verification window.

The generation number is the fleet-wide reset lever. When an operator bumps it, every outstanding ticket goes stale at once and the whole fleet re-places. The drain workflow that uses this lever is designed but not built yet.

A ticket is a routing hint, not a capability. The signature exists so that the edge accepts only tickets that the edge minted.

# When a follower dies

The edge marks an unreachable follower as failed for three seconds. A ticket that names a failed follower does not bounce the client off the fleet. The edge re-places the client on a live sibling and mints a fresh ticket.

The client protects itself too. After four failed reconnect attempts in a row, it clears the stored ticket. The next connection arrives ticketless, so the edge places it fresh. The result is one bounded reassignment, not a loop.

The lease is the correctness backstop. A lease token can reach a follower that does not hold it — the wrong node, or a node that restarted. That follower answers a retryable lease_expired error with a retry delay, and the client requests a new lease through the normal path. Correctness never depends on the ticket. The ticket only decides where the work lands.

# Placement is not authorization

The edge holds the fleet key and nothing else. It has no database credentials, and it forwards the browser’s own authorization header untouched. The api-server is the authority: it resolves (name, args) into an approved AST, and a follower materializes only approved ASTs. A follower mints a lease only after the api-server authorizes the query. A stolen ticket changes which follower serves you. It grants no data.

# When a follower falls behind

Stickiness has a failure mode: the follower you are pinned to can lag the master. Each follower watches its own backlog and its own drain rate, and walks a ladder of modes: Normal, Batching, Shed.

In Shed, the follower protects its catch-up speed. It drops every live pipeline. It answers each subscribe with a retryable “shed” error that says: retry in 5 seconds. It refuses cold materializations with a 503 and a Retry-After header. With no pipelines to feed, it applies the backlog at raw speed. On exit, a token bucket re-admits materializations at 4 per second, so the returning storm does not push the follower straight back over the threshold.

One thing the edge does not do today: it does not read lag as a placement signal. The ticket keeps you on your follower, and the follower itself tells you to back off. Lag-aware placement is designed, not shipped.

# What is built, and what is not

Everything above ships today, and not only in the cloud. rindle up runs the same edge on your machine, even for a fleet of one. Local development exercises the same ticket path as production.

The word “global” needs an honest note. The ticket carries a region, and the placement code prefers a follower in the client’s region. So the routing layer is region-aware end to end. But today the region signal is a configured hint, not a geographic lookup, and hosted apps run in one region. The hosted tier currently places one follower per app, while the self-host roles support many. The global rollout — the nearest edge by anycast, region-local follower sets, lag-aware placement — is designed and unshipped. We prefer to tell you which side of that line each claim sits on.

# Bottom line

Reads scale by adding followers, and routing is how a browser keeps one of them. The client connects, receives a signed ticket, and offers that ticket on both legs. The edge keeps the socket and the leases together on one follower. On failure, the ticket clears and the edge places the client once more. Authorization never moves: the api-server approves the query, the follower serves it, and the ticket only picks the node.