JetStream KV persistence patterns for terminal state

Close your laptop, keep your workspace. How rysh persists terminal state in two JetStream KV buckets — and why pane writes are time-gated.

A terminal multiplexer makes a quiet promise: your workspace outlives your attention. Detach, close the laptop, come back tomorrow — tabs, panes, scrollback, the agent you left mid-task — still there. tmux keeps that promise with a long-lived server process. rysh keeps it with a long-lived process plus a database — because rysh panes carry more than scrollback: agent conversations, per-mode output streams, modes, histories.

The database is one we didn't have to add: NATS JetStream KV, running inside the embedded NATS server that every rysh session already boots for its message bus. Persistence lives where the messages already are. Here's the shape of it, including the failure windows we accept.

Two buckets, two write disciplines

All state lands in two KV buckets, and the interesting design is that they're governed by different rules.

rysh-workspace — a single key, "state", holding the structural skeleton: tab list, lane/group arrangement, pane references, active indices. Owned by the WorkspaceActor. Written on every state change.

rysh-panes — one key per pane (the pane's UUID), holding the heavy stuff: output buffer, status, mode (shell vs prompt), last command. Owned by each PaneActor. Written at most once per 2 seconds per pane.

The asymmetry is the whole lesson:

Workspace state is small, changes at human speed, and is catastrophic to lose. You create a tab a few times an hour; the record is tiny; losing it means reattaching to a wrong-shaped workspace. Write-through on every change is cheap and correct.

Pane state is large, changes at machine speed, and is cheap to lose a little of. A go test ./... run or a streaming agent response mutates the output buffer many times per second. Write-through here would be a self-inflicted write storm — hammering JetStream storage to persist scrollback nobody will miss if the last 1.9 seconds evaporate. So pane persistence is time-gated: dirty state flushes on a 2-second budget.

One system, two consistency contracts, each matched to what the data is worth. The 2-second number isn't sacred; the asymmetry is.

The lifecycle: restore-or-bootstrap

Startup is a single honest branch:

state, err := kv.Get("rysh-workspace", "state")
if err == nil && restore(state) == nil {
    // rebuild the actor tree: tabs → lanes → groups → panes,
    // then rehydrate each pane from rysh-panes by UUID
} else {
    bootstrap() // fresh default tabs and panes
}

The WorkspaceActor reads the skeleton and re-spawns the actor tree it describes; each restored PaneActor pulls its own buffer, mode, and status from rysh-panes under its UUID. If restoration fails — corrupt payload, missing bucket, version mismatch — rysh doesn't die and doesn't limp: it bootstraps a fresh workspace. A terminal that won't start because persistence is confused has its priorities backwards.

Shutdown flushes everything — the time gate exists to throttle steady-state writes, not to skip the final one. Detach (Ctrl+O d) is the trivial case: the session process keeps running, actors keep flushing on cadence, and reattach doesn't even need the restore path.

Deletion is the underrated benefit of per-session storage. Each session's NATS data lives under ~/.local/state/rysh/nats/{session}/, so rysh delete-session foo is: stop the process, remove the directory. No orphaned keys, no cross-session garbage collection, no migration debt from sessions deleted months ago.

Why KV-in-the-bus and not SQLite

SQLite would work. We rejected it for reasons that are architectural, not performance folklore:

1. Zero new infrastructure. The broker is already embedded, already per-session, already durable. A second storage engine means a second set of write paths, file formats, and failure modes in the same binary.
2. The write path matches the data flow. Actors already own their state and already live on the bus; persisting is "owner serializes its own slice to its own key." No cross-actor transactions, no shared schema, no ORM ceremony. The KV data model is the actor ownership model.
3. Deletion semantics. See above. rm -rf beats DELETE FROM ... WHERE session = ? for operational honesty.

What we gave up: queryability. You can't SELECT across panes; you Get by key. For workspace restoration — fetch skeleton, fan out by UUID — that's exactly the access pattern, so the loss is theoretical so far.

Failure windows, stated plainly

• A hard crash can lose up to ~2 seconds of pane output. Not a detach — a kill or a panic between flushes. For scrollback and status, we judge that acceptable and say so out loud. Structural state doesn't share the window: the workspace bucket is write-through.
• Last-write-wins, no transactions. Two buckets, independent writes, no atomicity across them. A crash at the wrong instant can leave a skeleton referencing a pane whose latest buffer flush didn't land; restore then rehydrates that pane slightly stale. Bounded, self-correcting on next flush, and vastly simpler than distributed transactions inside a terminal app.
• Schema evolution is discipline, not machinery. Persisted payloads are JSON; surviving field additions is handled by additive-change discipline rather than a migration framework, and the restore-or-bootstrap branch is the backstop if an old payload becomes unreadable. Thin, honestly. A version tag per payload with explicit upgrade functions is the obvious next step.

What we'd do differently

Two things. First, exploit what we're already paying for: JetStream KV keeps per-key revision history, and we currently use none of it — "restore workspace as of an hour ago" is sitting in the storage layer waiting for a UI. Second, make the flush cadence adaptive — a pane idling at a prompt doesn't need the same budget as one mid-build; dirty-size-triggered flushes would shrink the crash window exactly when it matters most.

The takeaway that generalizes: don't ask "which database?" first. Ask which data has which value, at which write rate — then give each class its own contract. Ours happened to fit in two buckets and one embedded server; yours might too.

rysh is in early access and we're recruiting design partners — teams who want agents in their real workflow and will tell us what breaks. Direct line to the founders: rysh.ai/design-partner.

More in this series: An embedded NATS server in every session · Snapshot-driven rendering