Emulating a terminal inside a terminal: vt10x and raw mode

vim, htop, and less inside a Bubble Tea app: byte loops, alternate-screen detection, and turning KeyMsg back into terminal bytes.

The acid test for any terminal multiplexer is three letters: vim.

Plenty of "AI terminal" projects render a scrollback of lines and call it a terminal. Then someone opens vim — or htop, or less, or a curses installer — and the illusion collapses into escape-code confetti. A pane is only a real terminal if a full-screen program can own it: alternate screen, cursor addressing, arrow keys that arrive as the byte sequences the program expects.

rysh is an agentic terminal multiplexer — every pane is a conversation that can be a shell or an agent (running on Claude, your key). But "agentic" earns nothing if the terminal part is a toy. So each pane is a PTY-backed shell, and behind each pane sits a terminal emulator of its own. This article is how that works, and what it cost.

The recursive problem

rysh's UI is built with Bubble Tea, which owns the real terminal: raw input, the actual alternate screen, final rendering. But a program running inside a pane also believes it owns a terminal — because it does: each pane's shell runs attached to a PTY, with TERM=xterm-256color advertised, so programs feel entitled to the full escape-sequence repertoire.

Someone has to be the terminal those escape sequences address. That's the emulator: rysh wraps vt10x (in an internal/vterm package) as a thread-safe 2D screen buffer per pane. The pipeline:

program → PTY → rawReadLoop (4KB byte reads) → vt10x state machine
                                             ↘ output topics (line stream)

The read loop is byte-based, not line-based — an early and important correction. Line-based reading is fine for ls; it's fatal for full-screen programs, whose output is a stream of cursor moves and partial updates that never resemble lines. Every 4KB chunk feeds the emulator's state machine, which maintains what the screen should look like: a grid of cells, a cursor position, and mode flags.

The trick: let the program tell you what it is

How do you know when a pane stops being a scrolling shell and becomes a full-screen app? You don't guess — the program announces it. Full-screen programs enter the alternate screen buffer (the smcup dance) on startup and leave it on exit. vt10x tracks this as a mode flag (ModeAltScreen), and rysh watches it:

• Alt-screen activates → the pane automatically enters raw mode.
• Program exits, alt-screen deactivates → raw mode ends, scrollback rendering returns.

No configuration, no per-program allowlist. vim switches the pane because vim says it's a full-screen program. (For the odd program that misbehaves, ##raw toggles manually.)

Raw mode changes both directions of the pipe:

Output: the TUI stops rendering the line-oriented scrollback and instead paints the emulator's grid. The pane's snapshot — rysh's UI renders exclusively from snapshots requested over its NATS bus, never from shared memory — carries RawMode, VTScreen []string, and VTCursorRow/Col, and the view draws exactly that.

Input: every keystroke is forwarded to the PTY as bytes, except one escape hatch: Ctrl+O (prefix mode — switch panes, detach). Everything else, including keys that normally mean something to rysh, belongs to vim now. As it must: a multiplexer that steals Ctrl+P from your editor is a bug with a settings page.

Keys are a translation problem too

Bubble Tea hands the UI structured KeyMsg events — it already consumed the real terminal's raw input. But the program in the pane needs bytes: arrow-up must arrive as ESC [ A, F5 as its CSI sequence, Ctrl+C as 0x03. So rysh carries a reverse-translation table (internal/tui/keys.go) mapping key events back to the byte sequences a terminal would have produced: arrows, function keys, Ctrl combinations, plain runes.

Latency discipline matters here. Raw keystrokes take rysh's shortest path — published as MsgRawKeyInput straight from the TUI to the pane's subject, one hop, bypassing the entire actor hierarchy (the control-plane/data-plane split exists for exactly this). Interactive editing is the most latency-sensitive thing a terminal does; it gets the least machinery.

Size is a contract

Full-screen programs care intensely about dimensions — htop lays out to the column. Three rules keep the contract:

1. PTYs start at a sane 80×24 at shell spawn, corrected on first render.
2. On every window resize, the TUI computes each pane's content area (minus borders, title bars, stacked-pane decks) and sends MsgPaneResize.
3. The resize lands in both places that believe in a size: pty.Setsize() (so the kernel delivers SIGWINCH and the program reflows) and the vt10x emulator (so the grid matches what the program now draws). Miss either and you get the classic torn-screen artifacts.

One more audience: the remote viewer

rysh panes can be shared — to another local pane (##pane listen) or to a teammate through the collaboration server. A shared grid would be the wrong artifact for a remote text stream, so even in raw mode the shared-output pipeline keeps receiving ANSI-stripped text: a remote observer watching your pane while you're in vim sees readable content, not a replay of cursor-addressing codes. Two renderings of one PTY stream, each fit for its consumer.

Trade-offs, honestly

• Double emulation is real work. Program → vt10x grid → Bubble Tea render → your actual terminal emulator. Every full-screen frame is processed twice. That's the price of hosting; we've kept it acceptable by keeping the hot path short (4KB reads, direct topic publishes), not by pretending the cost isn't there.
• vt10x is not a complete xterm. It covers the sequences that matter for the vim/htop/less class exhaustively. Advertising xterm-256color while emulating a subset is a bet that mostly pays; when it doesn't, ##raw off is the honest fallback.
• This is where our no-locks purity bends. The rawReadLoop goroutine writes the emulator while snapshot builds read it, so the vterm wrapper holds a real sync.Mutex — one of exactly two documented exceptions to rysh's no-mutexes-in-actors rule. Byte-driven worlds and message-driven worlds meet somewhere; we chose to fence that meeting point and write it down.
• What we'd do differently: damage tracking. The emulator knows which cells changed; today's snapshots ship the whole grid. Dirty-region snapshots are the obvious optimization we haven't earned yet.

The test, though, is passed where it counts: open a pane, run vim, edit, quit — scrollback intact, agent conversation where you left it. A pane that can host your editor and your agent is the whole point.

rysh is in early access and we're recruiting design partners — teams who want agents in their real workflow and will tell us what breaks. Direct line to the founders: rysh.ai/design-partner.

More in this series: No mutexes: what sequential mailboxes buy you · Snapshot-driven rendering