10. End-to-End Flows

This chapter ties the sub-projects together with cross-cutting sequence diagrams. Each flow shows how a real user action travels through the actor system, the NATS bus, the agentic engine, and (where relevant) the cloud server and external surfaces.

10.1 Local AI coding turn (with approval)

A developer types a prompt in an AI pane; the agent edits a file, which requires approval.

sequenceDiagram
    autonumber
    participant TUI
    participant WS as WorkspaceActor
    participant Pane as PaneActor
    participant M as LLMPromptExecutionActor
    participant O as OrchestratorActor
    participant Claude
    participant Tool as file_edit tool

    TUI->>WS: MsgSubmitInput ("@ refactor X") → ws.inbox
    WS->>Pane: MsgPaneSubmitInput → pane.{id}.inbox
    Pane->>M: MsgAgenticPrompt → llm_prompt_execution.inbox
    M->>M: append user turn, trim 50, sys+memory
    M->>O: spawn orchestrator
    O->>Claude: CompleteWithTools(conv, specs, sys)
    Claude-->>O: text + tool_use(file_edit)
    O-->>TUI: stream text (output.ai)
    O->>Tool: Execute (compute diff)
    O->>TUI: MsgApprovalRequest(diff) → approval.request
    TUI-->>O: MsgApprovalResponse(yes) → approval.response
    O->>Tool: apply edit
    Tool-->>O: ToolOutput (success)
    O->>Claude: CompleteWithTools(... tool result ...)
    Claude-->>O: text, end_turn
    O-->>TUI: stream final text
    O->>M: MsgOrchestratorDone(conversation)
    M->>M: merge + persist to KV

Concrete behavior: the approval request (step 9) is published to pane.{id}.approval.request; waitForApproval blocks up to 5 minutes before defaulting to No. The diff is computed by executing file_edit first (preview-then-confirm). On MsgOrchestratorDone, the manager replaces its conversation with the orchestrator's full version, trims to 50 turns, and persists to KV ({paneID}.llm_conversation) at most every 2s.

10.2 Cloud-driven browser control

A server-side agent, reachable via the Chrome extension, navigates and reads a page.

sequenceDiagram
    autonumber
    participant U as User (side panel)
    participant Ext as Extension (React)
    participant Proxy as browser_pane_proxy
    participant M as LLMPromptExecutionActor (server)
    participant O as OrchestratorActor
    participant Claude
    participant BA as browser_action tool
    participant Exec as Extension executor
    participant Page as Active tab

    U->>Ext: prompt + page context
    Ext->>Proxy: MsgAgenticPrompt (NATS over WS)
    Proxy->>M: rysh.pane.{id}.llm_prompt_execution.inbox
    M->>O: spawn
    O->>Claude: CompleteWithTools
    Claude-->>O: tool_use(browser_action: click)
    O->>BA: Execute
    BA->>Proxy: MsgBrowserActionRequest → browser.request
    Proxy->>Exec: (NATS over WS)
    Exec->>Page: chrome.scripting (resolve selector, click)
    Page-->>Exec: result
    Exec->>BA: MsgBrowserActionResponse → browser.response
    BA-->>O: ToolOutput
    O->>Claude: CompleteWithTools(... result ...)
    Claude-->>O: text, end_turn
    O-->>Ext: stream output (.llm_prompt_execution.output)

Note the subject-ACL is bypassed for this trusted path: browser_pane_proxy uses an explicit allow-list of pane subjects rather than the streaming ACL.

10.3 Website chatbot session (with operator takeover)

A visitor chats with an embedded widget; a human operator takes over mid-conversation.

sequenceDiagram
    autonumber
    participant V as Visitor (widget)
    participant CP as chatbot_pane_proxy
    participant CPS as ChatbotPaneService
    participant M as LLMPromptExecutionActor (PaneToolsNone)
    participant Claude
    participant Op as Operator (dashboard)

    V->>CP: chatbot.visitor_message {type,payload}
    CP->>CPS: rysh.pane.{id}.output.chat (echo) + LLM inbox
    CPS->>M: MsgAgenticPrompt
    M->>Claude: CompleteWithTools (no tools)
    Claude-->>M: text
    M->>CPS: output.chat (assistant)
    CPS->>CP: chatbot.message
    CP->>V: assistant reply
    Op->>CPS: POST /sessions/:id/takeover
    CPS->>CPS: suppress LLM
    Op->>CPS: POST /sessions/:id/reply
    CPS->>CP: chatbot.message (human)
    CP->>V: operator reply
    Op->>CPS: POST /sessions/:id/release
    CPS->>CPS: resume LLM

The widget never sees raw NATS subjects — chatbot_pane_proxy translates between the closed {type, payload} protocol and the internal pane subjects (chatbot.visitor_message ↔ LLM inbox; output.chat → chatbot.message; status phase → chatbot.typing/typing_stop). The pane has zero tools (PaneToolsNone) because it runs on an untrusted third-party site. Transcript writes are batched (10 msgs / 500ms); idle sessions are reaped every 5 min past the configured timeout (default 30 min).

10.4 Humanoid answering a Slack thread

A teammate mentions the bot in Slack; a humanoid agent replies.

sequenceDiagram
    autonumber
    participant Slack
    participant SA as Slack adapter (CLI)
    participant H as HumanoidActor
    participant M as LLMPromptExecutionActor
    participant Claude

    Slack->>SA: AppMentionEvent (Socket Mode)
    SA->>H: MsgHumanoidInboundMessage (via inboundLoop)
    H->>H: build contextual prompt (thread history + memory)
    H->>M: MsgAgenticPrompt
    M->>Claude: CompleteWithTools
    Claude-->>M: text (end_turn)
    M->>H: stream output (external buffer)
    H->>SA: routeOutboundToChannel(reply)
    SA->>Slack: chat.postMessage (in-thread)

For email humanoids in human-governed mode, the agent drafts via email_draft, the human reviews, and email_send (which requires approval) sends — the agent never auto-sends.

10.5 Pane/tab sharing across machines

A developer shares a tab to a teammate, who mirrors it in control mode.

sequenceDiagram
    autonumber
    participant SrcCLI as Source CLI
    participant SR as ShareRegistryActor
    participant US as UpstreamShareActor
    participant Srv as rysh-server NATS
    participant SS as ShareService
    participant SubCLI as Subscriber CLI
    participant MTL as MirrorTabListenerActor

    SrcCLI->>SR: ##share tab (share.registry.inbox)
    SR->>US: spawn per share
    US->>Srv: register (ws.{wsID}.share.{id}.register)
    Srv->>SS: SetupNATSHandlers records SharedEntity
    US->>Srv: layout doc + output (600ms/150ms loop)
    SubCLI->>Srv: ##upstream subscribe {id} control
    Srv->>SS: subscriber event
    Srv->>SubCLI: replay state + live output
    SubCLI->>MTL: build synthetic mirror tab
    MTL->>MTL: reconstruct via own VTerm, reflow to width
    SubCLI->>Srv: control-mode op/input
    Srv->>US: command (.command) + ack (.command.ack)
    US->>SrcCLI: apply to local actors

The subject-ACL ensures both CLIs only touch ws.{workspaceID}.>. Stale shares are reaped after a 2-minute heartbeat gap.

10.6 Session attach / restore

sequenceDiagram
    autonumber
    participant User
    participant Main as main.go
    participant Sess as session store
    participant Daemon
    participant KV as JetStream KV

    User->>Main: rysh
    Main->>Sess: find live session?
    alt no daemon
        Main->>Daemon: spawn (rysh daemon ), goHeadless
        Daemon->>KV: actors restore tree in Started
    end
    Main->>Daemon: attach TUI (connect NATS client port)
    Daemon-->>Main: ws.snapshot (layout)
    Main->>Main: stream content, render
    User->>Main: Ctrl+? detach (SIGUSR1) — daemon keeps running

10.7 Voice prompt

sequenceDiagram
    participant U as User
    participant Ctl as voice.Controller
    participant Rec as Recorder (sox/ffmpeg)
    participant Tr as Transcriber (Deepgram/Whisper)
    participant TUI
    U->>Ctl: start (push-to-talk)
    Ctl->>Rec: record 16kHz mono WAV
    U->>Ctl: stop
    Ctl->>Tr: Transcribe(wav)
    Tr-->>Ctl: text
    Ctl->>TUI: fill prompt field
    Note over TUI: user reviews, then submits as a normal AI prompt (§10.1)