The governance gap: why most enterprise AI pilots die in the demo
The demo is great. Then security asks 'what exactly did it do?' — and nobody can answer. Pilot purgatory is a governance problem.
There's a graveyard in every mid-size company right now, and it's full of AI pilots.
The pattern is so consistent you can set your watch by it. An engineer or an innovation team builds something genuinely impressive — an agent that triages tickets, drafts responses, automates a workflow. The demo lands. Leadership is excited. The word "rollout" gets used.
Then the pilot enters review, and three people ask three questions:
- Security: "What exactly can this thing do, and what did it do last week?"
- Legal / compliance: "Where does our data go, and who can audit the decisions?"
- The team that has to live with it: "What happens when it does something wrong?"
Nobody has good answers. The pilot doesn't get killed — that would at least be a decision. It gets paused. Extended. Scoped down to a sandbox. Revisited next quarter. It dies the way most pilots die: not from failure, but from the inability to answer for itself.
This is pilot purgatory, and the cause is almost never the model. It's the governance gap.
The gap, precisely
Most AI pilots are built to demonstrate capability: can an agent do this work? The demo answers yes. But a company doesn't run on capability — it runs on accountability. Between the demo and production sit three questions that capability alone can't answer:
1. Can we see it? Not a log file someone could theoretically assemble — an actual, legible trace of every action the agent took: every command it ran, every file it touched, every message it sent, in order, attributable, reviewable. If the honest answer is "we'd have to reconstruct it," you don't have observability; you have archaeology.
2. Can we gate it? Is there a structural difference between the agent proposing a risky action and doing it? Deleting data, spending money, emailing a customer, touching production — can a human approve those before they happen, without a human babysitting every step? If the only settings are "fully autonomous" and "off," the review meeting will always choose off.
3. Who owns it? Where does the agent's logic live — in a vendor's black box, or in files your team can read, version, and change? Whose infrastructure does it run on? Whose keys? If the vendor disappears or triples the price, does the capability leave with them?
See it, gate it, own it. Every stalled pilot I've seen is missing at least one. Most are missing all three — because the pilot was built on tooling where those properties were never available to begin with.
Why bolting governance on afterward fails
The instinctive fix is to build the demo first and "add guardrails before rollout." It almost never works, for a structural reason: observability and approval aren't features you attach to an agent — they're properties of the runtime the agent lives in.
If the agent runs as an opaque script with an API key, there is no seam where you can insert an approval gate. If its actions aren't events in a system you control, there's nothing to audit. Retrofitting governance onto an ungoverned runtime means rebuilding the runtime — which is why "we'll add controls later" quietly becomes "the pilot is on hold."
The teams that escape purgatory invert the order. They start with a governed runtime, then put agents in it.
What governed-by-default looks like
This is the problem rysh was built around, so let me use it to make the abstraction concrete — the pattern matters more than the product.
The work happens in the open. In rysh, every agent lives in a pane — the same kind of pane your engineers' shells live in. Every tool call an agent makes is visible there as it happens, and the history persists. When security asks "what did it do last week," the answer is: here's the trace. Scroll.
Danger requires a human. Destructive operations — file writes, risky shell commands, commits — are gated behind explicit approval. The agent proposes with a preview (a diff, a command, a description); a person confirms. Safe, read-only operations flow without friction. The gate is policy you tune, not a demo feature.
Customer-facing autonomy is a dial, not a leap. An agent answering external channels can run in draft-and-confirm mode: every reply is a draft a human approves before it goes out. When the drafts are consistently right, you flip that channel to autonomous — and it's still fully observable. Trust is earned in production, with evidence, per channel.
Secrets stay home. Credentials are redacted client-side before output leaves the machine. Shared views never contain raw keys. The blast radius of "the agent saw our environment" is designed down, not waved away.
You own the whole thing. Agent logic is a markdown skill file in your git repo. The platform is self-hostable, running on Claude with your own API key. The audit story doesn't depend on a vendor's cooperation, and neither does your exit.
None of this slows the demo down. That's the point — when the runtime is governed by default, the demo is the compliant version. There's no second, harder project between the demo and the rollout.
The reframe for your next pilot
If you're sponsoring an AI initiative, change the acceptance criteria before anyone writes a prompt. The pilot succeeds when:
- It does real work (the old bar), and
- Anyone in the review meeting can pull up the full trace of what it did, and
- The risky actions demonstrably waited for a human, and
- Your team can point at the file where the agent's logic lives — in your repo, on your infra.
A pilot that clears that bar doesn't enter purgatory, because the questions that kill pilots are answered before they're asked. And the organization learns the most valuable lesson available right now: agents can be trusted with more — when you can see them earn it.
The companies stuck at "we ran some pilots" and the companies where agents quietly do real work aren't separated by model quality. They're separated by the governance gap. Close it first.
We're taking on a handful of design partners — companies that want their first governed agent workflow in production, not in purgatory. If that's you: rysh.ai/design-partner.
Related: What it actually means to be an AI-native company · Is it safe to let an agent loose on your systems?