You built the factory. Verification is the ceiling.
Replay QA is a runtime verification layer that fits into the pipeline you already built. It starts your real app, signs in, exercises the changed flows, and returns structured evidence your agents can act on. Trigger it from your own orchestration via REST API.
The architecture
Everyone who builds one arrives at the same five stages
Signals come in — support threads, customer feedback, production errors. Something triages them: clarifies intent, gathers context, holds the work that isn't ready. Agents build inside the patterns your codebase already has. Something verifies the result. Then the loop closes: the outcome gets reported back, and what you learned shapes the next decision.
Teams reach this architecture independently, without comparing notes, because it's what the work demands. Intake, triage, build, and close-the-loop can be assembled from things that already exist — issue trackers, error monitoring, model APIs, your own orchestration code.
Verify is the one stage where you had to build the machine yourself.
The bottleneck
The bottleneck moved four times. It's not where you started.
Code can be clean, fully tested, and wrong.
Parallel agents don't just produce more code. They produce more code than any fixed number of humans can hold in their heads. Hundreds of pull requests a month arrive at a review capacity that hasn't changed. Every reviewer you add — a second model, a cold-read pass, an adversarial reviewer on the risky changes — buys throughput against the same fundamental limit.
The gap
Every reviewer you added reads the code
Look at what's actually in your verify stage. A model reviewing the diff cold. An adversarial pass from a different model on high-risk changes. Static security analysis tracing data flow through sensitive modules. Unit and integration tests. Every one of them is excellent. Every one of them is reading source code.
| Layer | What it inspects | What it can't tell you |
|---|---|---|
| AI code review (CodeRabbit, Qodo, cold-read model passes) | The diff | Whether the app works |
| Adversarial model review | Intent vs. implementation, in code | What happens at runtime |
| Static security analysis (SAST) | Code paths and data flow | Whether the vulnerability is actually exploitable |
| Unit and integration tests | What someone thought to assert | The flows nobody wrote a test for |
| Runtime verification | The running app | — |
The only layer that answers “does it work” is the only layer you had to build yourself.
Owned vs. bought
You wrote a browser verification harness. Now you own it.
The shape is always roughly the same. It starts the real application. It signs in. It exercises the workflow that changed. It checks browser errors and whether the data actually got written. It captures before-and-after evidence, because a reviewer approving a change they can't see is guessing. And the rule that makes it work: if the evidence can't be produced, the change isn't ready.
It's the right design. It's also a product.
| What your harness does | Replay QA |
|---|---|
| Starts the real app, signs in, exercises the changed workflow | Autonomous journey discovery on every main-branch update or PR — no test authorship |
| Checks browser errors and data writes | Runtime bug detection: component failures, state mutations, race conditions, async timing |
| Captures before/after evidence for the reviewer | Full deterministic Chromium recording of the failing session, attached to the PR |
| Crawls product flows on a schedule | Scheduled daily or weekly runs, producing a prioritized queue |
| No evidence → the change stays unready | The GitHub app is the gate — the report posts to the PR and syncs to your tracker |
Browser drivers. Auth flows that change. Flake triage. Environment drift. Screenshot diffing that has to be good enough to trust. Every improvement to the harness is engineering time that isn't going into your product — and the harness is never finished, because your app keeps changing underneath it.
Is runtime verification infrastructure you want to own, or infrastructure you want to buy? You'd almost certainly buy your error monitoring. You didn't write your own model APIs. This is the same class of decision, and it's the last one still sitting on your side of the line.
The merge call
A human still makes the call. Give them something to look at.
Keeping a human on every merge isn't a limitation you'll grow out of. It's the thing that makes the factory safe to run. We're not trying to take that decision — we're trying to make it take ninety seconds instead of forty minutes.
Every bug report includes the reproduction, the root cause, and a suggested fix, attached to a full recording of the session where it broke. The reviewer doesn't reconstruct what happened from a diff. They watch it.
Integration
Your orchestration calls it. Your agents read the output.
Scheduling
Wake up to a queue your agents can work through
Most factories run a scan overnight, and the morning starts with a prioritized batch of real issues instead of an empty backlog and a feature meeting. Each item arrives with root cause and a suggested fix, formatted for an agent to pick up directly.
Set it in project settings, or ask the project chat in plain language: “Set up a nightly test run.”
If you already built a version of this, the difference is that you stop maintaining it.
Security Scan
Your security review reads code. Ours attacks the running app.
Static analysis traces data flow and flags what looks dangerous. It structurally cannot tell you whether a finding is exploitable, because it never executes anything.
Security Scan runs a full pentest-scope pass against your running app: injection, broken access control, bypassable authentication, insecure API behavior. Because it reasons about how your app is supposed to work, it finds the authorization and business-logic flaws generic scanners miss — IDOR and cross-tenant data exposure in particular. It confirms findings by exploiting them, so what you get is a list of real vulnerabilities, not a list of maybes.
These are the vulnerability classes AI agents introduce at scale. If your agents are writing thousands of pull requests against a multi-tenant database, cross-tenant leakage isn't a ticket — it's an account-losing event.
It sends real attack traffic, so ownership verification is required before a scan runs: serve a short plain-text file from your app at a given path. Staging is the recommended target. Billed at the same rate as a standard pass.
Honest boundary
What this doesn't solve
A passing check doesn't tell you that you understood the original problem. A merged pull request doesn't either. That's a judgment problem — domain understanding, talking to the people who reported the thing, knowing which requests are symptoms of something else. No verification layer touches it, and we're not going to claim ours does.
We narrow it from one side: your crawls test flows against written intent, and Replay QA explores flows nobody specified — so it finds failure paths that sit outside the written intent entirely. That's a smaller claim than solving the intent problem. It's the one that's true.
Proof
What we can show you
Web Debug Bench (Apr 2026)
177 hard, realistic bugs in agent-built web apps. An agent with runtime access solved 76%; the same agent reading code alone solved 61%. The largest single delta of any configuration tested. blog.replay.io/web-debug-bench →
Deterministic recording, not a JS shim.
Chromium-level capture, so a recorded session replays identically every time. This is why root cause analysis works on race conditions and async timing bugs instead of guessing at them.
The practitioner case.
Charity Majors (co-founder/CTO, Honeycomb) argues that AI-written code demands a shift from code review to validation — behavioral testing, observability, and capture/replay as a core primitive. She also notes that under 10% of engineering teams currently operate in short, fast feedback loops.
Frequently asked questions
Architecture questions? Schedule a call.
Bring your architecture
If you've built a factory, you'll want to know how this sits inside it before you try it — trigger model, output shape, throughput, failure modes, what happens when a run is wrong. That's the conversation we'd rather have than a demo.