Every AI vendor says they have "quality checks." Almost none of them will show you what that phrase actually contains. Here is what we mean when we say rig, and what to ask for when someone else uses the word.
Key takeaways
- A rig is a fixed, versioned set of scenarios the system must pass before any AI-generated change ships — not ad hoc testing done at the reviewer's discretion.
- A real rig has four layers: functional correctness, edge-case and adversarial scenarios, regression coverage over time, and a human sign-off gate for anything the rig can't fully verify.
- The rig is versioned and grows with the system. A rig that hasn't changed in three months is either testing a system that hasn't changed, or it has stopped being maintained.
- Ask a vendor for their rig's scenario count and how often it has caught something. A vague answer means there isn't a real rig behind the claim.
Why "we review the AI's output" is not an answer
Human review of AI-generated work is necessary but not sufficient. A reviewer checks whether a change looks correct against the ticket in front of them. They are not, in that moment, checking it against every scenario the system has ever needed to handle correctly. That is what a rig is for: a standing, automated memory of everything that has to keep working, run against every change, every time, regardless of how confident anyone feels about a particular diff.
Layer one: functional correctness
The baseline layer, and the one every vendor already has in some form: does the feature do what it is supposed to do, under normal conditions. This is standard automated testing. It is necessary and almost never sufficient on its own, because it only tells you the happy path still works.
Layer two: edge cases and adversarial scenarios
This layer is deliberately built to break things: malformed input, boundary values, concurrent access, the scenario a normal test writer wouldn't think to include because it is not how the feature is "supposed" to be used. AI-generated code is not more or less likely to fail here than human-written code in principle, but it is generated faster and reviewed under more time pressure, which means this layer catches more in practice than it would on a slower, human-only codebase.
Layer three: regression coverage over time
Every bug the rig has ever caught becomes a permanent scenario in the rig. This is the layer that compounds. A rig that is six months old and has caught thirty issues is meaningfully stronger than a fresh one, because it encodes thirty specific ways this particular system has previously gone wrong — not generic best practice, but this system's actual history of failure modes.
Layer four: the human sign-off gate
Some things a rig cannot verify on its own — whether a change is consistent with an unwritten business rule, whether a UI change reads correctly to a human, whether a contract clause was interpreted the way legal intended. The rig's job at this layer is not to replace the reviewer, but to hand them a shortlist: exactly the things it could not verify automatically, so the reviewer's attention goes where it is actually needed instead of being spread across the entire diff.
The question that separates a real rig from a marketing claim
Ask any vendor claiming AI-assisted delivery two questions: how many scenarios are in your rig, and can you show me one it has caught recently. A team running a real rig will answer both without hesitation, because they look at the numbers regularly. A team that only has the phrase "quality checks" on a slide will not.
Outcome-priced from day one
See what this would cost at Effektiv pace.
Pick a project that finished or stalled. Show us a quote you've received or an invoice you've paid. We'll price the same scope on outcomes, not hours.