Automated Software Verification in the Age of AI Coding Agents
We are residing in peak AI, with a growing number of designers delivering code without really composing (or often evaluating) it themselves. According to a 2026 research study by Faroswhich evaluated 2 years of telemetry from 22,000 designers throughout 4,000 groups, engineering throughput is increasing, while bugs, revamp, evaluate load, and production events are increasing even much faster in companies with high AI adoption.
The numbers stand out: Faros reported 54% more bugs per designer, an approximately fivefold boost in mean PR evaluation time, and more than 3 times as numerous events per PR when comparing durations of lower and greater AI adoption. These are connectionsnot evidence that AI triggered each of those results. A subsequent Q3 report Discovered that some of the earlier impacts were alleviating, while confirmation, removal, and functional expenses continued to stack up downstream.
We require to decrease enough to comprehend, and enhance, what we are doing here. One can not anticipate a little group to meaningfully examine massive AI-generated diffs. At the speed at which AI can now press out code, even normal evaluation practices are having a hard time to maintain.
How do we maintain the speed of AI-assisted advancement without compromising software application quality, dependability, and maintainability?
Simple: automated software application confirmationWith the exact same smart tools, abilities, and utilizes we utilize as AI engineers to press code at breakneck speed, we can develop intricate screening and confirmation automations.
Automated software application confirmation that constructs warranted trust and self-confidence is the method forward. Not since any system can ensure bug-free software application, however due to the fact that we require more methods to capture issues before they reach users.
What’s readily available today?
There are a couple of ideas circling at the minute.
Superpowers
One principle that has actually captured on is Superpowersan agentic advancement approach and collection of abilities that deals with coding representatives and supported harnesses. It motivates your representative to act more like a disciplined software application engineer rather of instantly delving into code.
It can assist improve what is really being developed rather of simply presuming things. It stresses preparation, test-driven advancement, debugging, and evaluations, consisting of having different representatives examine the work.
This is a terrific start. Depending on the job, following the entire approach can slow down advancement, and it still does not change the wider checks required to get self-confidence in our code’s accuracy and security. No plugin alone makes software application bug-free or vulnerability-free.
RoboRev
Another intriguing structure, RoboRevoriginates from Wes McKinney, the designer behind the extensively utilized pandas library. RoboRev brings constant AI code evaluation into the regional advancement loop.
The concept is engaging: similar to in-person peer evaluations, we can have another LLM evaluation code that our very first LLM composed. Having a different customer, possibly utilizing a various design, might capture errors or architectural issues that the authoring representative missed out on. It is another viewpoint, though not an assurance of an objective or appropriate evaluation.
In a normal RoboRev setup, you devote your code and a regional daemon immediately evaluates the dedicate utilizing a set up coding representative. It surface areas findings that you can feed back into your existing LLM and coding harness, repairing problems while the context is still fresh.
RoboRev likewise supports follow-up evaluation workflows and targeted evaluation jobs. The appeal is that confirmation enters into the coding loop, not something you keep in mind to do at the very end. Its paperwork covers the readily available commands and combinations.
The idea of code evaluation is not brand-new. In 2026, companies such as Claude, Codex, GitHub Copilot, Cursor, Greptile, CodeRabbit, and Qodo use AI-assisted code evaluations.
Discussing all of them would put even the most terminally online engineer to sleep, so rather let us concentrate on the primary ones.
Claude Code Review
Anthropic’s hosted Code Review is a durable PR customer. When activated on a GitHub pull demand, it sends out several specialized representatives to examine modifications in parallel and try to find reasoning mistakes, security vulnerabilities, regressions, and missed out on edge cases, to name a few things.
The objective is a much deeper evaluation that might take more time and cost more than a fast regional pass. Since October 2026, Anthropic lists this hosted service as a research study sneak peek for Team and Enterprise strategies, with different usage-based billing
There is likewise an Anthropic-maintained /code-review plugin for Claude Code. It runs an evaluation from your coding workflow and can publish findings to a PR. It is a various workflow from the hosted evaluation service, so I would not treat them as similar in depth, setup, or prices. Inspect your strategy and use terms before presuming the evaluation is consisted of.
OpenAI Codex Review
Codex code evaluation takes a comparable method to examining pull demands versus the repository’s context and guidelines. You can ask for an evaluation with @codex review on a linked GitHub PR or set up automated evaluations. Codex likewise supports examining regional modifications before a PR exists.
You can be more detailed about what you desire inspected: a diff, a PR, security issues, or compliance with the requirements. If the evaluation discovers an issue, you can ask Codex to prepare a repair, examine the resulting modification, and repeat the evaluation. That provides you a helpful review-and-repair loop, although evaluation and execution still require independent confirmation.
GitHub Copilot Code Review
GitHub has among the easiest, most hands-free techniques. You can set up Copilot Code Review to immediately examine PRs, follow repository directions, and recommend repairs that can be used or passed to the Copilot coding representative.
As smooth as it is, it presents issues if you begin blindly accepting its feedback. Copilot usually leaves remarks instead of an authorizing evaluation, though GitHub supports setting up approvals in some setups. In any case, automated evaluation is not the like a safe combine choice
A tradeoff is that the evaluation experience is more handled than a DIY representative workflow: you can personalize its directions, however you do not have the exact same direct, per-run design setup you get when invoking your own coding representative.
Cursor Bugbot
Cursor Bugbot immediately examines GitHub PRs, trying to find reasoning bugs and other issues, and lets you act upon the findings in Cursor. Its Autofix function can likewise propose repairs through cloud representatives.
Its most significant benefit is benefit for groups currently utilizing Cursor. The tradeoff is that much of the design choice and evaluation orchestration is managed by Cursor instead of exposed as a completely personalized workflow. Cursor states it utilizes a mix of frontier and internal designs.
One customer isn’t adequate
I like having a strong coding design compose the code and a different strong design evaluation it. Various designs can bring various point of views, however design variety isn’t magic: 2 representatives can settle on the exact same incorrect response, ignore the exact same requirement, or with confidence flag a bug that does not exist.
The useful takeaway is to integrate AI evaluation with checks that do not depend on an LLM’s judgment. And preferably, keep a human liable for the decision, particularly on high-risk modifications.
Okay, so we can evaluate code. There is much more we can do to increase our self-confidence that the code satisfies specification, does not present bugs or vulnerabilities, and does not unintentionally expose tricks.
That’s what we speak about next.
The CI/CD pipeline in the age of AI and automated software application advancement
Here is a detailed confirmation pipeline, constructed as a superset of what is possible. Each action can include self-confidence that the code works as meant, though not all groups will require every check, and no set of checks can show software application is best. The more comprehensive approach is well developed in software application engineering and lines up with structures such as NIST’s Secure Software Development Framework
Not a fan of walls of text? There’s a diagram of the entire pipeline at the end of this area.
Code quality & & construct checks
- Linter checks: Examine code for design offenses and typical errors.
- Formatter checks: Ensure format, such as imprint or spacing, follows our guidelines.
- Type monitoring: If utilizing a typed language such as TypeScript, confirm that types are utilized properly.
- Compilation/build checks: Ensure the application puts together or develops effectively.
Automated tests
- System tests: Test whether private systems of performance work as meant.
- Combination tests: Test whether numerous parts interact, such as a payment circulation that touches an API, database, and payment company.
- End-to-end (E2E) tests: Confirm crucial user workflows throughout the application from start to complete.
Security & & dependence checks
- Fixed application security screening (SAST): Scan source code for patterns related to prospective security vulnerabilities.
- Secret scanning: Try to find passwords, secrets, or delicate qualifications mistakenly consisted of in the repository.
- Reliance vulnerability scanning: Discover understood vulnerabilities in plan dependences.
- Dependence freshness and assistance status: Recognize out-of-date or unsupported plans that might require attention.
Agentic code evaluation
- General bug scanning: Usage AI representatives to discover and flag most likely bugs.
- Security-focused evaluation: Usage AI representatives to examine prospective security dangers beyond what a rules-based scanner may capture.
- Evaluation versus specification: Inspect whether the current PR or function branch executes the asked for habits, consisting of essential edge cases.
Facilities & & database checks
- Container and facilities setup scanning: Search for insecure settings or misconfigurations in containers and cloud facilities.
- Database migration recognition: Examine whether schema modifications are safe to release, may trigger information loss or downtime, or make rollback hard.
Architecture & & maintainability evaluation
- Bloat, intricacy, and architecture simplification checks: Usage AI representatives to recognize unneeded code, overengineering, extreme intricacy, and chances to streamline the execution.
Other checks worth keeping in mind
Depending upon the application, include efficiency and load screening availability checks personal privacy and compliance checksand software application license/ supply-chain checksA safe and secure implementation can still be sluggish, unattainable, or mishandle information. These checks do not need to operate on every devote, however they ought to have a location in the general procedure.
Putting everything together, how can we set up these checks?
The majority of the confirmation patterns we’ve gone over have actually been developed practices for years, and significant platforms currently supply tools to automate them. GitHub Actions, GitLab CI/CD, and Jenkins prevail methods to manage them.
Take GitHub Actions. Designers specify automated workflows in YAML files kept within their repositories. These workflows can be set off by pressing code, opening a pull demand, or combining modifications. GitHub then begins an execution environment (a runnerand carries out the workflow actions.
These actions frequently include setting up dependences and running deterministic checks such as linters, tests, and security scanners. Progressively, they can likewise conjure up AI representatives for more advanced, nondeterministic evaluations. See GitHub’s Actions paperwork and the main Codex GitHub Action
A streamlined workflow may appear like this, in .github/workflows/verification.yml:
name: Code Verification
on:
pull_request:
permissions:
contents: read
jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- run: ./ci/quality.sh
tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- run: ./ci/tests.sh
security:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- run: ./ci/security.sh
infrastructure:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- run: ./ci/infrastructure.sh
agent-review:
# This example skips external forks and Dependabot-triggered runs;
# bot reviews need separate setup.
if: github.event.pull_request.head.repo.full_name==github.repository && github.actor !='dependabot'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
persist-credentials: false
- uses: openai/codex-action@v1
with:
openai-api-key: ${{ secrets.OPENAI_API_KEY }}
permission-profile: ':read-only'
prompt: |
Review the changes in this PR relative to
base commit ${{ github.event.pull_request.base.sha }}.
Use git diff to inspect the changes.
Check for:
1. Logic bugs and missed edge cases
2. Security vulnerabilities
3. Compliance with the project specification
4. Unnecessary complexity or overengineering
Report actionable findings with file paths
and line numbers. Do not modify files.
This is an illustrative workflow, not a drop-in CI setup. The ./ci/*.sh scripts are placeholders for scripts you would compose in your repository. The representative task runs an evaluation however does not instantly publish the findings back to the PR; that needs an extra reporting action. In a genuine job, likewise pin third-party actions to relied on devote SHAs, limit approvals, and thoroughly different code from untrusted factors and workflows that can access API secrets. GitHub and OpenAI both release assistance on safe and secure PR workflows and Codex Action security
Keep in mind that with an agentic evaluation, you can configure your own LLM company and qualifications in GitHub Actions. The main Codex Action utilizes an OpenAI API secret; utilizing another service provider would need that supplier’s matching combination.
Running this entire pipeline for each PR can get pricey, particularly for groups practicing constant combination with numerous PRs throughout the day. Some checks and agentic evaluations take minutes instead of seconds, and the expense of the AI calls can build up.
It is much quicker and easier to run inexpensive checks early and conserve the much heavier ones for later phases or higher-risk modifications.
This is where the pre-commit, post-commit/push, pull demand, pre-production, and production phases can be found in.
Preferably, in a fully grown screening workflow, we classify the checks by phase and by the threat of the modification.
Still text-avoidant? Another diagram is awaiting you at the end of this area.
Pre-commit
This is the phase before code is dedicated in your area.
What need to take place here: Formatting, linting, type monitoring, focused system tests, develop checks when quick enough, and secret scanning. A few of these can go through Git hooks; others can run straight in your editor or coding-agent harness.
Post-commit/ push
This is the phase after code is dedicated and pressed to a remote repository.
What ought to take place here: More comprehensive system and combination tests, SAST, and dependence vulnerability scanning. Regional post-commit evaluation tools can likewise offer quick feedback after each devote.
Pull demand (PR)
This is where proposed modifications are evaluated and confirmed before combining into the primary branch.
What must take place here: Pertinent automated tests, agentic bug scanning, agentic security evaluations, requirements compliance evaluations, and architecture/complexity evaluations. More thorough checks must be booked for higher-risk modifications. Need essential checks to pass in the past combine utilizing repository rulesets
Pre-production/ release
This is where code is validated before being launched into the production environment.
What need to occur here: Complete combination and E2E tests where proper, facilities setup scanning, database migration recognition, implementation smoke tests, and last security checks. For dangerous releases, think about staging or canary releases and an evaluated rollback strategy.
Production/ post-deployment
This is the phase after code has actually been released and is running in production.
What need to occur here: Application medical examination, mistake tracking, efficiency tracking, security tracking, smoke tests, anomaly detection, and automated rollback when it is safe to do so.
How huge tech runs AI software application factories
Some big business have actually been playing with this subject for several years now, as they have the skill to pull it off and the deep pockets to tokenmax. As Adam Faik sets out in How to construct an AI-native software application factoryUber, Shopify, DoorDash, Spotify, and Stripe all use a look into where the cutting-edge lives today in this continuously developing software application factory lifecycle.
Simply put, groups normally get an entrance or front door that routes their demands, those demands are certified through appropriate authorizations and governance, then pressed through a mainly automatic pipeline. This pipeline has access to the code, business context, and internal tools it requires to look into the demand, arrive at a specification, propose a strategy, construct the important things, test the important things, and have evidence to reveal for it. Diffs, screenshots, tests, automated evaluations, and a great deal of analytics, which are then evaluated by designers before code is in fact combined or pressed to production.
Simply put, designers progressively concentrate on developing the issue to be resolved, offering some input on how that may take place, and letting the agentic workflow manage much of the application and confirmation. When done, designers act as glorified customers and last gatekeepers before the PR combine or production push. Obviously, the degree of autonomy differs in between business, and human beings are still associated with essential choices along the method.
There is a lot more to this procedure, especially around security, separated environments, confirmation, and expenses, and numerous information I’ve overlooked, however I believe Adam’s post is a remarkable read for anybody curious about the various methods these AI software application factories are set up and performed at scale.
The useful agentic software application advancement workflow I utilize
For those of you who have actually gotten this far, enable me to stroll through one PR to bring the case home.
A diagram summarizing my complete pipeline follows at the end of this area.
State I have a brand-new demand to develop a profile page for my app so users can see and upgrade their profiles.
Utilizing a strong coding design (frequently Claude in my setup) I discuss in as much information as I can summon what the profile page ought to do and appear like.
My agentic workflow returns a specification. We go back and forth a number of times, I evaluate it thoroughly, and we upgrade it till I’m pleased. I then ask the representative to produce an application strategy gotten into tactical, useful, consecutive actions that can realistically be divided into dedicates.
I likewise ask it to run the strategy by a 2nd top-tier design, such as Codex, to see if anything was missed out on or might be enhanced. The last strategy consists of the system, combination, and end-to-end tests we plan to construct.
I examine the strategy, repeat on it if required, and authorize it.
At that point, my primary coding representative begins dealing with the strategy, action by action or devote by dedicate. After each action, it asks another strong design to evaluate the modifications.
If the modification is high-stakes, for instance if it touches payments or delicate production circulations, I likewise examine each dedicate in the meantime. Otherwise, I might wait till the entire function is ended up before doing my deep human evaluation.
When the coding representative gets recommendations for enhancement, either from the 2nd LLM or from me, it includes those modifications and proceed to the next action.
This continues dedicate by dedicate, with modifications to the frontend, backend, or any other code required. Along the method, the representative runs system tests, combination tests, and E2E tests as proper. For the software application I construct and preserve, a number of these tests are fairly inexpensive and quick to run. Stops working, it gets flagged, and we remedy it.
When we get to the last dedicate, I ask another design to examine the whole modification— all devotes and all diffs– versus the initial specification and flag anything that was missed out on. If something is flagged, we examine and repair it. Otherwise, we carry on.
Now we are at the PR phase. We’ve executed, examined, and re-reviewed the code versus the specification and run tests along the method to guarantee whatever still works.
I then examine the complete diff myself, offering it attention commensurate with the gravity of the modification. The more crucial the modification, the more cautious the evaluation. I likewise run the other automated checks: SAST, reliance scanning, agentic bug evaluation, and agentic security evaluation. A few of these usage AI representatives; others utilize deterministic tools.
I ask the representative that developed the function to come up with brief and easy handbook tests I can do to validate that the modification works.
I introduce the code in your area or in a test environment, run those checks, and release to production if whatever passes.
If the modification is extremely complicated or touches delicate information, I might likewise run a minimal, safe set of smoke tests in production, without exposing genuine user information or making devastating modifications.
This circulation permits me and my group to deliver code rapidly while developing self-confidence that what we deliver satisfies specification and has actually gone through significant screening and security checks. Not ideal self-confidence(software application is never ever ensured bug-free or difficult to hack), however much more self-confidence than blindly combining whatever a representative creates.
And this is truly the point: if AI makes composing code low-cost, confirmation needs to end up being a superior part of how we develop software application.
If you’re curious about how I run this light-weight workflow harness, let me understand and I ‘d more than happy to share it with you.
Up until next time, might your tests pass and your context windows never ever go out.
~ Mihai
Sources and recommendations
The sources listed below assistance the research study findings, item abilities, and CI/CD/security assistance pointed out above. Item functions, strategy accessibility, and rates can alter.
- Faros– The Acceleration Whiplash (Q2 2026): 2026 AI engineering reportResearch study scope, throughput, evaluation load, bugs, and events.
- Faros– The Speed Trap (Q3 2026): Newest report takeawaysSubtlety on later AI-adoption patterns and downstream expenses.
- Superpowers (Jesse Vincent/ obra): GitHub repositorySkills-based advancement workflow, preparation, TDD, debugging, evaluations.
- RoboRev: Authorities site and documentsRegional constant code evaluation, Git hooks, evaluation lifecycle.
- Wes McKinney: GitHub profilepandas developer and Kenn Software creator.
- Anthropic: Establish Code Review for Claude CodeHosted evaluation architecture, eligibility, prices, and activates.
- Anthropic: Claude Code
/code-review pluginPlugin functions and workflow.
- OpenAI: Evaluation GitHub pull demands with Codex and Codex PR evaluation aidPR evaluations, triggers, and follow-up actions.
- GitHub: Utilizing GitHub Copilot code evaluationAutomatic evaluates, recommended modifications, evaluation status, and modification.
- Cursor: Bugbot and Bugbot AutofixPR evaluation and helped repairs.
- NIST: Protect Software Development Framework (SSDF), SP 800-218Security practices throughout the software-development lifecycle.
- GitHub: GitHub Actions documentsrepository rulesetsand protected
pull_request_target useWorkflows, status checks, and trust limits.
- OpenAI: Codex GitHub Action and security assistanceAction setup, read-only approvals, and secret-handling warns.
- Adam Faik, The AI Thinker: How to develop an AI-native software application factory (October 2026). How Uber, Shopify, DoorDash, Spotify, Stripe and others run coding representatives at scale, from consumption to combine.
