AI Regression Testing: A Guide for Practitioners

AI regression testing is regression testing in which an AI system helps create, select, maintain, run, or analyze tests after a software change. The model proposes scenarios from the diff, picks which tests matter, repairs stale steps, and explains failures. The tests themselves still run as code, so the same change gets the same verdict twice.

This guide covers the AI layer: how a diff becomes a test, how tests get selected, what self-healing may and may not change, and how a run stays repeatable. The wider picture sits on the AI testing guide. Below, one authorization bug goes from a diff to a saved test that later runs without a model.

What is AI regression testing?

Regression testing is rerunning tests after a change to check that what used to work still works. Building that suite is its own job, covered in the guide to building an effective regression test suite. AI regression testing is the layer above it, where a model helps with one or more of five verbs.

• Create. Read a change and draft the test cases that cover it.

• Select. Decide which existing tests to run for this change.

• Maintain. Propose a repair when a step or a selector goes stale.

• Run. Start the selected tests against the right environment.

• Analyze. Group failures, label them, and draft the explanation.

Vendors disagree about how much of that counts. Autify defines the term narrowly, with an agentic large language model as the engine running natural-language test steps in place of a strict script. TestMu AI defines it broadly: risk-based selection, self-healing maintenance, failure prediction, and pipeline execution. Ask a tool which of the five verbs it does.

Two other things carry the same name. Testing a machine learning model after retraining uses a fixed regression dataset to check that old failures have not returned, which is what Lakera's guide covers. Regression analysis is a statistical method for relating variables, and Cake's page mixes it with test regression. This page is about using AI to test software.

AI regression testing vs scripted regression suites

A scripted suite runs what you tell it to run, and someone maintains it by hand. AI changes who decides at five points and leaves execution alone.

The right-hand column is the whole argument. Use the model where variation is useful, at authoring and triage. Keep it out of replay, where variation is a defect.

AI also does not remove maintenance. It moves it. Instead of editing a selector you review a proposed one, and instead of writing a test you review a drafted one. Less typing, the same judgment.

How AI regression testing works, end to end

The pipeline is short. Each step has one input and one output.

1. Change context. In: the diff, its dependency path, any repository map. Out: the behaviors this change can affect.

2. Candidate scenarios. In: those behaviors. Out: draft cases, some new, some naming existing tests.

3. Selection and prioritization. In: the candidates and the suite. Out: an ordered list to run now, the rest deferred.

4. Live run. In: that list and a deployed preview of the branch. Out: a pass or fail per test.

5. Assertions. In: the response, the page state, or the side effect. Out: a verdict tied to a stated expectation.

6. Failure classification. In: the failures. Out: each one labeled a real defect, a stale test, or an environment problem.

7. Evidence. In: the failing execution. Out: the request, the response, the failing step, or the screenshot.

8. Approved test update. In: a drafted test or a proposed repair. Out: a reviewed commit.

9. Replay. In: the approved test on later branches. Out: the same verdict for the same code, no model in the loop.

Steps one, two, three, six, and seven are where AI earns its place. Steps four, five, and nine should look like a suite you wrote by hand, because that is what they are.

Self-healing tests: what changes and what must not

Self-healing is a test repairing itself when the way it finds an element stops working. A button gets renamed, an identifier is regenerated at build time, and a locator that matched yesterday matches nothing. Healing finds the same element by other means and updates the step.

The implementations differ in mechanism and agree on the inputs. Testim scores each locator and, below 70 percent, tries to improve it, replacing the old one only if the new locator scores better; it records a test revision, and runs automatically only on the master branch unless you enable it elsewhere. That 70 percent is Testim's implementation detail, not an industry threshold. mabl says its agentic tester uses multiple AI models to update locators and steps, builds locators from visual context and several attributes rather than one XPath, and asks a person for intent when it needs clarification. Functionize models element size, position, history, visual configuration, XPath, CSS selectors, parent and child elements, and visibility, then offers ranked one-click repairs. testRigor heals from recorded execution history rather than generating a selector, validates the substitute during execution, annotates the change as fixed by AI, and lets you roll back. All four are vendor descriptions, not independent measurements.

Notice what each exposes: a revision, a review annotation, a ranked suggestion, or a rollback button. No vendor treats a repair as automatically correct, and neither should you.

What healing may change. How the test finds a thing: the selector, the wait, the step that clicks. Repairing those keeps a suite alive through a redesign.

What healing must never change. What the test expects. If a page shows a different total and the test starts passing because the assertion was relaxed, healing has deleted the coverage it existed to protect. An expected value changing is a code change, not maintenance.

Two failure modes to plan for. A heal can match the wrong similar element, so the test keeps passing while clicking a different button. And it can preserve a test that was weak already. Set uncertain repairs to fail or wait for approval. When healed tests fail intermittently, read the guide to flaky tests; for locator and assertion fundamentals, see UI testing.

Test impact analysis vs risk-based test selection

These two get used interchangeably. They answer different questions.

Test impact analysis maps a code change to the tests that exercise the changed code, usually from coverage data collected on earlier runs. It answers: which existing tests touch what I just changed?

Risk-based selection scores tests by how likely they are to catch something, then orders them. TestMu AI describes scoring from code-change impact, past failures, and real user paths, running the highest-risk cases first. It answers: given fifteen minutes, which tests are worth running?

Microsoft's implementation is the clearest published description of the first. It selects impacted tests, previously failing tests, and newly added tests; it falls back to running all tests for commits it cannot reason about, such as changes to HTML or CSS files; and it lets you configure a periodicity at which everything runs anyway. Microsoft recommends setting that periodicity, which tells you how far they trust selection alone.

Use both. Impact analysis gives a defensible floor, risk ranking gives an order, and the scheduled full run stops either from quietly shrinking coverage. For more on the ranking half, see AI in risk-based test prioritization.

From a pull request diff to a replayable regression check

Here is the loop on one change. The service is a small Node API with an endpoint at GET /orgs/:orgId/billing. Pull request 512 is titled "let support staff open any org's billing page". This fixture stands in for your own service; the sequence is the point, not the code.

1. Read the diff and its dependency path. The change swaps the organization id from the session to the URL and adds no membership check. Any caller who knows another organization's id can read its billing record.

--- a/src/routes/billing.ts
+++ b/src/routes/billing.ts
@@ router.get('/orgs/:orgId/billing', requireAuth, async (req, res) => {
-  const orgId = req.session.orgId;
+  const orgId = req.params.orgId;
   const record = await billing.forOrg(orgId);
   res.json(record);
 });

2. Propose the cases. The diff touches an authorization boundary, so the useful cases are about who may read what: the owner reads their own record, a member of another organization is refused, an unauthenticated caller is refused. The second is the one that matters, because a happy-path test passes before and after this bug.

3. Approve it and save it as code. A person reads the drafted test before it can gate anything. What lands in the repository is an ordinary Playwright file, not a stored prompt.

import { test, expect } from '@playwright/test';

const base = process.env.BASE_URL || 'http://localhost:4000';
// Seeded by scripts/seed.ts: two orgs, one admin each, fixed ids, fixed clock.
const orgA = { id: 'org_a', token: process.env.TOKEN_ORG_A_ADMIN };
const orgB = { id: 'org_b', token: process.env.TOKEN_ORG_B_ADMIN };

test('org A admin cannot read org B billing', async ({ request }) => {
  const own = await request.get(base + '/orgs/' + orgA.id + '/billing', {
    headers: { Authorization: 'Bearer ' + orgA.token },
  });
  await expect(own).toBeOK();

  const cross = await request.get(base + '/orgs/' + orgB.id + '/billing', {
    headers: { Authorization: 'Bearer ' + orgA.token },
  });
  expect(cross.status()).toBe(403);
  expect(await cross.text()).not.toContain('org_b');
});

4. Run it against the pull request preview and watch it fail. A new regression test that passes on the branch that introduced the bug is not a regression test. It goes red first, on the deployed preview of this branch, before anyone writes the fix.

Running 1 test using 1 worker
  x  tests/billing-isolation.spec.ts:9:5 > org A admin cannot read org B billing (412ms)
     Error: expect(received).toBe(expected)
     Expected: 403
     Received: 200
  1 failed

5. Attach the evidence. The failing request and its response are what make this reviewable. Expected 403, received 200, on a cross-organization read with org A's token.

6. Fix it, and watch the same test go green.

   const orgId = req.params.orgId;
+  const allowed = req.session.memberships.includes(orgId) || req.session.isSupport;
+  if (!allowed) return res.status(403).json({ error: 'forbidden' });
   const record = await billing.forOrg(orgId);

The rerun prints 1 passed, and the test stays in the suite. Every later pull request executes that saved file. No model is asked for a verdict, so the hundredth run costs what the first one did.

Two controls make this safe rather than clever. A person approves every generated test and every proposed repair, so nothing enters the gating suite unread. And when selection confidence is low, the pipeline runs the full suite instead of guessing.

Run AI regression tests on every pull request

The workflow below seeds the fixture and runs the one test against it. Real pipelines point BASE_URL at a deployed preview of the branch and run the selected set rather than a single file.

name: regression-on-pr
on: [pull_request]
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npm run seed && (npm run start:test &)
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test tests/billing-isolation.spec.ts
        env:
          BASE_URL: http://localhost:4000
          TOKEN_ORG_A_ADMIN: ${{ secrets.TEST_TOKEN_ORG_A }}
          TOKEN_ORG_B_ADMIN: ${{ secrets.TEST_TOKEN_ORG_B }}
      - uses: actions/upload-artifact@v4
        if: failure()
        with: { name: playwright-report, path: playwright-report }

Five things make this a gate rather than a report. The environment is a preview of this branch, not staging shared with everyone. The selected fast checks run first, so common failures come back in minutes. API and interface checks run in the same job, because a bug does not care which layer finds it. The exit code blocks the merge; a check that only comments is a suggestion. And the report uploads on failure, so the evidence outlives the runner. The full suite then runs on a schedule, which is what catches the tests selection skipped. Wiring is covered in the guide to continuous integration testing.

Keep this separate from AI code review, which reads the diff and reasons about it: see Claude code review. Review is an opinion about code. A regression test is an execution against a running system. Teams with both catch different things.

Add security checks to the regression set

Look again at what the example asserts. Organization A's admin must not be able to read organization B's billing record. That is a security test, and it is also the most durable regression test on this page.

The durability comes from what it encodes. A test that asserts a screen renders breaks the moment the screen is redesigned. A test that asserts a 403 encodes a rule about who may read what, and that rule outlives the interface, the framework, and the person who wrote it.

So the fixture has to carry roles. Seed at least two tenants and one account per role, hold their credentials as secrets, and make the cross-tenant read a standing test rather than a check after an incident. Assertions in this set read backwards: the test passes when the request is refused. More classes are in the guide to API security testing.

Keep every replay inspectable and repeatable

A regression test is only useful if a failure means something. Same input, same verdict is a property you build, not one you get.

The split is the first half. Use the model where you want new ideas, at authoring and triage. Save the approved scenario as code, and replay that code. A run that re-invokes a model has a verdict that can move without your code moving, and it bills you per execution.

The environment is the second half. Fix the clock so date logic does not drift. Seed the data rather than reusing what the last run left behind. Fix the roles and their credentials. Pin browser and runtime versions. Stub external dependencies you do not control, because a third-party outage should not read as your regression. Assertion style matters too: Playwright's web-first assertions retry until a condition holds instead of checking a snapshot of the wrong moment, covered in Playwright assertions.

Now the honest half. Taking the model out of replay removes model variance. It does not remove flakiness. Google reported that about 1.5 percent of all test runs across its corpus returned a flaky result, that almost 16 percent of its tests had some level of flakiness, and that about 84 percent of the pass-to-fail transitions it observed involved a flaky test. Those are Google fleet figures from 2016, not a benchmark for your suite. The causes are ordinary engineering problems. Luo, Hariri, Eloussi, and Marinov studied 201 commits that likely fixed flaky tests across 51 open-source projects: the top three categories were asynchronous waiting, concurrency, and test-order dependency, and 78 percent of those tests were flaky from the moment they were written. Deterministic replay is necessary, not sufficient.

Limits and failure modes

Plan for each of these.

• Missed tests after selection. The model skips the test that would have caught it. Compare selected runs against the scheduled full run, and treat every escape as a selection bug.

• Wrong self-heals. A repair matches a different element and the test keeps passing against the wrong thing.

• Assertions that accept the wrong behavior. The worst outcome here: a green suite with no coverage.

• Generated tests that mirror the implementation. A model that reads the code and writes a test from it asserts what the code does, not what it should do. That test passes on the bug.

• Flaky environments. Google's own illustration: a project of about 1,000 tests, at a 1.5 percent flaky rate, produces roughly 15 failures per run that need investigation. Retrying a broken 15-minute test three times delays the reliable signal to 45 minutes.

• Dismissing failures as flakes. In the Luo study, 24 percent of flaky-test fixes changed the code under test, and 94 percent of those fixed a real bug in it.

• Selection blind spots. Coverage maps do not see configuration, feature flags, data migrations, or infrastructure changes.

• Test data leaking between runs. One test's leftover record makes the next run pass or fail for reasons unrelated to the code.

• Model cost per authoring pass. Ask what a rerun costs. If replay calls a model, the answer is not zero, and it scales with how often you deploy.

Visual checks deserve their own caution. A model deciding whether a layout change is a regression or an intended update is making a judgment call, and it will get some wrong in both directions. The tradeoffs are in automating visual regression testing.

An adoption checklist

Do not start by pointing a tool at the whole suite. Start where the pain is measurable.

• Pick one flow. The one whose tests you fix most often. Usually a checkout, an onboarding path, or a permissions-heavy screen.

• Track selection recall. Of the failures the scheduled full run found, how many would the selected set have caught? That number tells you whether selection is safe to trust.

• Track escaped regressions. Bugs that reached production in an area the suite claimed to cover.

• Track the flaky rate. Before and after. If it goes up, the tool is adding noise rather than coverage.

• Track repair approvals and rollbacks. A healing feature with a high rollback rate is guessing.

• Track pull request feedback time. Minutes from push to verdict, the number developers actually feel.

• Track replay cost. Per run, at your real deploy frequency.

Run that on one flow for a few weeks before expanding. The suite design underneath it is in the regression test suite guide.

How Qodex runs regression on every change

Qodex Autopilot starts from a failure that already happened. It reads a production error against the repository map it holds, writes a scenario that reproduces it as standard Playwright or HTTP code you keep, proves the reproduction against staging, then fails any pull request that would bring the bug back. Replay is plain code execution with no model in the loop, so the hundredth rerun costs what the first one did, and a failure arrives with the request, the response, or the screenshot attached.

See how Qodex Autopilot works.

Frequently Asked Questions

What is AI regression testing?

Regression testing where an AI helps create, select, maintain, run, or analyze tests after a change. The model reads the diff and drafts cases, ranks which tests to run, proposes repairs for stale steps, and labels failures. The tests themselves run as ordinary code.

How is it different from automated regression testing?

An automated suite runs fixed scripts that someone wrote and someone maintains. AI adds three things on top: authoring drafted from the code change, selection by impact and risk instead of running everything, and repair suggestions when the interface moves. The execution layer is the same.

Can AI generate regression tests from a pull request diff?

Yes. It reads the change and the dependency path around it, works out which behaviors are affected, and drafts the test. Two rules make that useful: a person approves it before it can gate a merge, and it must fail on the branch with the bug first.

What is a self-healing test, and can it hide bugs?

A test that repairs its own stale locator or step when the interface changes. It can hide bugs two ways: by matching a different but similar element, so the test passes against the wrong thing, and by keeping a weak assertion alive. Repairs need review and rollback.

What is test impact analysis, and how does it differ from risk-based selection?

Test impact analysis maps a code change to the tests that exercise the changed code, usually from coverage data. Risk-based selection scores tests by likely value, using change impact, past failures, and user paths, then orders them for a time budget.

Should AI regression tests run on every pull request?

Yes, in two parts. A selected fast set runs on every pull request against a preview of that branch, with an exit code that blocks the merge. The full suite runs on a schedule, and whenever selection confidence is low.

Can AI replace a full regression suite or manual testing?

No. It reduces the authoring and selection work, and it shortens triage. It does not replace the full suite, which is still the check on selection, and it does not replace exploratory testing, where a person hunts for problems nobody wrote a test for.

How do you keep AI regression tests deterministic?

Use the model at authoring and triage only. Save the approved scenario as code and replay that code with no model call. Fix the clock, seed the data, fix the roles and credentials, pin the environment, and stub external dependencies. Model variance goes; timing flakiness does not.