AI Pull Request Review: How Qodex Checks a Change
When Qodex reviews a pull request, up to 13 static scanners run first, on the files and lines you changed. A model then reads the diff and the code around it, and treats each scanner hit as a claim to check. What survives a second, skeptical pass posts as inline comments. A finding can fail the merge check only after a live request to your preview deployment proves it.
This page describes the Qodex AI pull request review as its source ran on 25 October 2026. To try it, connect a GitHub repository and Qodex reviews your newest open pull request from the last seven days. The AI code review page has the overview.
The AI pull request review pipeline, step by step
Here is the order of work for one pull request on the hosted Qodex review:
- Trigger. A pull request opens, gets new commits, reopens, changes its base branch or leaves draft. A comment of
@qodex-ai reviewor the Run review button also starts one. - Scanners. Qodex takes a shallow clone of the head commit and runs the scanners that fit the changed files.
- Filter. Hits outside the changed lines, and hits in fixture, mock and snapshot folders, are dropped.
- Model review. A model reads the diff, opens files with its tools and checks each remaining scanner hit against the code.
- Security checks. Up to four checks from a catalog of 48 are matched to the diff and added to the model's instructions.
- Second passes. A skeptical pass can only remove findings. If the first pass found nothing on a real change, an escalation pass looks harder.
- Live probe. Where a preview deployment exists, a finding can be checked with a read-only request.
- Output. Inline comments, one-click fixes, a summary and a check run post on the pull request.
Step 1: what starts a review
Drafts are skipped until they are marked ready for review. After the first review, a push is reviewed incrementally: Qodex reads the commits added since the last reviewed one. Automatic reviews can be turned off per repository, leaving the comment and the button as the only triggers.
Step 2: deterministic scanners run first
A scanner here is a program that checks code without a model, such as gitleaks for secrets. Scanners are fast and repeatable, and they give the model concrete places to look. Each one runs when the change holds a file it reads, so a TypeScript change does not start Bandit.
| Scanner | What it checks |
|---|---|
| semgrep | Security bugs and secrets, on any changed file |
| gitleaks | Secrets such as API keys and tokens, on any changed file |
| osv-scanner | Dependencies with a published advisory, when a lockfile or manifest changed |
| bandit | Python security problems, on .py files |
| brakeman | Rails security problems, on Ruby files or a Rails app |
| golangci-lint (gosec) | Go lint and security problems, on .go files |
| sqllint (Qodex) | Postgres and Supabase migration mistakes, on .sql files |
| hadolint | Dockerfile mistakes |
| actionlint | GitHub Actions workflow mistakes |
| shellcheck | Shell script bugs |
| ruff | Python lint |
| rubocop | Ruby lint |
| oxlint | JavaScript and TypeScript lint |
Semgrep runs three rule packs: p/default, p/security-audit and p/secrets.
The scanners are installed in the review container at fixed versions, so the scanner programs do not change between two reviews. On 25 October those were semgrep 1.94.0, gitleaks 8.21.2, osv-scanner 1.9.2, actionlint 1.7.7, hadolint 2.12.0 and shellcheck 0.10.0. Semgrep's rule packs come from its registry, so they can change between scans.
The SQL checker flags three migration mistakes: a new function left executable by the public role, a SECURITY DEFINER function without a fixed search_path, and a COMMENT ON FUNCTION without its argument list.
Step 3: keep only what the change touched
Before any model sees the results, code trims them. A hit on a line the pull request did not change is dropped, because it is not this author's problem. A hit inside a fixtures, mocks, snapshots, stubs or testdata folder is dropped, because those files are throwaway data shaped like real code. Test files themselves are kept. When two scanners flag the same lines for the same kind of problem, the higher-severity hit is kept. The rest are sorted by severity and capped at 50.
Step 4: the model checks each scanner hit against the code
The model gets the remaining hits in a block titled "Static analysis (precomputed)", with this instruction from the review prompt: the hits are "CANDIDATES rather than facts. Before surfacing any of them, verify against the diff or by reading the file." When it agrees, the finding cites the rule, for example semgrep:<rule_id>. When it disagrees, the hit is dropped and not repeated in the summary.
The model is not limited to the diff. It has tools to read any file in the clone, find a definition and search the code. It reviews correctness, security, performance, maintainability, style and conventions. A team can add its own review rules, but the prompt states that an owner instruction can never "skip a security or correctness check". A pull request description that calls a change intentional can excuse a behaviour change, but the prompt says "Stated intent never excuses a security issue".
Step 5: security checks matched to the diff
Qodex keeps a catalog of 48 written checks, 22 of them marked as security checks. Each check names the files and code patterns that trigger it. For each review, up to four checks whose triggers match the diff are added to the model's instructions, the most specific first. Three of the security checks:
- sql-string-concatenation: SQL built by joining strings or interpolating unsanitized input.
- ssrf-server-side-fetch: a server-side HTTP request whose host or URL comes from user input.
- auth-missing-on-state-change-route: a POST, PUT, PATCH or DELETE route registered with no auth middleware in sight.
When a non-trivial change comes back with zero findings, Qodex runs an escalation pass. It lowers the model's confidence bar from 0.7 to 0.6 and tells it to hunt specific classes. Two of them are permissions that are open by default, and an auth check present at one layer but missing at the layer that enforces it.
Step 6: a second pass that can only remove findings
After the review, a separate model call re-reads the findings with one tool that can only keep or drop them. It cannot add a finding or change one. This is a precision filter: the first pass reads broadly, the second reads skeptically.
Dropping a scanner hit silently is the right default for style, and the wrong one for security. So a high or critical hit from a security scanner that the model did not raise gets a second chance. It goes back in as a candidate and through the same skeptical pass. If that pass cannot run at all, those hits are kept rather than lost. Style and lint hits are not forced through; they stay advisory and are counted in the summary.
Step 7: a live probe against your preview
A finding can carry a probe: a request and the response the model expects if the bug is real. On GitHub, Qodex looks up the latest successful deployment for the pull request's head commit and sends the probe there. The probe is a GET request only, to the same host, with the model's headers dropped. An optional host allowlist in .qodex.yaml limits where probes go.
When the response matches, the finding is marked verified and the comment carries the request and response as proof. When it does not match, the finding still posts, marked unverified, so the author decides.
What you see on the pull request
Each finding posts as an inline comment on the line it cites. It carries a severity (critical, major, minor, nitpick or info) and a category (bug, security, performance, maintainability or style). Low-severity style and convention findings are collected into one collapsed group in the summary instead of separate threads.
When a fix fits in five lines or fewer at the cited line, the comment includes it as a suggestion. GitHub shows it with a Commit suggestion button, so the author applies it in one click.
A Qodex check run appears in the checks panel while the review runs. It stays advisory until a repository turns blocking on in .qodex.yaml. With blocking on, the check fails only when a verified finding sits at or above your chosen severity. An unverified finding never blocks a merge. Blocking is part of the paid plan and becomes a real gate when your branch protection requires the check. These are the keys in .qodex.yaml, as named in the configuration code. The block below is a configuration excerpt, not a runnable example.
pr_review:
# lowest severity to comment on
severity_threshold: minor
# default false (advisory)
block_pr_merge: true
# only verified findings at or
# above this severity fail it
block_on_severity: critical
paths:
exclude:
- "vendor/**"
If the review itself does not finish, for example because it ran out of budget, the check fails and says why instead of reporting a clean pass. The next push retries the review.
How dismissed findings stop coming back
On later pushes to the same pull request, Qodex drops a finding someone dismissed, marked as a false positive or won't fix, or whose thread a person resolved. It matches by a fingerprint of the title, file and anchored code, with a fallback for the same bug reworded nearby. The @qodex-ai ignore and @qodex-ai resolve comments set the same states from the thread. A full review on request starts fresh and can raise them again.
On later pull requests, patterns your team dismissed are weighed down, not hidden. The reviewer still raises one when the evidence is strong or the severity is critical.
Why every scanner hit is checked before it reaches you
A scanner matches patterns. It knows what a credential looks like and what string-built SQL looks like. It does not know whether this credential works or whether this SQL ever sees user input. Two examples show the gap.
A secret pattern in a test. Say a test file holds const token = "glpat-secret-1234" to check that a parser rejects bad tokens. Gitleaks flags it as a GitLab token, and a review bot that posts every scanner hit posts it. Qodex hands it to the model as a candidate. The model reads the test, sees a made-up value in an assertion, and drops it.
An injection pattern in a seed script. Say a migration helper builds an INSERT by formatting a list of role names into a string. Semgrep flags string-built SQL. The model reads the file, sees that every value is a constant in the same file and no request reaches it, and drops it. The same rule on a request handler that formats req.query.orgId into SQL is kept.
In both cases the hit was cheap to produce and cheap to settle by reading the code. Raw scanner output leaves that reading to a person on each pull request. A bot that posts these hits unchecked can train people to skip its comments, and a real finding gets skipped with them.
Beyond the diff: the whole-repository code scan
A pull request review only sees the change. The Code scan page runs the same scanners over a whole linked repository, plus a model that reads the code in bundles looking for security problems. A scan starts when someone presses the button. A second scan of the same repository can start 10 minutes after the last one.
Dependency advisories from osv-scanner, and high-severity hadolint and actionlint hits, are filed with no model call, because a pinned vulnerable version is a fact. Every other hit is checked in code first. The file must be one the scan listed, and the quoted line must sit within two lines of the line the hit names. A hit that passes gets a confirm call that asks what input reaches the line, what guards it, and what an attacker gets. Only a "supports" answer files a finding, rated critical, high, medium or low.
Secret hits get their own question: a working credential, or a fixture, sample or key meant to be public. The model sees the value masked, and file text in a code scan passes through secret redaction before it reaches a model. A finding found again on a later scan is matched to the one already filed, and one you closed as a false positive or won't fix is not filed again.
OpenQodex: the same scanners and a stricter review before you push
OpenQodex is the open-source review engine from the same team, published on npm as openqodex under the Apache 2.0 license. It reviews your change on your own machine before you push: the commits not yet pushed plus everything uncommitted. init installs a git pre-push hook by default.
It runs 22 built-in scanners, including the infrastructure checks the hosted review lacks: trivy, Checkov, TFLint, kube-linter and kubeconform for Terraform and Kubernetes files. Each scanner download from GitHub releases is checked against a sha256 pinned in the package, and the Python and Ruby tools install from lock files of hashes.
The review itself is stricter than the hosted one:
- A separate reviewer. OpenQodex starts a fresh reviewer process from the coding agent you are signed in to, with no memory of your session. It reads a frozen, secret-redacted copy of the change and cannot edit files.
- Finders, then verifiers. Two finder sessions read each part of the change. A verifier then confirms or refutes each candidate. A scanner candidate must be confirmed or refuted, never left unresolved.
- Quotes are checked. Every quoted evidence line must match the snapshot within two lines of the cited line. A quote that matches nothing is sent back for correction or dropped.
- Coverage is measured. A changed range counts as reviewed when it was given to the reviewer or its recorded reads covered it, not when the reviewer says it read it.
- An honest "incomplete". An unsettled scanner candidate, an unreviewed changed range or a snapshot that changed mid-review each give the result "Review incomplete". It exits with code 2 and lists what is missing.
OpenQodex itself collects nothing and sends no telemetry. The reviewer sends the brief and the files it reads to the model your own coding agent login uses. Beyond scanner downloads, four scanner steps use the network: semgrep rule packs, osv.dev for dependency names and versions, kubeconform schemas and the RustSec database. --offline skips those four.
The hosted Qodex review can run on this engine, but that setting is off by default. Unless your project has been moved to it, your pull requests get the hosted pipeline described above.
What Qodex review does not do yet
- The hosted review has no container image scanning, no Terraform or Kubernetes checker, and no licence-risk or malicious-package feed.
- brakeman and golangci-lint run once at the root of the analysed folder, so a Rails app or Go module nested deeper in a monorepo is missed by them.
- There is no Java grammar, so finding the callers of a Java function falls back to a plain text search.
- A semgrep run killed by its 60-second timeout is recorded as a clean pass. This is an open bug, not yet fixed.
- Live probes run on GitHub only, and only when a preview deployment exists. Without one, findings post unverified and the check cannot block.
- The code scan runs when you press the button; there is no schedule yet, and SARIF export is planned.
- Running your saved API and UI tests on a pull request is in development. Today the live probe is the only part of the review that runs against your app.
Conclusion
Qodex review puts deterministic scanners first, then has a model check each hit against the code before it posts. High-severity security hits get a second look, and are kept if that look cannot run. A second pass can only remove findings, and a finding can close the merge gate only after a live request proved it. OpenQodex brings the same scanners and a stricter, measured review to your laptop. For the wider picture of static analysis and AI reviewers, read automated code review.
Frequently Asked Questions
Does Qodex post every scanner hit?
No. Hits outside the changed lines and in fixture folders are dropped by code. The model reads the code behind each remaining hit and drops the ones that do not hold. High or critical security hits it skipped get a second, skeptical look.
Can Qodex block a merge?
Yes, on the paid plan, once a repository sets block_pr_merge: true in .qodex.yaml and branch protection requires the check. It fails only on a verified finding at or above block_on_severity, which defaults to critical.
What happens when my repository has no preview deployment?
The review runs as usual, and findings post marked unverified. No finding can fail the check, because only a finding a live probe verified can block. A review that does not finish still fails the check and says why.
Is the hosted review the same as OpenQodex?
Not by default. They share scanners and ideas, but hosted pull requests run the pipeline on this page unless a project is switched to the OpenQodex engine.
