Skip to main content

​
Inverted semantics

Security tests use inverted semantics: pass means the app blocked the attack, and fail means the app may be vulnerable. This is one of the most important rules in Qodex security testing. It prevents the agent from turning a real vulnerability into a green test by weakening the assertion.

​
The rule

For a security scenario:
  • The expected result is what a secure app should do.
  • A blocked attack is a passing test.
  • A successful attack is a failing test.
  • A failing security scenario should stay failing until the product is fixed.
  • The agent should not change the expected result just to make the scenario pass. When an expectation really is wrong for your rules, Qodex corrects it openly and says why in the chat.
For example, if User B tries to read User A’s order, the expected response is usually 403 or 404. If the app returns 200, that is a vulnerability signal, not a reason to change the expected status.

​
Why this matters

Many agents are optimized to make failing tests pass. That is useful for normal product bugs, but dangerous for security testing. If a BOLA scenario expects 403 and receives 200, the wrong “fix” is to change the expectation to 200. The test becomes green, but the app still leaks data. Qodex treats that as a rule violation. The failing scenario is the evidence that the security issue still exists.

​
Common mistakes Qodex avoids

​
What happens on save

When Qodex saves a security scenario, it auto-verifies the scenario against the selected environment:
  1. If the app blocks the attack, the scenario verifies as pass and becomes regression coverage.
  2. If the check fails, Qodex treats that as evidence to classify, not as a finding yet. It confirms the legitimate request works, reproduces the attack, and reads your rules.
  3. If the attack really succeeded and a rule, endpoint note, or your own words say it should not have, Qodex keeps the scenario in its failing state and opens a finding with the request and response evidence.
  4. If the failure came from broken setup (for example an expired session) or an expectation your rules contradict, no finding is filed and Qodex says so.
This gives you both a vulnerability report and a saved test that can prove the fix later.

​
How it reads on a test run

When a saved security scenario fails in a later run, the test run page labels it Vulnerability present. If the matching finding was already open before the run, it reads Still present instead, so a known issue is not mistaken for a new regression.

​
Examples

​
BOLA on GET /api/orders/

User B requests User A’s order ID.
If the app returns 403, the security control works. If it returns 200 with User A’s order, Qodex opens a critical finding.

​
SQL injection on POST /login

The scenario sends a tautology payload in the email field.
If the app returns a session token, the scenario should fail. Qodex should not remove the payload or change the expected status.

​
Mass assignment on PATCH /api/users/me

The update request itself may return 200, because the user is allowed to update their name. The important assertion is the follow-up check that role did not change to admin.
If the role becomes admin, the assertion fails and Qodex opens a finding. The expected value should remain user.

Security scenarios

See how security scenarios are authored and verified.

OWASP API Top 10 in Qodex

Learn which attack classes Qodex can test.

Findings

Review evidence, severity, and lifecycle states.

Auto-verification on save

Understand the verification step that runs when a scenario is saved.

​
On the roadmap

Qodex plans to add a dedicated security reviewer pass on scenario save. The reviewer will check whether the assertion matches the attack goal and flag scenarios that accidentally weaken the inverted-semantics rule.