Skip to main content

​
Inverted semantics

Security tests use inverted semantics: pass means the app blocked the attack, and fail means the app may be vulnerable. This is one of the most important rules in Qodex security testing. It prevents the agent from turning a real vulnerability into a green test by weakening the assertion.

​
The rule

For a security scenario:
  • The expected result is what a secure app should do.
  • A blocked attack is a passing test.
  • A successful attack is a failing test.
  • A failing security scenario should stay failing until the product is fixed.
  • The agent should not change the expected result just to make the scenario pass.
For example, if User B tries to read User A’s order, the expected response is usually 403 or 404. If the app returns 200, that is a vulnerability signal, not a reason to change the expected status.

​
Why this matters

Many agents are optimized to make failing tests pass. That is useful for normal product bugs, but dangerous for security testing. If a BOLA scenario expects 403 and receives 200, the wrong “fix” is to change the expectation to 200. The test becomes green, but the app still leaks data. Qodex treats that as a rule violation. The failing scenario is the evidence that the security issue still exists.

​
Common mistakes Qodex avoids

MistakeWhy it is wrong
Changing expectedStatus: 403 to 200Hides that the attack succeeded.
Removing the attack payloadTurns the security test into a happy-path test.
Changing tags from security or bola to smokeRemoves the scenario from security reporting.
Deleting the failing scenarioRemoves the regression check that should prove the fix later.
Treating a server 500 as successA crash is still a vulnerability signal.

​
What happens on save

When Qodex saves a security scenario, it auto-verifies the scenario against the selected environment:
  1. If the app blocks the attack, the scenario verifies as pass and becomes regression coverage.
  2. If the attack succeeds, the scenario verifies as fail.
  3. Qodex keeps the scenario in its failing state and opens a finding with the request and response evidence.
This gives you both a vulnerability report and a saved test that can prove the fix later.

​
Examples

​
BOLA on GET /api/orders/

User B requests User A’s order ID. 
{
  "name": "BOLA: userB cannot read userA order",
  "method": "GET",
  "target": "${API_BASE_URL}/api/orders/42",
  "auth": "${USER_B_TOKEN}",
  "expectedStatus": 403
}
If the app returns 403, the security control works. If it returns 200 with User A’s order, Qodex opens a critical finding.

​
SQL injection on POST /login

The scenario sends a tautology payload in the email field. 
{
  "name": "SQLi: login rejects tautology payload",
  "method": "POST",
  "target": "${API_BASE_URL}/auth/login",
  "body": {
    "email": "' OR '1'='1",
    "password": "x"
  },
  "expectedStatus": 400
}
If the app returns a session token, the scenario should fail. Qodex should not remove the payload or change the expected status.

​
Mass assignment on PATCH /api/users/me

The update request itself may return 200, because the user is allowed to update their name. The important assertion is the follow-up check that role did not change to admin. 
{
  "name": "Mass assignment: regular user cannot self-promote",
  "steps": [
    {
      "method": "PATCH",
      "target": "${API_BASE_URL}/api/users/me",
      "body": {
        "name": "Ada",
        "role": "admin"
      },
      "expectedStatus": 200
    },
    {
      "method": "GET",
      "target": "${API_BASE_URL}/api/users/me",
      "expectations": [
        {
          "kind": "jsonpath",
          "path": "$.role",
          "equals": "user"
        }
      ]
    }
  ]
}
If the role becomes admin, the assertion fails and Qodex opens a finding. The expected value should remain user.

Security scenarios

See how security scenarios are authored and verified.

OWASP API Top 10 in Qodex

Learn which attack classes Qodex can test.

Findings

Review evidence, severity, and lifecycle states.

Auto-verification on save

Understand the verification step that runs when a scenario is saved.

​
On the roadmap

Qodex plans to add a dedicated security reviewer pass on scenario save. The reviewer will check whether the assertion matches the attack goal and flag scenarios that accidentally weaken the inverted-semantics rule.