Skip to main content

​
Scenarios

Scenarios are the reusable tests Qodex creates from your testing requests. A scenario describes what should be tested, how to test it, what result to expect, and whether it is ready to run automatically.

​
What a scenario contains

A scenario is stored as structured JSON. It includes:
  • A plain-English goal.
  • Steps such as navigate, click, fill, assert, or api_call.
  • Expected results and assertions.
  • Tags and priority (critical, high, medium, low).
  • The endpoints or pages it covers.
Every scenario produces an executable script. UI scenarios become Playwright Test code. API scenarios become Node.js HTTP tests. Both are parameterized with environment variables such as TARGET_URL, API_BASE_URL, and AUTH_TOKEN, so the same scenario can run against staging, production, or a preview deploy. Scenarios can also belong to test groups. A group can run related scenarios sequentially as one ordered flow or in parallel as a folder of independent checks. See Test groups and folders for the group rules, CSV behavior, and run results.

​
Advanced step behavior

Most scenarios should read like a direct user flow: do this, check that, then stop. When the product needs more control, Qodex can add two advanced behaviors:
  • Conditional steps. A step can include a when guard so it only runs when earlier data, current state, or an environment value makes that step relevant.
  • Delays for async work. A step can wait briefly before the next assertion when the app needs time to process a job, send an email, update a webhook, or finish another backend task.
Use these sparingly. A scenario is easier to trust when it waits for a visible signal, a response, or a status change instead of relying on fixed time.

​
Dynamic values

Use dynamic values when a scenario needs fresh data on every run, such as a unique email, idempotency key, timestamp, or display name. Qodex supports two forms:
  • Fresh every time: ${$timestamp}, ${$isoTimestamp}, ${$uuid}, ${$guid}, ${$randomInt}, ${$randomEmail}, and ${$randomString} generate a new value each time that token appears.
  • Stable for one run: ${$run.uuid}, ${$run.timestamp}, or any other ${$run.<generator>} form generates one value and reuses it across the same scenario run.
Use the run-stable form when a scenario creates something and checks it later. For example, create a user with qa+${$run.randomString}@example.com, then search for that same email in a later step.

​
Lifecycle

Scenarios start in draft lifecycle. A human reviews and promotes to active. Only active scenarios run on a schedule. The agent recommends; humans ship. 
agent generates  ->  draft  ->  human promotes  ->  active  ->  runs on schedule

​
Auto-verification on save

API scenarios run against the target environment when they are saved. Qodex attaches the verdict (pass, fail, or error) to the scenario immediately. The agent uses that verdict differently depending on the scenario type. A failing happy-path scenario usually needs to be rewritten. A failing security scenario may be the evidence of a real vulnerability.

​
Example shape

An abbreviated scenario, with details stripped: 
{
  "id": "scn_abc123",
  "project_id": "prj_xyz",
  "type": "api",
  "title": "Reject login with invalid password",
  "goal": "Verify the login endpoint returns 401 for a wrong password.",
  "tags": ["auth", "negative"],
  "priority": "high",
  "lifecycle": "draft",
  "covered_endpoints": ["POST /auth/login"],
  "steps": [
    { "type": "api_call", "method": "POST", "path": "/auth/login",
      "body": { "email": "${AUTH_EMAIL}", "password": "wrong" } }
  ],
  "assertions": [
    { "path": "status", "equals": 401 },
    { "path": "body.error", "exists": true }
  ],
  "script_id": "scr_def456",
  "last_run_status": "passed",
  "last_verification": { "status": "pass", "ran_at": "2026-06-07T10:12:00Z" }
}

​
When to use it

  • Use a scenario for behavior worth rechecking after a code change.
  • Add negative paths, validation errors, and auth checks, not just happy paths.
  • Use security scenarios when pass means the attack was blocked. See Findings.
  • Create coverage for every endpoint in an imported OpenAPI spec, including IDOR and mass-assignment cases.

​
When not to use it

  • One-off exploratory probing. Use the chat directly; let the agent decide whether to save anything.
  • Tests that depend on shared state across other scenarios. Each scenario should set up what it needs.
  • Hardcoded data that should be an ${ENV_VAR} substitution. The critic flags this on save.

​
On the roadmap

Planned: self-critique on scenario save. A second LLM call reviews every generated scenario against the goal before persistence, with a single revision retry. Weak scenarios are flagged for humans, never blocked. See backlog.md.
Planned: findings-aware generation via endpoint_brief and page_brief tools. The author and the critic both read prior findings, existing scenarios, and observed auth before writing a new scenario for a known endpoint.

Scripts

The executable form of a scenario.

Findings

What a failed scenario becomes when the failure is a real bug.

Memory

The context every authoring call reads.

API testing scenarios

How API scenarios are authored, chained, and verified.