Skip to main content

​
Run tests in Buildkite

Use a Buildkite step to trigger Qodex, poll the run result, and fail the build if Qodex reports a failed run.

​
Before you start

You need:
  • A Qodex project with runnable scenarios.
  • A project API key from Settings > Platform > API keys.
  • A Buildkite pipeline connected to your repository.

​
Store the secrets

Expose these values to the Buildkite job through Buildkite secrets, an agent environment hook, Vault, AWS Secrets Manager, or another secret manager:
  • QODEX_API_KEY: your qk_... key.
  • QODEX_PROJECT: your Qodex project slug.
Treat the API key like any other deploy credential.

​
Add the step

Save this file at .buildkite/pipeline.yml. 
steps:
  - label: ":qodex: smoke"
    key: qodex-smoke
    timeout_in_minutes: 30
    plugins:
      - docker#v5.11.0:
          image: alpine:3.20
          environment:
            - QODEX_API_KEY
            - QODEX_PROJECT
          command:
            - sh
            - -lc
            - |
              apk add --no-cache curl jq

              ENV_NAME=staging
              if [ "$$BUILDKITE_BRANCH" = "main" ]; then ENV_NAME=prod; fi

              RESPONSE=$$(curl -fsS -X POST \
                "https://agents.qodex.ai/api/projects/$$QODEX_PROJECT/webhooks/trigger" \
                -H "Authorization: Bearer $$QODEX_API_KEY" \
                -H "Content-Type: application/json" \
                -d "{\"environment\":\"$$ENV_NAME\",\"tags\":[\"smoke\"]}")

              RUN_ID=$$(echo "$$RESPONSE" | jq -r .testRunId)
              if [ -z "$$RUN_ID" ] || [ "$$RUN_ID" = "null" ]; then
                echo "Failed to start run. Response: $$RESPONSE"
                exit 1
              fi

              DEADLINE=$$(( $$(date +%s) + 1500 ))
              while [ "$$(date +%s)" -lt "$$DEADLINE" ]; do
                STATUS=$$(curl -fsS \
                  "https://agents.qodex.ai/api/projects/$$QODEX_PROJECT/test-runs/$$RUN_ID" \
                  -H "Authorization: Bearer $$QODEX_API_KEY" | jq -r .status)
                echo "status=$$STATUS"
                case "$$STATUS" in
                  completed) exit 0 ;;
                  failed) exit 1 ;;
                  *) sleep 10 ;;
                esac
              done
              echo "Timed out waiting for Qodex run $$RUN_ID"
              exit 1
Buildkite uses $$ so variables are expanded inside the job rather than at pipeline upload time.

​
Gate deploys

Add depends_on: qodex-smoke to any deploy step that should wait for Qodex. 
  - label: ":rocket: deploy"
    depends_on: qodex-smoke
    command: ./scripts/deploy.sh
For post-deploy validation, add a second Qodex step after deploy and point it at the production environment.

Jenkins recipe

Generic shell recipe