api-vv-toolkit v1.3.0

Open source · MIT · v1.3.0

Requirements in. Verdicts out. Every test traced.

api-vv-toolkit reads an OpenAPI 3.0 or 3.1 contract and your written requirements, generates API tests (boundary, negative, auth and seeded property cases), runs them against a live service and gates CI on which requirements are actually verified.

Open the traceability matrix Source and quickstart

Live sample run

Built by GitHub Actions run 42851062290 from commit fc94f8b on 2026-11-10 16:05 UTC, against the bundled demo API on a loopback port.

Gate: FAIL (REQ-003, REQ-008)

12 requirements, each linked to its matrix row

requirements verified
10
requirements failed
2
tests passed
30
tests failed
2

What the gate caught

The bundled demo API carries two deliberately seeded defects. The gate named both:

  • REQ-003failed

    A widget name longer than eight characters is rejected.

    Whystatus: expected 422, got 201

  • REQ-008failed

    Deleting a widget returns 204 with no body.

    Whystatus: expected 204, got 200; response status is undocumented

What CI ran

cli-output.txtRaw file
Validated 12 requirementsGenerated 32 cases (32 enabled)LLM proposals: 1 accepted, 1 rejectedExecuted: 30 passed, 2 failed, 0 skippedGate: FAIL (REQ-003, REQ-008)Wrote Markdown, HTML, CSV and JUnit reports to runs/demoGate: FAIL (REQ-003, REQ-008)Expected defects confirmed: REQ-003, REQ-008

Gate your own API

GitHub Actions step
- uses: muhib-karim/api-vv-toolkit@v1
  with:
    spec: openapi.yaml
    requirements: requirements.yaml
    base-url: http://localhost:8000

Or locally: install the wheel from a GitHub release, then run api-vv generate, run and report.

Contract drift, classified

api-vv diff old.yaml new.yaml exits 1 when a change breaks existing clients. Sample: the bundled breaking fixture (14 breaking, 2 non-breaking).

SeverityRule OperationDetail
breaking operation-removed DELETE /orders/{orderId} operation removed
breaking path-removed GET /legacy path removed
breaking required-parameter-added GET /orders parameter query.tenant: required parameter added
breaking response-type-changed GET /orders response 200 application/json[].id: type ['integer'] -> ['string']
breaking response-schema-narrowed GET /orders response 200 application/json[].status: property no longer guaranteed
breaking response-schema-narrowed GET /orders response 200 application/json[].total: property removed
breaking security-added GET /orders/{orderId} operation now requires authentication
breaking response-type-changed GET /orders/{orderId} response 200 application/json.id: type ['integer'] -> ['string']
breaking response-schema-narrowed GET /orders/{orderId} response 200 application/json.status: property no longer guaranteed
breaking response-schema-narrowed GET /orders/{orderId} response 200 application/json.total: property removed
breaking response-status-removed POST /orders documented 201 response removed
breaking request-property-required POST /orders request body.currency: required property added
breaking enum-value-removed POST /orders request body.priority: enum values removed: ["low"]
breaking request-constraint-tightened POST /orders request body.qty: maximum tightened
non-breaking response-status-added GET /orders/{orderId} documented 401 response added
non-breaking response-status-added POST /orders documented 200 response added