Open source · MIT · v1.3.0
Requirements in. Verdicts out. Every test traced.
api-vv-toolkit reads an OpenAPI 3.0 or 3.1 contract and your written requirements, generates API tests (boundary, negative, auth and seeded property cases), runs them against a live service and gates CI on which requirements are actually verified.
Live sample run
Built by GitHub Actions run 42851062290 from commit fc94f8b on 2026-11-10 16:05 UTC, against the bundled demo API on a loopback port.
Gate: FAIL (REQ-003, REQ-008)
12 requirements, each linked to its matrix row
- requirements verified
- 10
- requirements failed
- 2
- tests passed
- 30
- tests failed
- 2
What the gate caught
The bundled demo API carries two deliberately seeded defects. The gate named both:
What CI ran
Validated 12 requirementsGenerated 32 cases (32 enabled)LLM proposals: 1 accepted, 1 rejectedExecuted: 30 passed, 2 failed, 0 skippedGate: FAIL (REQ-003, REQ-008)Wrote Markdown, HTML, CSV and JUnit reports to runs/demoGate: FAIL (REQ-003, REQ-008)Expected defects confirmed: REQ-003, REQ-008Gate your own API
- uses: muhib-karim/api-vv-toolkit@v1
with:
spec: openapi.yaml
requirements: requirements.yaml
base-url: http://localhost:8000Or locally: install the wheel from a
GitHub release, then run api-vv generate,
run and report.
Contract drift, classified
api-vv diff old.yaml new.yaml exits 1 when a change breaks existing clients.
Sample: the bundled breaking fixture (14 breaking,
2 non-breaking).
| Severity | Rule | Operation | Detail |
|---|---|---|---|
| breaking | operation-removed |
DELETE /orders/{orderId} |
operation removed |
| breaking | path-removed |
GET /legacy |
path removed |
| breaking | required-parameter-added |
GET /orders |
parameter query.tenant: required parameter added |
| breaking | response-type-changed |
GET /orders |
response 200 application/json[].id: type ['integer'] -> ['string'] |
| breaking | response-schema-narrowed |
GET /orders |
response 200 application/json[].status: property no longer guaranteed |
| breaking | response-schema-narrowed |
GET /orders |
response 200 application/json[].total: property removed |
| breaking | security-added |
GET /orders/{orderId} |
operation now requires authentication |
| breaking | response-type-changed |
GET /orders/{orderId} |
response 200 application/json.id: type ['integer'] -> ['string'] |
| breaking | response-schema-narrowed |
GET /orders/{orderId} |
response 200 application/json.status: property no longer guaranteed |
| breaking | response-schema-narrowed |
GET /orders/{orderId} |
response 200 application/json.total: property removed |
| breaking | response-status-removed |
POST /orders |
documented 201 response removed |
| breaking | request-property-required |
POST /orders |
request body.currency: required property added |
| breaking | enum-value-removed |
POST /orders |
request body.priority: enum values removed: ["low"] |
| breaking | request-constraint-tightened |
POST /orders |
request body.qty: maximum tightened |
| non-breaking | response-status-added |
GET /orders/{orderId} |
documented 401 response added |
| non-breaking | response-status-added |
POST /orders |
documented 200 response added |