13. Verification and testing
How Vord proves it works and stays secure. The strategy decision is ADR-0028; this section is the living plan — the tiers, the tooling, and the map from each threat to the test that defends it. The spine is deterministic simulation (madsim), and security testing is woven through every tier, not a phase at the end. The tiers are diagrams/test-tiers.mermaid.
13.1 The three tiers
| Tier | What it proves | Tooling | From |
|---|---|---|---|
| 0 — DST properties & abuse-cases | correctness + security properties (edge-independence, no cross-tenant read, key rotation, read-your-writes); canonical attacks asserted to fail | madsim, seed-reproducible, no containers — laptop + CI | M0 |
| 1 — integration & adversarial | the real backends behave; protocol conformance; behaviour under load and fault | containers (KMS, SpiceDB/OpenFGA, Wyrd metadata, NATS); conformance suites; load/fault; Jepsen-style consistency | M1+ |
| 2 — community & ecosystem validation | independent scrutiny without hiring | vulnerability-disclosure program; OSS-Fuzz, CodeQL, OpenSSF Scorecard/Badge; the free OpenID conformance suite; inherited upstream-library audits | continuous + pre-release |
A DST seed that finds a violation, and every fixed vulnerability, becomes a permanent regression test (ADR-0028 §4, ADR-0010).
One honesty note on the Tier-0 read-your-writes properties (V6/V9): Tier 0
explores them against testkit's simulation-faithful fakes — an Authorizer
with real zookie semantics and an in-sim event/push transport
(ADR-0028 §1) — because the real engines cannot
run inside madsim; Tier 1 validates the same properties against the real engines.
The fakes make the properties explorable; the containers make them true of the
shipped binaries.
13.2 The security-testing map
Each threat class from the attack catalog (§12.5) traced to the test that proves it defended, the tier, and the milestone it attaches at.
| Threat class | Test that defends it | Tier | At |
|---|---|---|---|
Token forgery / alg:none / alg-confusion |
abuse-case property test (forged token rejected) | 0 | M0 |
| Verification under core outage / cold-start | edge-independence property (V1) — correct verification inside the projection freshness window and fail-closed past it; cold-edge-fails-closed | 0 | M0 |
| Cross-tenant token / isolation breach | no-cross-tenant-hot-path-read property (V7); wrong-tenant-token rejected | 0 / 1 | M4 |
| Privilege escalation / New-Enemy | composite-token read-your-writes property (V9) | 0 | M3 |
| Key compromise / rotation | rotation-without-orphaned-token property (V5); custody integration | 0 / 1 | M1 |
| OAuth code injection / mix-up / CSRF / redirect | OIDC conformance suite (V8) + abuse-case tests | 1 | M4 |
| SAML signature-wrapping (XSW) / XXE | fuzzing the XML parser + abuse-case (wrapped signature rejected) | 1 | M6 |
| Token theft / replay; revocation evasion | revocation-propagation lag test (V2); DPoP binding test (when built) | 1 | M2 |
| Credential stuffing / brute force; write-path DoS | load/fault tests under login-storm (V3/V4); rate-limit tests | 1 | M4 |
| SSRF (JWKS / SCIM / upstream fetch) | egress-allowlist integration test | 1 | M5–M7 |
| Supply-chain compromise | cargo-deny advisories + Dependabot in CI (ADR-0008) |
0 | M0 |
| Logic bugs across the whole surface | the free OpenID conformance suite + CodeQL + OSS-Fuzz + the community disclosure program (a funded audit only if sponsored) | 1–2 | continuous |
The per-milestone security review additionally checks the new surface against the OWASP ASVS v5.0 requirements (ADR-0027); a failing requirement becomes an abuse-case test. Once code begins, the threat model itself is maintained as code with OWASP Threat Dragon, its checks run in CI.
13.3 CI gates
- Every PR (from M0):
fmt·clippy -D warnings·build·test(the DST property and abuse-case tests run here, on madsim) ·cargo xtask conformance(the token-format vectors) ·cargo-deny(RUSTSEC + permissive-license wall) · CodeQL code scanning + secret scanning · DCO. No containers — Tier 0 is deterministic and laptop-runnable. Every gate here is free for a public repository. - Nightly / pre-merge (from M1+): container integration against real backends, the conformance suites, parser fuzzing, and the load/fault scenarios.
- Continuous (free OSS tooling, outside the per-PR gate): OSS-Fuzz, OpenSSF Scorecard, and Dependabot.
- Around release points: a focused maintainer / community security review plus the full automated suite and the free OpenID conformance run — not a commissioned audit (ADR-0027, ADR-0025). A sponsored third-party audit is welcome if funded, never a gate.
13.4 What is deliberately not covered, and where
- Performance / throughput is not a DST result — DST abstracts time. The write-path scale claims (V3/V4) and revocation lag (V2) are Tier-1 measurements on real hardware, not Tier-0 properties.
- Conformance proves protocol-correctness, not the absence of logic bugs — hence the abuse-case tests and the continuous fuzzing.
- No commissioned audit is assumed — Vord is an open-source project. A coordinated disclosure program plus continuous free tooling (OSS-Fuzz, CodeQL, Scorecard) is the substitute, and being continuous it does not go stale the way a point-in-time pentest would (ADR-0027).
- Fuzzing and container integration attach when the code and the real backends exist (M1+); their commitment is recorded now so they are scheduled, not discovered. A pentest / commissioned audit is welcome if funded, never assumed and never a gate (ADR-0027).