Vord

Architecture Decision Record

0028. Testing strategy

Context

Vord adopted deterministic simulation by reference (ADR-0002, citing Wyrd ADR-0009) and the M0 proposal sketches test tiers, but Vord has recorded no consolidated testing strategy of its own — and for a system whose correctness is security, the security testing in particular is scattered: the edge-independence DST property (ADR-0010), conformance (ADR-0025), and abuse-case tests / fuzzing / community disclosure (ADR-0027) each live apart. "How is security testing planned?" cannot be answered from one place. This ADR records the strategy and makes security testing a first-class spine, woven through every tier, not a phase bolted on at the end.

Wyrd ADR-0009 established a DST-first discipline for a correctness-critical system. Vord is correctness-critical in the security sense — a verification bug is a silent security failure, not a crash (ADR-0002) — so the same discipline applies, extended with the security-specific layers an identity provider needs (conformance, fuzzing, abuse-case property tests, external review).

Decision

  1. Deterministic simulation is the correctness-and-security spine (adopting Wyrd ADR-0009). Production logic is written against abstract time / clock / network plus the trait seams and run in a single-threaded, seed-reproducible simulator (madsim). The headline property is edge-independence (ADR-0010); the other security properties are also DST properties — no cross-tenant hot-path read (V7), key-rotation without orphaned tokens (V5), read-your-writes / the New-Enemy problem (V6/V9). A seed that finds a violation is committed as a permanent regression.

    For that claim to be honest, testkit must carry simulation-faithful fakes of the external services that cannot run inside madsim — the Zanzibar engine (SpiceDB/OpenFGA) and NATS are out-of-process Go services, so V6/V9's hard case (cross-domain interleavings the composite token must fence, ADR-0026) would otherwise never be deterministically explored. Required in-sim models: an Authorizer fake with real zookie semantics (its own version space, configurable replica lag, fenced reads honored) and an event/push transport model with reordering, delay, and loss. The Wyrd side runs as a fake core at M0 (a fault-injectable in-memory MetadataStore); embedding Wyrd's actual metadata-model logic in-sim is preferred as soon as its crates permit, since both projects share the madsim discipline. Tier 1 then validates the same properties against the real engines — the fakes make the properties explorable, the containers make them true of the shipped binaries.

  2. Three tiers (identity-tailored, mirroring Wyrd's harness):

    • Tier 0 — DST correctness & security properties (madsim, no containers, from M0): the property tests and the abuse-case tests asserted to fail (ADR-0027). The fast loop — runs on a laptop and in CI.
    • Tier 1 — integration & adversarial (containers, from M1+): real backends (KMS, SpiceDB/OpenFGA, the Wyrd metadata tier, NATS); the conformance suites (ADR-0025); load / fault tests (the write-storm scenarios V3/V4, revocation lag V2); and Jepsen-style adversarial consistency against a real cluster — the Vord analog of Wyrd's Jepsen run.
    • Tier 2 — community & ecosystem validation (continuous + pre-release): the independent scrutiny an open-source project can get without hiring — a coordinated vulnerability-disclosure program (SECURITY.md); free automated tooling (OSS-Fuzz, GitHub CodeQL + secret scanning + Dependabot, the OpenSSF Scorecard / Best Practices badge); the free OpenID conformance suite (ADR-0025); and assurance inherited from vetted upstream crypto libraries. A commissioned pentest / audit is a welcome bonus if funded, never a dependency.

  3. Security testing is woven through every tier, not a separate phase. Abuse-case property tests (Tier 0), conformance + adversarial integration (Tier 1), and community validation (Tier 2), plus continuous fuzzing of every parser that reads untrusted input: the JOSE/token codec, the SAML/XML parser, the SCIM JSON surface (ADR-0018), OIDC request objects and the authorize-request parser, and the admin gRPC/REST decoders (ADR-0021). Fuzzing runs via OSS-Fuzz — with the stated fallback of cargo-fuzz in scheduled CI if OSS-Fuzz admission is declined (admission gates on project criticality and is not guaranteed pre-1.0) — plus CodeQL code scanning (best-effort while its Rust support matures, not a load-bearing gate) and dependency-advisory scanning (cargo-deny + Dependabot, ADR-0008) — all free for an open-source project. The threat catalog (architecture §12.5) maps to concrete tests in architecture §13.

  4. Every bug and every vulnerability becomes a seed-reproducible regression test — the ADR-0010 rule generalized: a DST seed that finds a correctness or security violation, and every fixed vulnerability, lands a permanent test (ADR-0027).

  5. State what each tier does not cover. DST abstracts time, so it proves correctness and security properties, never throughput — performance and load-scaling are Tier 1, on real hardware (the write-path scenarios V3/V4). Conformance proves protocol-correctness, not the absence of logic bugs. A pentest is a point-in-time check, never a substitute for the standing tests.

  6. CI gates per tier. Tier 0 gates every PR (fmt / clippy / build / test including the DST property and abuse-case tests, the token-format conformance vectors, CodeQL + secret scanning, cargo-deny, and DCO) from M0; Tier 1 (container integration, the conformance suites, OSS-Fuzz fuzzing, load/fault) runs nightly / pre-merge from M1+; Tier 2 is continuous (Scorecard, Dependabot, the disclosure program) with a focused maintainer/community review around release points. Every gate named here is free for a public repository.

Consequences

  • "How is security testing planned?" now has one answer — this ADR and architecture §13 — tier by tier and milestone by milestone, with each threat traced to the test that defends it.
  • Security testing is a spine through the build, not a gate at the end — the same move ADR-0027 makes for the threat model and ADR-0025 for conformance.
  • Security properties get the same seed-reproducible, regression-on-failure rigor as correctness; the differentiator's guardrail (ADR-0010) extends to the whole threat catalog.
  • The tier split keeps the fast loop (Tier 0, laptop, no containers) distinct from the slow real-backend loop (Tier 1) and the human loop (Tier 2), so the cheap checks run constantly and the expensive ones run when they pay.
  • This consolidates, it does not contradict: ADR-0010 (the headline property), ADR-0025 (conformance), and ADR-0027 (abuse-case / fuzz / disclosure) are the pieces; this ADR is the frame that holds them.
  • The validation posture is open-source-native and unfunded by design: continuous free tooling (OSS-Fuzz, CodeQL, Scorecard), a public disclosure program, and inherited upstream-library audits stand in for a commissioned pentest — and being continuous, they do not go stale the way a point-in-time audit would. A funded audit, if it ever comes, is a bonus, not a prerequisite.