Vord

Architecture Decision Record

0040. Backup, restore, and disaster recovery across the two consistency domains

Context

ADR-0026 named two strongly-consistent domains — the Wyrd identity core and the Forseti relation store — and said each "carries its own residency, backup, and fence." That phrase hides the hard part. The two domains keep independent version spaces (they do not share a clock — the very reason the consistency token is composite, ADR-0026 §2), so an uncoordinated restore — core to time T, relation store to time T′ — yields a state the composite token cannot fence coherently: permissions visible for principals or credentials the restored core predates, or grants referencing relations that no longer exist. The append-only event stream (ADR-0022) is replayable, but replay is not a point-in-time restore and does not by itself bound recovery time or recovery point. No record states the cross-domain restore invariant, the RPO/RTO posture, or where the edge projections sit in a recovery. This closes that gap before the two-domain model hardens at M3/M4.

Decision

  1. Per-domain backups, version-stamped against a common ordering reference. Each domain is backed up on its own, but every backup records the identity event-stream offset (ADR-0022) it is consistent with, so a coherent recovery point can be chosen across both. (Under the reserved single-domain collapse — ADR-0026 §4 — this is trivial: one clock, one offset.)

    The full CQRS stream is milestone M5, but the two-domain model hardens at M3/M4 — so the ordering reference cannot wait for M5. A minimal ordering offset ships with M3: a durable, monotonic sequence stamped on every authoritative write in both domains (the embryo of the ADR-0022 stream, which later adopts it as its offset), sufficient for backups to record a coherent point. The M4 release therefore ships with a joint-restore story, not ahead of one.

  2. The cross-domain restore invariant: authorization may lag identity, never lead it. After a joint restore the Forseti relation store MUST be at an event-stream offset ≤ the Wyrd core's offset. A restore that leaves relations ahead of identity is forbidden — it would surface grants for principals and credentials the restored core does not yet contain (the New-Enemy failure, ADR-0004, in reverse). The recovery procedure restores the core to a chosen offset, then brings the relation store to an offset at or before it.

  3. Projections are rebuilt, never restored. The status list (ADR-0005), the JWKS (ADR-0023), sessions (ADR-0016), and read-model projections are CQRS-derived; they are reconstructed from the restored authoritative domains plus the event stream, not backed up and restored — so they cannot be a restore inconsistency. A restore re-publishes a fresh, monotonically newer status list (ADR-0032), advancing the replay-protection counter rather than rolling it backward.

    Outstanding consistency tokens need the same care: a composite token (ADR-0026) issued before the disaster may reference a Wyrd version or Forseti zookie ahead of the restored state. A fenced read presented with a fence the restored domain cannot recognize or satisfy fails closed (the read is refused or routed as too-new, never silently served from the rolled-back state), and the caller re-acquires a token from a fresh authoritative operation. Erasure events recorded after the recovery point are re-applied before serving traffic (ADR-0034), so a restore never resurrects a destroyed key.

  4. RPO and RTO are stated budgets and first-class SLOs. Per domain and composed, they join the freshness budgets (ADR-0039) as instrumented promises (proposal 0012) — the recovery analog of the kill-now budget. Numbers are operator- and milestone-set; the posture — continuous backup of both authoritative domains and the event stream, projections excluded, the offset invariant enforced — is fixed now.

  5. Key material and erased data are governed by their own decisions. Signing keys are HSM/KMS-resident with custody-side backup/escrow (ADR-0003, ADR-0023); a core restore predating a rotation leans on the retiring-key overlap (ADR-0023), and an unrecoverable key takes the emergency path (ADR-0036). Crypto-shredded data stays unrecoverable by design (ADR-0034): a restore never resurrects an erased principal's PII, because the per-principal key is destroyed, not the ciphertext — the correct and legally required behavior.

  6. A restore is an audited, abuse-case-tested operation. It emits events (ADR-0022); a restore that would violate the offset invariant fails closed; an abuse-case test (ADR-0028) asserts that a relations-ahead-of-identity restore is rejected.

Consequences

  • The two-domain model's hardest operational consequence — a coherent joint recovery point — is defined, not left to "each has its own backup."
  • RPO/RTO become stated, instrumented promises alongside the freshness budgets (ADR-0039), so disaster recovery is a budget, not a hope.
  • Erasure and DR are reconciled: crypto-shredded PII survives a restore as gone, which is what right-to-erasure (ADR-0034) requires.
  • The reserved single-domain collapse (ADR-0026 §4) gains another recorded payoff: one clock makes coherent restore trivial — a point in its favour.
  • Refines ADR-0026; relates to ADR-0022, ADR-0034, ADR-0036, and ADR-0039.