Skip to content

Attempt log — S2-02 (Replay verification)

Attempt 1 — 2026-05-25 — GREEN

Story: docs/phases/06-sherpa-vuln-loop/stories/S2-02-replay-verification.md (HARDENED, 15 ACs). Validation report: _validation/S2-02-replay-verification.md. Executor model: Opus 4.7 via phase-story-executor skill (inline four-lens validation first; ReAct + red-green-refactor TDD; per-AC mutation-resistance check). Outcome: GREEN — all 15 ACs satisfied with runtime evidence; full local gate passes (lint, mypy --strict, import-linter, full phase-6 + S2-01 + cross-phase test suite); 4 pre-existing env flakes unchanged (tsconfig perf, pre-commit/lint-imports/mkdocs PATH issues — all green when re-run with venv on PATH; identical set the S2-01 attempt log surfaced).

Files shipped (new)

Files modified

  • src/codegenie/workflows/checkpoints.py — additive Protocol extension (ADR-0003 amendment 2026-05-25): added sixth method iter_persisted_chain(workflow_id) -> Iterator[tuple[TransitionEvent, ChainHead]]. No method removed; no signature changed; one new method whose docstring documents the S2-02 verifier rationale and the detection-substrate-only contract S2-01 AC-11 inherits.
  • src/codegenie/workflows/sqlite_checkpoints.py — implemented iter_persisted_chain (single SELECT event_bytes, next_head ORDER BY sequence ASC); changed append() to fold the chain head over the sanitized-reconstructed event rather than the live event (S2-02 AC-3 sanitization-aware chain discipline — the chain protects bytes on disk so the verifier can reproduce the head from the persisted bytes; for events with no secret-shaped content the reconstructed event is byte-equal to the live event, so chain heads / goldens are unchanged).
  • src/codegenie/workflows/in_memory_checkpoints.py — implemented iter_persisted_chain; mirrored the sanitization-aware chain change so parity holds byte-for-byte.
  • tests/fence/test_chain_head_purity.py — extracted _scan_module_for_impurity helper; added test_replay_module_has_no_impure_names walking _replay.py with the same forbidden-name set (AC-3 fence extension).
  • tests/fence/test_checkpoint_sanitizer_imports.py — added _REPLAY_MODULES = (replay, _replay) and a test_s202_no_local_regex_in_replay_modules test asserting no re / regex imports in either new module (AC-12 fence extension).
  • tests/fence/test_checkpoint_adapter_slots.py — added test_s202_replay_verifier_declares_slots asserting ReplayVerifier.__slots__ == ("_store",) (AC-4 fence extension).
  • tests/integration/test_phase6_sut_contract_snapshot.py — extended build_snapshot with replay_verdict_schema, hydration_result_schema, replay_verifier_methods (signatures), replay_module_functions, replay_verdict_kinds, hydration_result_kinds, integrity_error_id; extended classify_snapshot_diff to classify verdict-shaped deltas (removed verdict kind = breaking; new = additive; verifier method/signature change = breaking; verdict schema diff = breaking; error_id change = breaking) (AC-14 extension).
  • tests/unit/workflows/test_checkpoint_store_protocol.py — five-method → six-method expected-set (additive Protocol extension; docstring updated to call out ADR-0003 amendment 2026-05-25).
  • tests/golden/phase6-contract/snapshot.json — regenerated via PHASE6_CONTRACT_GOLDEN_REWRITE=1 pytest tests/integration/test_phase6_sut_contract_snapshot.py to capture the additive verifier-shape entries.

AC-by-AC evidence (each AC tied to a named test + a mutation-resistance assertion)

  • AC-1 (verdict discriminated union shape): test_replay_verdict_shape.py — 7 tests verify (i) all five frozen-forbid configs; (ii) the byte-equal kind literals; (iii) ReplayVerdict has exactly four members; (iv) HydrationResult is exactly Hydrated | FailedUnrecoverable; (v) ChainMismatch.divergence_index rejects negatives via Field(ge=0); (vi) TornWrite.reason is the closed three-element Literal; (vii) variants are immutable.
  • AC-2 (__all__ unchanged): test_workflows_public_surface.py — 4 tests continue to pass byte-equal to the 14-name allowlist. The verifier types are package-private (not in codegenie.workflows.__all__).
  • AC-3 (pure-core fold + sanitization-aware): test_replay_sanitization_aware.py — 4 tests verify (i) SQLite round-trip with AKIA… secret-shaped payload yields Verified not ChainMismatch; (ii) InMemory same; (iii) empty iterable returns genesis; (iv) fold-over-sequence equals chained append() -> ChainHead. The AST no-side-effects fence at test_chain_head_purity.py::test_replay_module_has_no_impure_names walks _replay.py.
  • AC-4 (ReplayVerifier __slots__): test_checkpoint_adapter_slots.py::test_s202_replay_verifier_declares_slots — asserts __slots__ == ("_store",).
  • AC-5 (verdict-classification matrix): test_replay_verify_classifications.py — 8 tests covering empty-workflow, clean-completion, tail-tamper (divergence_index=last), middle-tamper (divergence_index=1 — catches back-to-front verifiers), bytes-swap, truncated-bytes torn-write, verdict-kind is one of four closed slugs, and source-grep that confirms no SQLite-specific shortcut in replay.py.
  • AC-6 (parity across both adapters): test_replay_verifier_parity.py — 5 tests parametrized over [InMemoryCheckpointStore, SqliteCheckpointStore]: empty-workflow, clean-completion, tail-tamper byte-equal across adapters (divergence_index + offending_transition_id match).
  • AC-7 (AST exhaustiveness gate): test_replay_exhaustiveness.py — 2 tests: _dispatch_verdict has exactly four named case arms (one per verdict kind) plus exactly one wildcard case _: arm that raises AssertionError.
  • AC-8 (hydrate_or_fail routing): test_hydrate_or_fail_routing.py — 6 tests: empty→Hydrated(events=(), latest_state_kind="needs_plan"); verified→Hydrated(events=..., latest_state_kind="completed"); chain-mismatch→FailedUnrecoverable with divergence_index=1 + chain_mismatch substrings; torn-write→FailedUnrecoverable with torn_write + unparseable_event substrings; _INTEGRITY_ERROR_ID matches Phase-1 ADR-0007 grammar; Hydrated.kind == "hydrated" is NEW (not a LedgerStateKind).
  • AC-9 (tamper integration golden — Scenario #4): test_replay_tamper_golden.py — 1 test: raw-SQLite UPDATE of middle row's next_head, hydrate_or_fail returns FailedUnrecoverable(reason="checkpoint_integrity") with divergence_index=1 + offending transition_id substrings (catches back-to-front verifiers).
  • AC-10 (torn-write integration golden): test_replay_torn_write_golden.py — 2 tests: truncated-JSON UPDATE → torn_write + unparseable_event; valid-JSON-but-not-event-shape UPDATE → same classification (covers schema-incompatible payload).
  • AC-11 (fail-closed-before-hydrate): test_hydrate_no_state_leak.py — 2 tests: the failure path returns only FailedUnrecoverable (no non-terminal variant) + AST scan of replay.py for forbidden constructors. Companion AST fence at test_hydrate_no_state_construction.py gates the same invariant at the source level.
  • AC-12 (no regex fork): test_checkpoint_sanitizer_imports.py::test_s202_no_local_regex_in_replay_modules — AST-walk over replay.py + _replay.py rejects any re / regex import.
  • AC-13 (mypy --strict clean): make typecheck — 245 source files clean.
  • AC-14 (contract snapshot extension): test_phase6_sut_contract_snapshot.py + the meta-test classifier — golden regenerated; the classifier now handles verdict-shaped deltas (verdict kind removal = breaking; addition = additive; verifier method/signature drift = breaking; integrity_error_id rename = breaking).
  • AC-15 (parity meta-test): test_replay_verifier_parity_meta.py — plants a _BrokenVerifier that returns Verified for any input; asserts the parity assertion would fail on it (demonstrating the assertion is non-trivial).

Decisions of record

  • Additive sixth CheckpointStore Protocol method iter_persisted_chain. The verifier needs per-row persisted next_head to compute divergence_index (AC-1 / AC-5 / AC-9). The S2-01 Protocol exposed only the tail. Three options were considered: (a) widen the Protocol additively, (b) drop divergence_index from the verdict, (c) have the verifier do substrate-specific reads. Chose (a) — the cleanest substrate-agnostic answer; the Protocol stays the kernel. Recorded as ADR-0003 amendment 2026-05-25; the existing five methods are byte-equal-unchanged; the contract-snapshot meta-test classifies this delta as additive.
  • Chain head folded over sanitized-reconstructed event, not live event. The S2-01 attempt log explicitly surfaced this as a "design point for S2-02": the chain head was computed over the LIVE event (cleartext) while the on-disk row was the SANITIZED bytes — meaning the verifier could not reproduce the head from persisted bytes when sanitization triggered. The fix is in SqliteCheckpointStore.append() and InMemoryCheckpointStore.append(): reparse sanitize_for_persistence(canonical_bytes) into a TransitionEvent and fold over that reconstructed event. For events with no secret-shaped content the reconstructed event is byte-equal to the live event (sanitize is a no-op), so existing chain heads + the tests/golden/phase6-checkpoint/clean_completion_chain.json golden are unchanged. This is the recommended approach (i) from the S2-01 attempt log.
  • Single closed _INTEGRITY_ERROR_ID constant in replay.py. No project-wide error_id registry today; when one lands (Phase 9+), the constant migrates additively (one-line move). Phase-1 ADR-0007 grammar (dotted_snake_case.dotted_snake_case) enforced by test_ac8_integrity_error_id_matches_phase1_grammar.
  • Hydrated.kind = "hydrated" is a NEW closed-set tag, not a reused LedgerStateKind. The two unions answer different questions (LedgerStateKind = "what state is the workflow IN?"; Hydrated.kind / FailedUnrecoverable.kind = "what HAPPENED during hydration?"). Reuse would let Hydrated(kind="needs_plan") slip through as a category error. Test sanity-checks that "hydrated" is NOT a member of LedgerStateKind.
  • while True: next(iterator) loop instead of enumerate(...) in verify(). Initial implementation used for sequence_count, (event, persisted_next_head) in enumerate(self._store.iter_persisted_chain(...)). Bug: when ValidationError raised by the iterator's next() (parse failure on the NEXT row), sequence_count was still at the PREVIOUS row's index. The while True / try / next(iterator) shape advances the counter only after a successful iteration, so TornWrite.offending_sequence reports the correct (failing) row.
  • No null_event_bytes / duplicate_chain_link test today (kept in the TornWrite.reason Literal for forward extensibility). The SQLite NOT NULL constraint + UNIQUE index defend against both modes in production; through the Protocol, NULL bytes manifest as model_validate_json(None) raising ValidationError → classified as unparseable_event. The two reserved reasons stay in the closed Literal for when Phase-9 substrates need them (the contract-snapshot meta-test would classify dropping them as breaking).
  • AST exhaustiveness test, not just mypy --strict. Mypy's match exhaustiveness check via assert_never is Python 3.11+ and interacts inconsistently with pydantic discriminated unions across mypy versions. The AST test counts case-arms at the source level and is version-independent. Both gates land — they catch overlapping but non-identical mutation classes.
  • Per-mapping hydrate_or_fail routing tests, not parametrize. The four mappings have different setup costs (Verified needs a 3-event fixture; EmptyWorkflow needs zero; ChainMismatch needs a raw-SQLite tamper; TornWrite needs a constraint-relaxed fixture). Parametrize would force a single setup path and either over-build for the simple cases or under-build for the complex ones.

Refactor decisions

  • Composition over inheritance for the verifier + the pure-core fold. Pure helper _replay_fold lives in _replay.py (module-level, no class). The ReplayVerifier class composes via constructor injection of CheckpointStore. No BaseReplayVerifier ABC, no VerifierMixin. Anti-refactor #1 honored.
  • No VerifierStrategy Strategy abstraction. The fold is the one canonical policy (sanitize-then-fold). A "loose" toggle would silently mask AC-3. Anti-refactor #2 honored.
  • No ReplayCache. Verification is idempotent and cheap (fold over ≤ N rows for any realistic resume). Caching couples invalidation to tamper detection — exactly the failure mode this story prevents. Anti-refactor #6 honored.
  • No verify_or_raise() convenience wrapper. The tagged-union return discipline is the canonical contract; raising defeats the discriminated-union exhaustiveness guarantee. Anti-refactor #8 honored.
  • Callable[[], datetime] as the verifier's clock type (not needed — verifier has no clock site; the only clock site in the substrate is the SQLite adapter's written_at column from S2-01). Confirms Anti-refactor #6 from S2-01 inherited here.
  • No async verify(). The orchestrator wraps in asyncio.to_thread (same pattern S2-01 pinned for append). Anti-refactor #5 honored.
  • _fold_one thin wrapper around _replay_fold([event]). Keeps the imperative shell free of _chain.py's import name so the AST fence over _chain.py cannot accidentally trip on a re-export in replay.py. Single canonical fold path.

Test counts touched

  • Suite-level: 7365 passed, 44 skipped, 9 xfailed, 4 pre-existing env flakes (tsconfig perf flake + 3 PATH-dependent failures — same set surfaced by the S2-01 attempt log; all pass when re-run with the venv on PATH). Net delta from S2-01: +44 passes (50 new tests + 1 amended test_checkpoint_store_protocol test that now expects 6 methods; minus expected zero-failure regressions).
  • Phase-6 verifier suite: 35 tests across 9 new files + 4 amended fence/contract files.
  • Mypy: 245 source files clean under --strict.
  • Ruff: all checks passed; ruff format clean.
  • Import-linter: 12 contracts kept, 0 broken.

Notes for downstream stories

  • S3-01 (plugin-local subgraph) consumes hydrate_or_fail(store, workflow_id) at the subgraph's "verify + hydrate" entry edge. The orchestrator should branch on result.kind: "hydrated" → start the subgraph with result.events + result.latest_state_kind; "failed_unrecoverable" → emit the typed terminal state (the orchestrator should NOT construct any non-terminal ledger state when the result is FailedUnrecoverable — the AC-11 AST fence enforces this in the verifier module; the subgraph module needs its own consistency).
  • S4-01 (HITL interrupt-and-resume) reads result.events to find the latest awaiting_human_review row via next((e for e in reversed(result.events) if e.next_state_id == "awaiting_human_review"), None). The HITL resume validator's stale-approval rejection is separate from this story's chain-integrity rejection.
  • S5-01 (LocalVulnRemediationSut) constructs the SqliteCheckpointStore(root, clock=...) and threads it into the orchestrator. The Phase-9 Postgres adapter swap (S5-01 in Phase 9) is a single-line constructor change — the verifier is substrate-agnostic by construction.
  • Phase-9 G5 byte-equality. The verifier reproduces the rolling chain head BYTE-FOR-BYTE from the persisted (possibly sanitized) bytes — the load-bearing invariant the Phase-9 SQLite ↔ Postgres byte-equality test depends on. Phase-9's parity-matrix addition is one row in ADAPTER_FACTORIES + an additive parametrize row in tests/integration/test_replay_verifier_parity.py.

Follow-ups surfaced this attempt

  • ADR-0003 amendment recording the additive sixth Protocol method. Should land as a small in-repo edit to docs/phases/06-sherpa-vuln-loop/ADRs/0003-checkpointed-ledger-replay-boundary.md noting "Amendment 2026-05-25: CheckpointStore Protocol gains additive sixth method iter_persisted_chain(workflow_id) -> Iterator[tuple[TransitionEvent, ChainHead]] to enable replay-verifier per-row inspection without coupling to a substrate-specific read path." (Trivial; could fold into a doc-only commit.)
  • ADR-0003 amendment recording the chain-input shift to sanitized-reconstructed event. The S2-01 attempt log surfaced this as a design decision for S2-02; this attempt closed the decision but the ADR itself doesn't record it. A small "Amendment 2026-05-25" paragraph in the same ADR file would be the right home.
  • S2-01 story prose follow-up. S2-01's AC-3 prose (sanitization-aware fold "load-bearing invariant") now refers to S2-02's resolution; could append a short forward-reference to S2-01's story file pointing at this attempt log. (Trivial; doc-only.)
  • null_event_bytes / duplicate_chain_link test coverage. Currently un-tested (kept in the closed Literal for forward extensibility). If a Phase-9 substrate emerges that can produce either mode through the Protocol surface, add coverage at that point. (Minor; deferred per Rule 2.)
  • CI tsconfig perf flake. Pre-existing; appears in S2-01 + earlier attempt logs. Out of scope for this story but worth a follow-up phase-shakedown to investigate root cause (likely macOS-specific timing variance under load).