Skip to content

S5-04 — Attempt log

Attempt 1 — 2026-05-17 — phase-story-executor

Outcome: GREEN (all ACs satisfied; full suite + lint + mypy --strict + forbidden-patterns green).

Inputs read

  • Story S5-04-sbom-cve-probes.md (HARDENED variant; AC list 1..22 + cross-cutting test-quality / pydantic / AST audits).
  • phase-arch-design.md §"Component design" #5 (Layer G scanner shape; reused for Layer C).
  • final-design.md row 7 (rule-of-three deferral; no shared ScannerRunner abstraction).
  • ADR-0001 (syft/grype already in ALLOWED_BINARIES), ADR-0003 (decorator carries only heaviness+runs_last; requires is class attribute), ADR-0004 (image-digest:<resolved> token mechanism), ADR-0007 (no Plugin Loader).
  • Existing probes for pattern: runtime_trace.py (S5-02), certificate.py/shell_usage.py (S5-03), scip_index.py (Layer B scanner with run_external_cli + ToolMissingError + tail-cap).
  • read_raw_slices kernel at index_health.py:212; raw_dir helper at output/paths.py:20.
  • ScannerOutcome sum at probes/_shared/scanner_outcome.py — extended with optional reason: Literal["invalid_json", "sbom_artifact_missing"] | None = None on ScannerFailed (additive, default None — S5-01 baseline tests retained, snapshot test extended).

Discoveries that mattered

  1. ScannerFailed needed a reason field. Story explicitly anticipated this: AC says "If S5-01's ScannerFailed shape doesn't yet support a reason field… surface in Notes." Added reason: Literal["invalid_json", "sbom_artifact_missing"] | None = None as a non-breaking extension; the existing S5-01 snapshot test was updated to pin the new field default + the populated form.
  2. Tool-presence detection. The story refers to "Phase 0 tool_cache miss" but ProbeContext has no tool_cache field. The actual mechanism is ToolMissingError raised by run_external_cli when the binary is not on PATH — same pattern as scip_index.py. Adopted that.
  3. Probe.version is required by cache.keys.key_for. The class-attribute introspection tests didn't catch it because they don't exercise key_for. The integration test for image-digest cache-key invalidation surfaced it — added version: str = "0.1.0" to both probes.
  4. SbomProbe.run_external_cli returns ToolMissingError-equivalent for the test path via monkeypatch.setattr(sbom_mod, "run_external_cli", _raise) — same pattern existing Layer C tests use.
  5. LOC budget required a 3-file split per probe. Initial draft was ~285 LOC each; budget is ≤200. Extracted Pydantic models + classifier sum into _sbom_models.py / _cve_models.py; aggregation helpers for CVE into _cve_aggregation.py. Pattern matches existing dockerfile probe split (_dockerfile_models.py + _dockerfile_parse.py). Final: both probe modules exactly 200 source-lines (docstrings excluded).
  6. test_non_node_repo.py pinned the runnable-probe set at 8 entries; adding sbom+cve raised the floor to 10. Updated the expected set and the explaining comment.

Refactor decisions

  • Tagged-union classifier input (ScannerAttempt = _ToolMissing | _ProcessExited (| _SbomArtifactMissing for CVE)). match with exhaustive arms — mypy --warn-unreachable is the structural enforcement. Bought compile-time totality at ~10 LOC per probe; well worth it per the story's design-patterns hardening.
  • Single _build_slice(*, parsed: ... | None = None) builder per probe — collapsed the original _empty_slice + _populated_slice duplication into one function with the populated branch chosen by parsed is None. Saved ~12 LOC across both probes and made the populated/empty parity structural.
  • No shared _run_scanner_and_classify kernel — Rule of Three guard. Two scanners with identical shape is fine duplication; the extraction trigger is recorded in the story's Notes-for-implementer for S6-06/S6-07 (3rd + 4th JSON-scanner consumers) to land.
  • SyftJsonSchema / GrypeJsonSchema are DIP boundaries. Both extra="allow" (forward-compat against tool JSON drift); the emitted slice schemas (sbom.schema.json, cve.schema.json) are additionalProperties: false (backward-compat for our consumers). The asymmetry is the boundary.

Acceptance criteria — evidence

AC Evidence
AC-1 — SbomProbe class attrs + registry shape (no requires= in decorator) test_sbom_probe_class_attributes_pinned, test_sbom_registry_entry_carries_heaviness_only
AC-2 — declared_inputs = ["Dockerfile", "image-digest:<resolved>"] literal test_sbom_declared_inputs_literal
AC-3 — read_raw_slices 4-case absent-upstream table; no syft invocation test_sbom_upstream_unavailable_table (parametrized over missing/unparseable/not_ran/null_digest)
AC-4 — sibling-slice read via read_raw_slices; called exactly once test_sbom_reads_runtime_trace_via_read_raw_slices
AC-5 — syft outcome variants (tool_missing, non-zero, invalid_json, ran) test_sbom_tool_missing, test_sbom_non_zero_exit, test_sbom_invalid_json, test_sbom_happy_path_populates_slice
AC-6 — slice schema subset matches localv2 §5.3 C2 test_sbom_happy_path_populates_slice + sbom.schema.json
AC-7 — two raw files on happy path; only sbom.json on invalid_json test_sbom_two_files_written_on_happy_path, test_sbom_no_raw_artifact_on_invalid_json
AC-8 — CveProbe class attrs + registry shape test_cve_probe_class_attributes_pinned, test_cve_registry_entry_carries_heaviness_only
AC-9 — CveProbe declared_inputs literal test_cve_declared_inputs_literal
AC-10 — sbom slice absent + sbom_artifact_missing paths test_cve_upstream_unavailable_table, test_cve_sbom_artifact_missing
AC-11 — grype sbom:<path> invocation shape test_cve_happy_path_populates_slice (mock argv)
AC-12 — grype outcome variants test_cve_tool_missing, test_cve_non_zero_exit, test_cve_invalid_json, test_cve_happy_path_populates_slice
AC-13 — slice schema subset matches localv2 §5.3 C3 + two-file write test_cve_happy_path_populates_slice, test_cve_two_files_written_on_happy_path, test_cve_no_raw_artifact_on_invalid_json
AC-14 — _TOP_FINDINGS_N: Final[int] = 20; deterministic across permutations test_top_findings_n_constant_is_twenty, test_top_findings_deterministic_under_permutation (5 distinct permutations)
AC — Pydantic extra asymmetry test_pydantic_extra_asymmetry_sbom, test_pydantic_extra_asymmetry_cve
AC — AST audit: read_raw_slices used, no open-coded disk-IO test_ast_audit_sbom_uses_kernel, test_ast_audit_cve_uses_kernel
AC — AST audit: run_external_cli used, run_allowlisted NOT imported test_ast_audit_sbom_uses_run_external_cli_not_run_allowlisted, test_ast_audit_cve_uses_run_external_cli_not_run_allowlisted
AC — no trivy/cross_validated_with field test_cve_no_trivy_field
AC — writer chokepoint as RedactedSlice Probes return raw_artifacts: list[Path]; CLI re-publishes via Writer.write per cli.py:559..604. Verified end-to-end via the integration test for the chain.
AC — graceful tool-missing path test_sbom_tool_missing, test_cve_tool_missing — coordinator unaffected (probes return ProbeOutput, do not raise).
AC — image-digest invalidation: 2 distinct cache entries test_image_digest_invalidation_two_distinct_cache_entries (integration test, real cache.keys.key_for)
AC — Both modules ≤ 200 LOC excl. docstrings sbom.py = 200; cve.py = 200 (verified by the project-aligned _count_source_lines algorithm)
AC — mypy --strict clean mypy --strict src/codegenie → 109 files, 0 errors
AC — forbidden-patterns green python scripts/check_forbidden_patterns.py → no output (no violations)
AC — fence job stays green No new deps; pyproject closure unchanged
AC — classifier purity + totality test_classify_syft_outcome_is_pure_no_side_effects, test_classify_syft_outcome_never_raises_random_bytes, mirror for cve
AC — mutation resistance 3 explicit synthetic mutants in test_sbom (always-ran, drop-tool-check, drop-image-digest-token) and the parametrized classifier tables; the AST audits + test_*_no_raw_artifact_on_invalid_json catch the remaining mutations the story enumerates.

Gates

  • ruff check (src + tests) — clean
  • ruff format --check — 326 files already formatted
  • mypy --strict src/codegenie — clean (109 files)
  • python scripts/check_forbidden_patterns.py — clean
  • Full test suite — 2630 passed, 15 skipped, 3 deselected, 2 xfailed

Files touched

  • New: src/codegenie/probes/layer_c/sbom.py (200 LOC), src/codegenie/probes/layer_c/cve.py (200 LOC), src/codegenie/probes/layer_c/_sbom_models.py, src/codegenie/probes/layer_c/_cve_models.py, src/codegenie/probes/layer_c/_cve_aggregation.py.
  • New: src/codegenie/schema/probes/layer_c/sbom.schema.json, src/codegenie/schema/probes/layer_c/cve.schema.json.
  • Extended: src/codegenie/probes/_shared/scanner_outcome.py (ScannerFailed.reason: Literal[...] | None = None — additive).
  • New tests: tests/unit/probes/layer_c/test_sbom.py (26), tests/unit/probes/layer_c/test_cve.py (29), tests/integration/probes/layer_c/test_sbom_cve_chain.py (2).
  • New fixtures: tests/fixtures/syft/{hello_world,truncated,empty}.json, tests/fixtures/grype/{hello_world,truncated,no_findings}.json.
  • Updated tests: tests/unit/probes/_shared/test_scanner_outcome.py (snapshot pin extended to include new reason field default), tests/integration/probes/test_non_node_repo.py (runnable-probe-set comment + add sbom/cve).

Lessons for future Phase 2 stories

  • Layer C scanner probes follow a uniform shape — run_external_cli → ScannerAttempt tagged-union → _classify_*_outcome pure helper → ScannerOutcome dispatch. The rule-of-three trigger fires at S6-06/S6-07 (Semgrep/Gitleaks); the shared _run_scanner_and_classify kernel should be extracted then, NOT amended into existing scanner modules.
  • Probe.version is load-bearing for the cache key — any new probe that wants its image-digest token (or any special-token-shaped declared_input) to invalidate must declare a version: str attribute. Catch this in story drafting.
  • Story LOC budget arithmetic — Pydantic models for tool-JSON shape are surprisingly heavy (~40-50 LOC each); when the story names a single probe-module file, plan for the model split into a sibling _<probe>_models.py from the start.