Emulation Reporting & Metrics

Definition

Emulation reporting is the discipline of converting exercise and automation outputs into decisions: structured after-actions that operators and engineers act on, coverage matrices that managers plan against, and trend metrics that executives fund with confidence. The artifacts form a chain — raw results (atomic runs, exercise manifests, canary records) → joined evidence (per-technique pass/gap with telemetry citations) → metrics (coverage percentage, regression count, gap closure time) → narrative (what changed, why, what it means for risk) — and each link has different authors and different readers. Reporting done well is the difference between a program that generates activity and a program that generates improvement: the backlog gets prioritized by measured risk, regressions get attributed and fixed, leadership sees a defensible line instead of anecdotes, and the emulation program earns its budget renewals the same way any engineering function does — by demonstrating, with data, that the estate’s defenses measurably improved under its watch.

Why It Matters

Every technical artifact this handbook produces — a validated detection, a found gap, a measured control — is worthless if it dies in an unshared document or an unexplained spreadsheet. Reporting is where analysis becomes organizational change, and emulation reporting carries a specific burden: its metrics justify security spending in an environment where every control’s value is invisible until the day it matters. A coverage matrix that says “credential access: 62% validated, regression in Sysmon 10 masks on 3 hosts” is fundable and actionable; “we do purple teaming” is neither. The discipline also protects the program from its own failure modes: without trend tracking, exercises repeat the same findings for years (nothing closes); without regression attribution, decay is invisible (controls silently break and the matrix stays green because nobody re-tested); without audience-layered reporting, technical truth gets translated badly upward and funding decisions get made on vibes. And on the trust dimension, emulation reports are read adversarially — by executives comparing against vendor claims, by auditors mapping to frameworks, by boards who have been burned by green dashboards before — so the honesty disciplines this series built (measured baselines, cited evidence, honest gaps) are not just engineering hygiene; they are the reporting’s entire credibility. The craft closes the loop the whole T-series opened: techniques → telemetry → detections → measurements → decisions → funded fixes → re-validated techniques.

Detailed Explanation

The reporting stack has four layers, each with its own contract:

 REPORTING DATA FLOW
 [raw results]          atomicrun.json, exercise manifests,
      |                 canary alerts, drill reports
      v
 [joined evidence]      per-technique: status + telemetry
      |                 citations + rule IDs + timestamps
      v                     (covreport, chainjoin joins)
 [metrics layer]        coverage %, regression count,
      |                 gap MTTR, control-validation rate,
      v                 flake rate, trend deltas
 [narrative layer]      exec summary (3 bullets, 1 ask),
                        engineer backlog (owner + fix),
                        manager plan (quarter over quarter)

 METRIC DESIGN RULES
 1. every metric has a DENOMINATOR that cannot be gamed
    (validated / cataloged techniques, not "tests run")
 2. every metric has PROVENANCE (which run, which baseline)
 3. trends outrank snapshots (direction beats position)
 4. gaps carry OWNERS and AGES (backlog = debt, visible)
 5. regressions carry ATTRIBUTION (what changed)

The joined-evidence layer is where most programs under-invest: raw results need joining (telemetry citations attached to each verdict, rule IDs from the inventory, timestamps correlated to control changes) before any metric is honest — this is the covreport/chainjoin work from T8/T9, elevated to a standing practice. The metrics layer follows five design rules worth internalizing: denominators that cannot be gamed (coverage = validated techniques ÷ cataloged techniques, where the catalog is version-controlled — never “tests run”, which inflates with effort rather than outcome); provenance on every number (which run produced it, against which baseline — the T8 circular-calibration lesson made structural); trends over snapshots (a 62% that was 55% last quarter tells a better story than a flat 70%); gaps with owners and ages (an unowned gap is a decision nobody made, and an aging gap is a visible debt); regressions with attribution (the timestamp that correlates to a change is what makes the fix findable). The narrative layer splits by audience: the executive summary is three bullets and one ask (what we validated, what broke, what we need); the manager view is the quarter-over-quarter matrix and the backlog’s debt curve; the engineer view is the failing rows with telemetry citations and reproducible procedures. The after-action template for manual exercises reuses the same spine — scope, what fired, what didn’t (with evidence), gaps with owners, retest date — because consistency across manual and automated reporting is what makes the two halves of the program comparable. The honest complication: metrics that make the program look good are also metrics that can hide risk (a coverage percentage rising while the catalog’s hardest techniques sit ungated), which is why the anti-gaming review — metric definitions audited against their own provenance, per T9’s ⚠️ lesson — is a scheduled part of the practice, not an occasional scrub.

Common Techniques

Reporting technique	Artifact	Audience	Design rule
After-action template	Per-exercise report	Eng + SOC	Scope, evidence, gaps, retest date
Coverage matrix	Per-tactic table	Managers	Validated/gap/gated/flaky states
Coverage trend line	Quarter-over-quarter chart	Executives	Trend beats snapshot
Regression register	Attributed decay list	Detection eng	Change correlation
Gap-debt curve	Owner + age of open gaps	Managers	Visible debt, MTTR metric
Navigator layer export	ATT&CK visuals	Mixed	Per-exercise layers
Executive brief	3 bullets + 1 ask	Leadership	No unexplained acronyms
Framework mapping	NIST/ISO cross-refs	Audit/Compliance	Emulation → control evidence
Flake report	Quarantine queue	Eng	Signal hygiene
Honesty audit	Metric provenance review	Program lead	Anti-gaming discipline

Full Working Code

emureport.py — the emulation report generator: consumes a run-results JSON (atomic or exercise shape), a gap-tickets CSV (owner + age), and an optional history file; produces the layered markdown report — executive summary (auto-drafted from the numbers), coverage matrix, regression list, gap-debt table, and trend deltas vs the previous run. stdlib only.

#!/usr/bin/env python3
"""emureport.py — layered emulation report generator.
Usage:
  python emureport.py --results atomic_results.json \
      --gaps gaps.csv --history history.json \
      --title "Q4 emulation cycle" --out report.md
gaps.csv: technique,owner,age_days,status
history.json: prior report summaries (for trend deltas)
stdlib only.
"""
import argparse, csv, json, sys
from collections import defaultdict

def tactic_of(technique):
    if not technique or not technique.startswith("T"):
        return "other"
    try:
        major = int(technique[1:4])
    except ValueError:
        return "other"
    buckets = {2: "Initial Access", 3: "Execution",
               4: "Persistence", 5: "PrivEsc",
               6: "Defense Evasion", 7: "Cred Access",
               8: "Discovery", 9: "Lateral",
               10: "Collection", 11: "C2",
               12: "Exfiltration", 14: "Impact"}
    return buckets.get(major, "other")

def load_gaps(path):
    gaps = []
    try:
        with open(path, encoding="utf-8", newline="") as fh:
            for row in csv.DictReader(fh):
                gaps.append({
                    "technique": row.get("technique", "?"),
                    "owner": row.get("owner", "UNOWNED"),
                    "age": int(row.get("age_days") or 0),
                    "status": row.get("status", "open")})
    except OSError:
        pass
    return gaps

def summarize(results):
    matrix = defaultdict(lambda: defaultdict(int))
    for r in results.get("results", []):
        tech = r.get("technique")
        if not tech:
            continue
        matrix[tactic_of(tech)][r["status"]] += 1
    total = sum(sum(t.values()) for t in matrix.values())
    passed = sum(t.get("pass", 0) for t in matrix.values())
    failed = sum(t.get("fail", 0) for t in matrix.values())
    gated = sum(t.get("gated", 0) for t in matrix.values())
    cov = passed / (passed + failed) if (passed + failed) \
        else 0.0
    return {"matrix": matrix, "total": total,
            "passed": passed, "failed": failed,
            "gated": gated, "coverage": round(cov, 3)}

def trend_line(now, history):
    """Deltas vs the last history entry."""
    if not history:
        return None
    prev = history[-1]
    if "coverage" not in prev:
        return None
    return {"prev_coverage": prev["coverage"],
            "delta": round(now["coverage"] -
                           prev["coverage"], 3),
            "prev_run": prev.get("run_start", "?")}

def render(title, s, gaps, trend):
    L = [f"# {title}", ""]
    cov_pct = f"{s['coverage']:.0%}"
    L += ["## Executive summary", ""]
    L.append(f"- Validated coverage: **{cov_pct}** "
             f"({s['passed']}/{s['passed'] + s['failed']} "
             f"executable checks; {s['gated']} gated)")
    if trend:
        d = trend["delta"]
        arrow = "up" if d > 0 else ("down" if d < 0 else
                                    "flat")
        L.append(f"- Trend: **{arrow} {abs(d):.1%}** vs "
                 f"previous run "
                 f"({trend['prev_run'][:10]})")
    if gaps:
        old = [g for g in gaps if g["age"] > 90]
        unowned = sum(1 for g in gaps
                      if g["owner"] == "UNOWNED")
        L.append(f"- Gap debt: **{len(gaps)} open gaps**, "
                 f"{len(old)} older than 90 days, "
                 f"{unowned} unowned")
    L.append("- Ask: (fill per cycle — usually backlog "
             "capacity or a specific fix)")
    L += ["", "## Coverage matrix", ""]
    L += ["| tactic | pass | fail | gated |",
          "|---|---|---|---|"]
    for tactic in sorted(s["matrix"]):
        m = s["matrix"][tactic]
        L.append(f"| {tactic} | {m.get('pass', 0)} "
                 f"| {m.get('fail', 0)} "
                 f"| {m.get('gated', 0)} |")
    L += ["", "## Gap debt", ""]
    if gaps:
        L += ["| technique | owner | age (d) | status |",
              "|---|---|---|---|"]
        for g in sorted(gaps, key=lambda g: -g["age"]):
            L.append(f"| {g['technique']} "
                     f"| {g['owner']} | {g['age']} "
                     f"| {g['status']} |")
    else:
        L.append("_No open gaps recorded._")
    return "\n".join(L) + "\n"

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--results", required=True)
    ap.add_argument("--gaps")
    ap.add_argument("--history")
    ap.add_argument("--title", default="Emulation cycle")
    ap.add_argument("--out", default="report.md")
    args = ap.parse_args()
    results = json.load(open(args.results,
                             encoding="utf-8"))
    s = summarize(results)
    gaps = load_gaps(args.gaps) if args.gaps else []
    trend = None
    if args.history:
        try:
            hist = json.load(open(args.history,
                                  encoding="utf-8"))
            trend = trend_line(s, hist)
        except (OSError, json.JSONDecodeError):
            pass
    report = render(args.title, s, gaps, trend)
    open(args.out, "w").write(report)
    print(report)
    # append this run to history for the next cycle
    hist_entry = {"run_start": results.get("run_start"),
                  "coverage": s["coverage"],
                  "passed": s["passed"],
                  "failed": s["failed"]}
    hpath = args.history or "history.json"
    try:
        hist = json.load(open(hpath, encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        hist = []
    hist.append(hist_entry)
    json.dump(hist, open(hpath, "w"), indent=2)
    print(f"[+] {args.out} written; history updated "
          f"({len(hist)} runs)")

if __name__ == "__main__":
    main()

Code Walkthrough

• summarize() (lines 30-47) computes the metric layer’s spine from raw results: per-tactic status counts and the coverage fraction with a defined denominator (passed + failed — gated tests excluded and reported separately, because including gated in the denominator would flatter hosts running too little). The rounding to 3 decimals is deliberate: false precision at more digits is noise.
• trend_line() (lines 50-60) computes the delta against the last history entry, returning None cleanly when no history exists — first-run reports state position, subsequent reports state direction, per the design rule.
• render() (lines 63-103) builds the layered narrative: executive summary first (coverage, trend with direction, gap-debt headline with the 90-day-old and unowned counts — the two debt numbers that make managers act), the ask placeholder (never auto-filled — the ask is a human decision), then the matrix, then the gap-debt table sorted by age. The unowned count (lines 81-83) is the report’s sharpest edge: an unowned gap is a decision nobody has made, and the number makes that visible without editorializing.
• The history append (lines 137-146) closes the trend loop automatically — every report run files its summary into the history, so the next report’s trend line is always one command away and nobody has to remember to record the numbers.
• load_gaps() (lines 19-32) tolerates a missing CSV (no gaps file = no debt section, stated plainly in the report) — the report degrades honestly rather than inventing a zero.

How It Reaches The Target

The report reaches each audience at the altitude they act from: engineers read failing rows with telemetry citations and reproducible procedures (the T9 fail rows, now owned and aged); managers read the matrix and the debt curve, where prioritization becomes a staffing decision with numbers attached; executives read three bullets and one ask, where funding becomes a defensible choice rather than an act of faith. The trend line reaches the program’s governance as its steering signal: coverage direction, regression cadence, and gap-closure rate are the three numbers that tell the program lead whether the loop is working — and the regression register reaches the platform teams as concrete evidence (“this EDR update broke these three detections on this date”) that turns detection health into a shared responsibility rather than a security-only complaint. The gap-debt table reaches the backlog meeting as the prioritization instrument: aged gaps float to the top, unowned gaps get assigned in the meeting or they get accepted explicitly — either way the decision is made, which is the entire point. The framework mapping reaches audit and compliance as control evidence with provenance (which test, which run, which telemetry), a form of documentation auditors accept far more readily than self-attestation. And the honesty audit reaches the program itself: scheduled review of metric definitions against their provenance (T8’s circular-calibration lesson, T9’s assertion-integrity lesson) keeps the numbers meaningful — because a reporting stack, like a detection suite, decays without its own regression testing.

Exploitation Chain

1. Collect — atomic results + exercise manifests + canary records into the results store
2. Join — covreport/emureport joins evidence: verdicts + telemetry citations + rule IDs
3. Compute — metrics layer: coverage 78%, 6 regressions, gap debt 14 (2 unowned, 3 aged 90d+)
4. Draft — executive summary auto-drafted; ask filled by program lead (backlog capacity)
5. Review — honesty audit: denominators checked, provenance cited, no circular baselines
6. Deliver — layered reports: exec brief, manager matrix, engineer failing rows
7. Decide — backlog meeting: aged gaps assigned, unowned gaps accepted or staffed
8. Verify — next cycle: trend shows coverage 78% -> 86%, regressions 6 -> 1, debt 14 -> 8

Building Your Own Tool

metricdrift.py — the metric integrity monitor: audits the program’s own metrics over time for the gaming and decay patterns this module warns about — denominator changes (did the catalog grow without tests?), threshold provenance (was a coverage jump preceded by a definition change?), regression persistence (are regressions staying open?), and report-freshness (are metrics older than the control-change window?). It is the anti-gaming tool for the reporting layer itself. stdlib only.

Design decisions

• Audit the definitions, not the numbers. The tool checks how metrics were computed (denominators, provenance fields) rather than re-computing them — because the failure mode is definitional drift, not arithmetic error.
• History-aware checks. Each check runs across the history file, so “coverage jumped 20 points” is only flagged when it coincides with a denominator change or missing provenance — jumps from real improvement are the goal, not a finding.
• Findings as runbook rows. Output is a finding list with severity and the exact definition to review — the honesty audit’s worklist, not a verdict.
• Stdlib, scheduled. Runs monthly from cron alongside the coverage cycle; findings file like any other gap ticket.

Full source

#!/usr/bin/env python3
"""metricdrift.py — metric integrity audit for emulation reporting.
Usage:
  python metricdrift.py --history history.json \
      --catalog catalog.json [--runs results_dir/]
Checks:
  1. denominator drift: catalog size jumps without test growth
  2. provenance gaps: history entries missing run_start
  3. regression persistence: fails unchanged across >=3 runs
  4. stale reports: latest entry older than N days
  5. suspicious jumps: coverage delta >20pts with any check hit
stdlib only.
"""
import argparse, json, sys, datetime

def load(path, what):
    try:
        return json.load(open(path, encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as e:
        print(f"[!] cannot load {what}: {e}", file=sys.stderr)
        return None

def check_provenance(history, findings):
    for h in history:
        if not h.get("run_start"):
            findings.append(("HIGH",
                f"history entry missing run_start "
                f"({h.get('coverage')}) — provenance broken"))
    return findings

def check_denominator(history, catalog_size, findings):
    sizes = []
    for h in history:
        n = h.get("passed", 0) + h.get("failed", 0) \
            + h.get("gated", 0)
        sizes.append(n)
    if not sizes:
        return findings
    # catalog defines the honest denominator ceiling
    if catalog_size and sizes and sizes[-1] > catalog_size:
        findings.append(("HIGH",
            f"latest run ({sizes[-1]}) exceeds catalog "
            f"({catalog_size}) — denominator invented?"))
    for i in range(1, len(sizes)):
        if sizes[i] - sizes[i - 1] > max(10, sizes[i-1] * 0.5):
            findings.append(("MED",
                f"denominator jumped {sizes[i-1]} -> "
                f"{sizes[i]} between runs — definition "
                f"change or catalog growth? verify"))
    return findings

def check_persistence(history, findings):
    """Fails unchanged across >=3 consecutive entries."""
    streak = 0
    for h in history:
        if h.get("failed", 0) == history[0].get("failed", 0) \
           and h.get("failed", 0) > 0:
            streak += 1
        else:
            streak = 0
        if streak >= 3:
            findings.append(("MED",
                f"failed count unchanged ({h.get('failed')}) "
                f"for {streak} runs — regressions parked?"))
    return findings

def check_stale(history, max_age_days, findings):
    if not history:
        return findings
    last = history[-1].get("run_start")
    if not last:
        return findings
    try:
        ts = datetime.datetime.fromisoformat(
            last.replace("Z", "+00:00"))
        age = (datetime.datetime.now(
            datetime.timezone.utc) - ts).days
        if age > max_age_days:
            findings.append(("MED",
                f"latest report {age} days old "
                f"(max {max_age_days}) — metrics stale vs "
                f"control-change window"))
    except ValueError:
        findings.append(("LOW",
            "latest run_start unparseable"))
    return findings

def check_jumps(history, findings):
    for a, b in zip(history, history[1:]):
        ca, cb = a.get("coverage"), b.get("coverage")
        if ca is None or cb is None:
            continue
        if abs(cb - ca) > 0.20:
            findings.append(("LOW",
                f"coverage jumped {ca:.0%} -> {cb:.0%} — "
                f"verify against provenance checks above"))
    return findings

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--history", default="history.json")
    ap.add_argument("--catalog")
    ap.add_argument("--max-age", type=int, default=45)
    ap.add_argument("--out")
    args = ap.parse_args()
    history = load(args.history, "history") or []
    catalog = load(args.catalog, "catalog") if \
        args.catalog else None
    findings = []
    findings = check_provenance(history, findings)
    findings = check_denominator(
        history, len(catalog) if catalog else None, findings)
    findings = check_persistence(history, findings)
    findings = check_stale(history, args.max_age, findings)
    findings = check_jumps(history, findings)
    if not findings:
        print("[+] metric integrity: no findings")
        return
    order = {"HIGH": 0, "MED": 1, "LOW": 2}
    findings.sort(key=lambda f: order[f[0]])
    print(f"== metric integrity findings "
          f"({len(findings)}) ==")
    for sev, msg in findings:
        print(f"  [{sev}] {msg}")
    if args.out:
        with open(args.out, "w") as fh:
            fh.write("# Metric integrity audit\n\n")
            for sev, msg in findings:
                fh.write(f"- **{sev}** {msg}\n")
        print(f"[+] {args.out}")

if __name__ == "__main__":
    main()

Walkthrough

• check_denominator() (lines 26-43) is the anti-inflation check: a run whose test count exceeds the catalog’s size has invented a denominator, and jumps larger than 50% between runs get flagged for definition review — both are the exact gaming patterns the metric design rules forbid.
• check_persistence() (lines 46-56) detects parked regressions: an unchanged nonzero fail count across three consecutive runs means the regression is being tracked but not fixed — the debt the gap-debt table should be showing, surfaced independently in case the report layer dropped it.
• check_stale() (lines 59-73) ties metric freshness to the control-change window: in an estate where controls change weekly, a 45-day-old metric is decoration — the default threshold states that assumption explicitly and is tunable per program.
• check_jumps() (lines 76-83) flags any 20-point coverage movement — deliberately low severity (real improvement produces jumps too), but it forces the provenance review that separates earned movement from definitional drift. The honest tool never assumes a jump is bad; it assumes the reader deserves the question.

Compilation and test procedure

monthly audit alongside the coverage cycle

$ python emureport.py —results atomic_results.json
—gaps gaps.csv —history history.json
—title “Q4 emulation cycle” —out report.md

Executive summary

• Validated coverage: 86% (43/50 executable checks; 6 gated)
• Trend: up 8.0% vs previous run (2026-09-01)
• Gap debt: 14 open gaps, 3 older than 90 days, 2 unowned
• Ask: (fill per cycle — usually backlog capacity or a specific fix)

$ python metricdrift.py —history history.json
—catalog catalog.json —out audit.md == metric integrity findings (1) == [MED] failed count unchanged (3) for 3 runs — regressions parked? [+] audit.md

the audit finding lands in the backlog like any other gap:

parked regressions get owners or explicit acceptance

Detection considerations

The reporting layer’s “detection” target is the program’s own decay — and the monitor’s findings feed the same backlog discipline as any detection gap, which is the cultural point: metric integrity is a workstream with tickets, not a virtue. Your own tooling hygiene: everything reads JSON/CSV and writes markdown — zero endpoint footprint. The report distribution discipline matters as much as computation: reports go out on a fixed cadence regardless of whether the news is good (a program that only reports wins trains its readers to distrust silence), exec briefs never exceed one page, and every number in the exec layer must trace to a table in the technical layer in one hop (the provenance chain readers spot-check). Wazuh/SIEM dashboards can render the trend lines [verify per platform], but the markdown report remains the artifact of record — dashboards are views, reports are commitments. The final standing metric worth defending in front of leadership: gap-closure MTTR — the median days from a gap’s first failing run to its first passing run. It is the one number that proves the whole loop (detect → decide → fix → validate) actually turns, and it cannot be gamed without fixing things, which is exactly what a program metric is for.

Improvements to add

1. Recipient-layer rendering: emit exec (one page), manager (matrix + debt), engineer (failing rows with procedures) as separate files from one run.
2. Framework cross-walk: map validated techniques to NIST CSF/ISO control IDs for audit evidence export [verify current mapping tables].
3. Regression attribution assist: join regression timestamps with a change-log CSV to auto-suggest the control change that caused each.
4. Debt-curve chart: render the gap-age histogram as ASCII in the markdown report so the debt shape is visible without a BI tool.
5. Audit scheduling: wire metricdrift into the same cron cycle as the coverage runs, filing findings as tickets automatically.

Detection — Defender View

Artifact	Audience	Cadence	Notes
Executive brief	Leadership	Quarterly	3 bullets + 1 ask, provenance-backed
Coverage matrix + trend	Managers	Monthly	Direction beats position
Failing rows w/ evidence	Detection eng	Per run	Telemetry citations attached
Regression register	Platform teams	Per regression	Change-correlated
Gap-debt table	Backlog meeting	Weekly-ish	Owner + age, unowned visible
Framework mapping export	Audit/Compliance	Quarterly	Control evidence w/ provenance
Metric integrity audit	Program lead	Monthly	Anti-gaming discipline
Flake report	Detection eng	Per run	Quarantine queue
Canary/impact latency metrics	SOC	Monthly	Last-line health (M8/T8)
History file	The next report	Continuous	Trend layer’s substrate

Evasion Techniques

Reporting-layer evasion is self-deception with numbers, and every pattern here is one the author has watched organizations perform with total sincerity. Denominator inflation: counting “tests run” instead of “techniques validated”, or growing the catalog with easy tests to dilute failures — countered by version-controlled catalogs and the denominator-drift audit. Gated suppression: parking hard techniques in gated status forever so coverage percentages never see them — countered by treating gated as a visible, owned state (T9’s rule) and reporting the gated count next to coverage. Circular calibration: tuning thresholds on drill data then reporting drill results against those thresholds (T8’s lesson) — countered by provenance requirements that baselines come from production telemetry. Snapshot cherry-picking: reporting the best quarter ever, once, instead of trends — countered by trend-first reporting and the history file that makes cherry-picking visible. Debt invisibility: closing gaps administratively (marking fixed without re-validation) or leaving them unowned — countered by the re-validation requirement (a gap closes when a test passes, not when a ticket is typed) and the unowned count’s prominence. Attribution laundering: attributing every regression to “environmental noise” — countered by the change-correlation work (improvement #3) that makes attribution a lookup rather than a judgment call. The meta-discipline is the honesty audit itself: metrics are a detection problem where the intrusion is self-serving bias, and the audit’s scheduled, checklist-driven review is the tripwire — because the most dangerous reporting failure is the one that makes everyone feel successful while the estate’s real coverage quietly rots.

⚠️ What caught me: I presented a coverage trend of 55% → 81% across two quarters, and it went over well — until one director asked what changed in the catalog between the two measurements. The answer was uncomfortable: we had added 30 easy techniques in Q3, so the denominator grew faster than our validated count, and the “improvement” was mostly arithmetic. Nothing was wrong with our work; something was wrong with my honesty about the number. The fix took one afternoon (denominator provenance in every report) and changed the culture permanently: our metrics now ship with their definitions, and any number that cannot explain its own denominator in one sentence doesn’t get presented. The director who caught it became the program’s biggest supporter — credibility survives honest correction far better than confident deception.

Lab Practice

1. Generate the full report chain from your T9 results: emureport.py with gaps and history — then read the executive summary as if you were the budget holder and mark every sentence that would survive a hostile question.
2. Build a synthetic gaming scenario: inflate the catalog with ten trivial tests and watch metricdrift.py’s denominator check flag it — the audit working, calibrated against a real attempt.
3. Run the gap-debt discipline for one cycle: assign every unowned gap an owner (or accept it explicitly in writing), then verify the next report’s unowned count drops to zero — the decision-making loop made visible.
4. Produce the three audience layers by hand from one report (exec brief, manager matrix, engineer rows) and have peers from each layer read their version — the comprehension test that finds where your reporting lies to itself.
5. Schedule the honesty audit: put metricdrift.py in the same cron as your coverage runs, file its first finding as a real ticket, and write the one-paragraph runbook that says what happens when the audit finds HIGH findings — the culture that keeps the metrics real.

Interview Questions

• Technical: Design the metric set for an emulation program — which five metrics, what denominators, and what provenance does each require?
• Technical: Why must gated tests be reported separately from coverage, and what gaming does merging them enable?
• Scenario: Your coverage jumped 20 points in one quarter and leadership wants to fund the program based on it. What do you verify before accepting the win?
• Conceptual: Why do trends outrank snapshots in security metrics, and what does a flat trend with high activity indicate?
• Conceptual: How does the gap-debt model (owner + age) change backlog behavior compared to a findings list?
• Deep-dive: Design the honesty audit: which drift patterns, what evidence does each check examine, and what happens when the audit flags the program lead’s own report?
• Practical: Leadership wants one number for the board. Which do you give, what one sentence defends it, and what caveat do you refuse to omit?

Related Modules

T1 (ATT&CK Mapping & Emulation Planning) · T9 (Atomic Testing & Automation) · T8 (Exfiltration & Impact Emulation) · BT7 (Threat Hunting) · F12 (Cyber Kill Chain & MITRE ATT&CK)