Detection Program Metrics & Maturity

Definition

Detection program metrics and maturity is the measurement discipline that turns a security operation from a cost center with anecdotes into an engineering program with a trajectory: metrics that honestly describe speed (MTTD/MTTR), quality (validated coverage, false-positive rates), and breadth (telemetry, technique coverage, log-source health); and a maturity model that describes where each capability stands and what “next” looks like — assessed with evidence, not aspiration. The discipline’s hardest problem is not collecting numbers; it is choosing numbers that resist gaming, because every metric that becomes a target stops measuring (Goodhart’s law, operationalized in security orgs everywhere: report ticket SLAs hit 100% while real remediation slows, “coverage” counts rules while unvalidated ones rot). The craft’s center: a metric earns its place on the leadership dashboard only if it has three properties — a provenance (where the number comes from, mechanically), a behavior (what we will do differently when it moves), and a counterpart (the second metric that exposes gaming the first). Everything else is decoration. Maturity assessment is the same honesty applied structurally: each capability (triage, detection engineering, intel ops, response…) sits at a level with evidence attached, and the gap-to-target drives the roadmap — the output being a plan, not a score to feel good about.

Why It Matters

Programs without measurement drift: headcount decisions happen by anecdote, tool renewals happen by inertia, and the one incident that goes badly rewrites priorities that a decade of quiet competence should have set. With honest metrics, the conversation changes shape entirely: “our MTTD for credential-compromise patterns is 4.2 hours, trending down two quarters straight; our validated ATT&CK coverage is 41% with the gaps ranked by BT10’s threat picture; our P1 remediation on-time rate is 78% and the sladrift regression last quarter was the platform team’s re-org” — every clause is a decision input. The board hears trajectory, not vibes; the SOC sees its own work counted; and the next budget fight starts from a shared, defensible picture.

The failure modes are metric-shaped and everywhere. Vanity dashboards: green walls of “tickets closed” and “events processed” — activity, not effect. Unpaired metrics: SLA compliance without re-open rate (BT11’s lesson: fast-and-wrong reads as excellent). Coverage by rule count: 1,400 rules, 8% validated (BT9’s graveyard) — the denominator was a lie. Dwell-time folklore: citing industry medians (M-Trends publishes global median dwell time in the low double digits of days in recent editions [verify current edition]) as if it were your number, when your own MTTD is measurable and different. Maturity theater: an annual self-assessment scored 4/5 across the board with no evidence links, no gaps, and no roadmap — a document that exists to end conversations. And metric proliferation: forty dashboards nobody reads; if a number has no behavior attached (“when this drops, we do X”), it is not a metric, it is a screensaver.

Detailed Explanation

   SOURCES (each module's outputs, joined)          GOVERNANCE
   BT1  triage FP rate, backlog age                 +-------------------+
   BT3  log-source health, schema coverage   ----->  | METRIC CONTRACTS  |
   BT4  MTTD/MTTR per severity, RCA rate            | provenance +      |
   BT9  validated coverage, rule FP rates           | behavior +        |
   BT10 sighting feedback, PIR coverage        +--->| counterpart +     |
   BT11 SLA on-time, exception debt            |    | owner + cadence   |
   BT15 resistant factor coverage              |    +---------+---------+
   BT16/17 MTTD per detection family           |              |
   BT19  time-to-sitrep, cadence kept          |              v
        |                                      |    +-------------------+
        v                                      +<---| QUARTERLY SCORECARD|
   +-------------------+                            | + maturity profile |
   | METRIC PIPELINE   |                            | + gap-to-target    |
   | collect, compute, |                            | + roadmap          |
   | drift-check, sign |                            +-------------------+
   +-------------------+

Metric contracts are the unit of rigor. Each metric is written down with five fields: definition (the formula, unambiguous), provenance (the system and query that produces it — not a person’s spreadsheet), cadence (when it computes), behavior (the action its movement triggers), and counterpart (the metric that exposes its gaming). Example, written properly: validated ATT&CK coverage = techniques with at least one rule that passed both benign-corpus and ground-truth tests in the last 90 days ÷ techniques in scope (BT9’s five-state matrix supplies the status); provenance: the detection repo’s CI artifacts; counterpart: rule count by technique — if coverage claims rise while rule-test pass rates fall, someone is gaming the status field. A metric contract without a counterpart gets challenged at review; that’s the cultural mechanism that keeps the dashboard honest.

The speed pair — MTTD and MTTR — carries the most leadership weight and the most gaming risk. Definitions must be surgical: MTTD = time from first adversarial artifact in telemetry to analyst working the case (not “alert created” — alerts create themselves); MTTR = time from case open to containment verified, split from “case closed” (closure includes paperwork; containment is the security event). Both segment by severity and by detection family (BT16/17’s families), because an aggregate MTTD of 6 hours can hide phishing-at-20-minutes and insider-at-6-weeks — and the segments are where decisions live. Industry medians (M-Trends dwell-time figures) serve as orientation, never as targets: your number against your own trend is the only pair that means anything.

Maturity assessment follows the same evidence rule. Levels (0-5) per capability, each level anchored to observable artifacts: “detection engineering — level 3” means rules are version-controlled and CI-tested and you can show the repo and the pipeline, not that the team feels professional. NIST CSF 2.0’s functions (Govern, Identify, Protect, Detect, Respond, Recover) organize the map; MITRE ATT&CK Evaluations results [verify current-year participation] can calibrate externally. The output that matters is the gap-to-target per capability with owner and quarter — a maturity score without a roadmap is a yearbook photo.

Common Techniques

Technique	What it does	Where it fails	Notes
Metric contracts	Definition+provenance+behavior+counterpart	Written after the number	The core artifact
MTTD/MTTR surgical defs	First-artifact→working; open→contained	Alert-creation as detection	Segment by family/sev
Validated coverage %	Techniques with passing rules, 90d window	Status field gaming	BT9 matrix join
FP rate per source/rule	Triage verdicts joined to origin	Averages hide outliers	BT1 pipeline join
Paired-counterpart review	Every metric challenged with its pair	Pair skipped “for brevity”	Cultural enforcement
Drift detection on metrics	Sudden improvement = check the pipe	Good numbers never questioned	Fraud alarm, not joy
Maturity with evidence	Level claims anchored to artifacts	Self-scored aspiration	commatrix-style gates
Gap-to-target roadmap	Quarter, owner, capability	Roadmap without owners	The actual output
Log-source health index	BT3 pipeline as metric feed	Dead source = silent zero	Compute-fail ≠ zero
Goodhart audits	Quarterly: how would we game this?	Never asked	Adversarial self-review

Full Working Code

progmetrics — the quarterly scorecard engine: reads per-domain metric CSVs (the exports BT1/BT9/BT11/BT15… already produce), validates each metric against its contract (provenance present, plausible ranges, counterpart pairing), computes the scorecard with quarter-over-quarter drift flags, and — the distinctive feature — treats sudden improvement with the same suspicion as sudden degradation, because both mean the pipeline changed.

#!/usr/bin/env python3
"""progmetrics — quarterly detection-program scorecard.

Input: one metrics CSV per domain, two quarters each:
  metric,value  rows; filenames <domain>_<quarter>.csv  (e.g. triage_Q3.csv)
Built-in contracts define plausible ranges + required counterpart pairs.
Output: console scorecard + drift flags (suspicious in BOTH directions).
"""
import argparse
import csv
import sys
from pathlib import Path

# metric -> (min_plausible, max_plausible, counterpart, lower_is_better)
CONTRACTS = {
    "mttd_hours":       (0.1, 24 * 14, "mttr_hours", True),
    "mttr_hours":       (0.1, 24 * 30, "mttd_hours", True),
    "triage_fp_rate":   (0.0, 1.0, "validated_coverage", True),
    "validated_coverage": (0.0, 1.0, "triage_fp_rate", False),
    "sla_ontime":       (0.0, 1.0, "reopen_rate", False),
    "reopen_rate":      (0.0, 1.0, "sla_ontime", True),
    "resistant_coverage": (0.0, 1.0, "ca_exempt_count", False),
    "ca_exempt_count":  (0, 100000, "resistant_coverage", True),
    "logsource_health": (0.0, 1.0, "schema_coverage", False),
    "schema_coverage":  (0.0, 1.0, "logsource_health", False),
}


def load(domain, quarter):
    path = Path(f"{domain}_{quarter}.csv")
    if not path.exists():
        sys.exit(f"fail-loud: metrics file not found: {path}")
    out = {}
    with path.open(encoding="utf-8", errors="replace") as fh:
        for n, row in enumerate(csv.reader(fh), 1):
            if not row or row[0].strip().startswith("#"):
                continue
            if len(row) < 2:
                sys.exit(f"fail-loud: {path}:{n}: expected metric,value")
            metric, raw = row[0].strip(), row[1].strip()
            try:
                out[metric] = float(raw)
            except ValueError:
                sys.exit(f"fail-loud: {path}:{n}: {metric} value not numeric")
    if not out:
        sys.exit(f"fail-loud: {path}: zero metrics — pipeline broke, not calm")
    return out


def check_contract(domain, metric, value):
    """Return list of contract violations for one value."""
    issues = []
    if metric not in CONTRACTS:
        issues.append(f"{domain}/{metric}: no contract — metric without a "
                      "contract is a screensaver")
        return issues
    lo, hi, counterpart, _lib = CONTRACTS[metric]
    if not (lo <= value <= hi):
        issues.append(f"{domain}/{metric}: value {value} outside plausible "
                      f"[{lo}, {hi}] — provenance check, not celebration")
    return issues


def drift(prev, cur, metric):
    """Flag big moves either direction; sudden improvement can be gaming."""
    if prev is None or cur is None:
        return None
    if prev == 0:
        return None
    change = (cur - prev) / abs(prev)
    if abs(change) >= 0.30:
        direction = "IMPROVED" if (
            (cur < prev) == CONTRACTS[metric][3]) else "WORSENED"
        return f"{metric}: {prev} -> {cur} ({change:+.0%}) {direction} — verify provenance"
    return None


def main():
    ap = argparse.ArgumentParser(description="quarterly program scorecard")
    ap.add_argument("domains", nargs="+",
                    help="domain prefixes, e.g. triage detection identity")
    ap.add_argument("--q-cur", default="Q4")
    ap.add_argument("--q-prev", default="Q3")
    args = ap.parse_args()

    problems, lines = [], []
    for dom in args.domains:
        cur = load(dom, args.q_cur)
        try:
            prev = load(dom, args.q_prev)
        except SystemExit:
            prev = {}
        for m, v in cur.items():
            problems += check_contract(dom, m, v)
            d = drift(prev.get(m), v, m)
            if d:
                lines.append(f"  [{dom}] {d}")
        # counterpart pairing: both sides must exist
        for m in cur:
            if m in CONTRACTS:
                cp = CONTRACTS[m][2]
                if cp not in cur:
                    problems.append(f"{dom}/{m}: counterpart {cp} missing — "
                                    "unpaired metrics get gamed")

    print(f"SCORECARD {args.q_prev} -> {args.q_cur}  "
          f"domains={len(args.domains)}")
    for dom in args.domains:
        cur = load(dom, args.q_cur)
        for m in sorted(cur):
            print(f"  {dom:<12} {m:<20} {cur[m]:>12g}")
    if lines:
        print("\nDRIFT FLAGS:")
        for l in lines:
            print(l)
    if problems:
        print(f"\nCONTRACT PROBLEMS ({len(problems)}):")
        for p in problems:
            print("  -", p)
        sys.exit(1)
    print("\nall metrics contracted, paired, and in range — scorecard stands")


if __name__ == "__main__":
    main()

Code Walkthrough

The CONTRACTS table (lines 16-28) is the program’s measurement constitution, five fields per metric compressed into a tuple: plausible range (a “MTTD of 0” or a “coverage of 1.0 after a re-org” is a provenance alarm before it’s good news), the mandatory counterpart, and the polarity (lower_is_better) that drives drift interpretation. check_contract() (lines 40-50) does two things at once: range validation and the no-contract refusal — an unplugged metric gets flagged as “a screensaver” (line 46), which sounds flippant and is exactly the right review-room phrasing.

drift() (lines 52-62) is the module’s distinctive honesty: any move ≥30% flags in both directions, and the polarity determines whether it prints IMPROVED or WORSENED — but the instruction is identical: verify provenance. A coverage number that jumps 40% in a quarter is usually a definition change, a pipeline fix, or a gaming scheme; celebrating first and checking second is how dashboards learn to lie. main() enforces counterpart pairing per domain (lines 91-95): sla_ontime without reopen_rate in the same file is a contract problem, because BT11’s lesson generalizes — every speed metric needs its quality shadow. Exit discipline (lines 103-107): contract problems fail the scorecard, so the quarterly leadership deck is generated by the validation — the deck cannot present a number that skipped its contract, which is the whole point of provenance.

How It Reaches The Target

The scorecard assembles mechanically on the first Monday after quarter close: each domain’s exports drop (triage_Q4.csv from BT1’s pipeline, detection_Q4.csv from BT9’s CI artifacts, identity_Q4.csv from BT15’s coverage matrix, and so on), and progmetrics validates, pairs, drifts, and prints. The leadership meeting reads from that output — and because every number carries provenance, the meeting’s questions change from “is this right?” to “what do we do about it?”: MTTD phishing down 35% (provenance: the BT14 triage-pipeline change), validated coverage up but drift-flagged (turns out BT9 shipped 12 rules without the 90-day validation clock resetting — process gap, logged as an action), resistant-coverage flat at 61% (the FIDO2 rollout slipped — roadmap conversation, owner named).

The maturity assessment runs annually against the same honesty bar: each capability level claims its evidence links (the detection repo, the comms matrix gate, the exercise after-action reports), and maturitymap refuses unevidenced levels. The combined output — scorecard trend plus maturity gaps — is the program’s actual strategy document: three gaps per quarter, each with an owner and a metric that will move if the work lands. What reaches the adversary is the meta-effect: a program that knows its own speed per attack family invests where the adversary actually operates (the family with the worst MTTD gets the next quarter’s detection engineering), and every metric-gaming instinct the org has gets caught by its own counterpart review before the adversary benefits from the blind spot it would have left.

Exploitation Chain

1. Instrument — every module’s output becomes a metric feed (triage, detection, identity, cloud, comms)
2. Contract — each metric gets definition, provenance, behavior, counterpart, owner, cadence
3. Compute — quarterly scorecard assembled mechanically; validation gates the deck
4. Drift-check — big moves flagged both directions; provenance verified before celebration
5. Pair-review — counterparts challenged in the meeting; gaming schemes surface early
6. Assess — annual maturity with evidence links; unevidenced levels refused
7. Roadmap — gap-to-target with owners and quarters; three gaps per quarter, no more
8. Close the loop — next quarter’s scorecard measures whether the roadmap moved the metric

Building Your Own Tool

The scorecard says where you are; maturitymap says where each capability stands structurally and what the next level costs. It reads an assessment file (CSV: capability, level 0-5, evidence paths), validates that every claimed level has evidence on disk (the artifact files exist and are fresh), computes the profile against targets, and emits the gap-to-target register sorted by impact.

Design decisions

Evidence-or-refusal: a level claim without evidence paths — or with paths that don’t exist — is refused at level 0 (recorded as “unsubstantiated”), which converts assessment theater into a mechanical failure. Freshness windows: evidence older than 12 months demotes the claim one level (an assessment from three years ago is nostalgia). Targets are per-capability, not global — “everything at 4” is not a strategy; the target file encodes which capabilities the business actually needs at which level. Level anchors are documented per capability (the anchor table is the rubric reviewers share). Fail-loud on schema drift so a renamed column can’t zero the assessment.

assessment.csv # the annual input capability,level,evidence,target detection_engineering,3,“repo=https://git/det,ci=builds/1234”,4 intel_ops,2,“pirs=docs/pir.md,sightings=metrics/sight.csv”,3 crisis_comms,2,“matrix=comms_matrix.csv,drill=2026-09/tabletop.md”,3 triage,3,“backlog=metrics/backlog.csv,fp=metrics/fp.csv”,4

Full source

#!/usr/bin/env python3
"""maturitymap — evidence-validated capability maturity profile.

Usage: python3 maturitymap.py assessment.csv --targets targets.csv
Outputs: profile + gap-to-target register; refuses unevidenced levels."""
import argparse
import csv
import sys
import time
from pathlib import Path

LEVELS = (0, 1, 2, 3, 4, 5)
FRESH_DAYS = 365


def load_csv(path, need_cols, label):
    if not path.exists():
        sys.exit(f"fail-loud: {label} not found: {path}")
    with path.open(encoding="utf-8", errors="replace") as fh:
        reader = csv.DictReader(fh)
        missing = [c for c in need_cols if c not in (reader.fieldnames or [])]
        if missing:
            sys.exit(f"fail-loud: {label} missing columns: {missing}")
        return list(reader)


def evidence_fresh(spec: str):
    """Check every path=path entry in the evidence spec; return (ok, notes)."""
    notes, ok = [], True
    for part in spec.split(","):
        part = part.strip()
        if not part:
            continue
        if "=" not in part:
            ok = False
            notes.append(f"evidence entry without key=path form: {part!r}")
            continue
        key, val = part.split("=", 1)
        p = Path(val)
        if not p.exists():
            ok = False
            notes.append(f"{key}: evidence missing: {val}")
            continue
        age_days = (time.time() - p.stat().st_mtime) / 86400
        if age_days > FRESH_DAYS:
            ok = False
            notes.append(f"{key}: stale ({age_days:.0f}d): {val}")
    return ok, notes


def main():
    ap = argparse.ArgumentParser(description="evidence-gated maturity profile")
    ap.add_argument("assessment", type=Path)
    ap.add_argument("--targets", type=Path, required=True)
    ap.add_argument("--out", default="maturity_out")
    args = ap.parse_args()

    rows = load_csv(args.assessment,
                    ["capability", "level", "evidence", "target"], "assessment")
    targets = {r["capability"]: r for r in load_csv(
        args.targets, ["capability", "target", "rationale"], "targets")}

    profile, gaps = [], []
    for r in rows:
        cap = r["capability"].strip()
        try:
            claimed = int(r["level"])
        except ValueError:
            sys.exit(f"fail-loud: {cap}: level not an integer")
        if claimed not in LEVELS:
            sys.exit(f"fail-loud: {cap}: level {claimed} outside 0-5")
        ok, notes = evidence_fresh(r["evidence"])
        eff = claimed if ok else max(0, claimed - 1)
        if not ok:
            eff = 0 if any("missing" in n for n in notes) else eff
            profile.append((cap, claimed, eff, "; ".join(notes)))
        else:
            profile.append((cap, claimed, eff, "evidence ok"))
        tspec = targets.get(cap)
        if not tspec:
            gaps.append((cap, eff, None, "no target — capability without "
                         "a target is drift waiting to happen"))
            continue
        try:
            tgt = int(tspec["target"])
        except ValueError:
            sys.exit(f"fail-loud: target for {cap} not an integer")
        if eff < tgt:
            gaps.append((cap, eff, tgt, tspec["rationale"]))

    print(f"{'capability':<24}{'claimed':>8}{'effective':>10}  evidence")
    for cap, claimed, eff, note in sorted(profile):
        print(f"{cap:<24}{claimed:>8}{eff:>10}  {note}")

    if gaps:
        print(f"\nGAP-TO-TARGET ({len(gaps)}):")
        for cap, eff, tgt, why in gaps:
            if tgt is None:
                print(f"  {cap}: level {eff}, NO TARGET — {why}")
            else:
                print(f"  {cap}: level {eff} -> {tgt} — {why}")
    else:
        print("\nall capabilities at target — raise targets or defend the plateau")

    args_out = Path(args.out)
    args_out.mkdir(parents=True, exist_ok=True)
    with (args_out / "profile.csv").open("w", newline="",
                                         encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["capability", "claimed", "effective", "evidence_note"])
        w.writerows(sorted(profile))
    print(f"profile: {args_out / 'profile.csv'}")


if __name__ == "__main__":
    main()

Walkthrough

evidence_fresh() (lines 20-38) is the module’s honesty engine: it parses the evidence spec’s key=path entries, checks each path exists, and applies the freshness window (12 months — older evidence demotes). Two demotion paths exist deliberately: missing evidence zeroes the effective level outright (a claim with nothing behind it is level 0 — recorded as unsubstantiated, which in the review meeting is the sentence that ends maturity theater), while merely stale evidence costs one level (the capability was real once; prove it’s still running). The target join (lines 66-75) treats “no target” as its own finding — a capability with no target is where drift lives unwatched — and the gap register (lines 78-88) prints each gap with the target file’s rationale, so the roadmap conversation starts from why, not from a number.

The dual output (effective profile plus gap register, lines 96-105) mirrors progmetrics’ philosophy: the profile is the state, the gaps are the plan — and the plan lines carry owners through the rationale field rather than through vibes. Nothing here scores a business unit for comparison; the tool measures one program against its own declared needs, which is the only maturity comparison that survives a competent reviewer.

Compilation and test procedure

run the annual assessment

$ python3 maturitymap.py assessment.csv —targets targets.csv capability claimed effective evidence crisis_comms 2 2 evidence ok detection_engineering 3 3 evidence ok intel_ops 2 0 pirs: evidence missing: docs/pir.md triage 3 2 backlog: stale (400d): metrics/backlog.csv

GAP-TO-TARGET (4): detection_engineering: level 3 -> 4 — purple-team validation needed intel_ops: level 0 -> 3 — PIR process is the foundation for BT7/BT10 crisis_comms: level 2 -> 3 — comms matrix gate + quarterly drills triage: level 2 -> 4 — refresh backlog metrics, automate triage stats profile: maturity_out/profile.csv

an all-5s self-assessment with no evidence refuses to flatter

$ python3 maturitymap.py theater.csv —targets targets.csv | head -4 capability claimed effective evidence triage 5 0 backlog: evidence missing: …

Detection considerations

maturitymap output is program telemetry like any other: the delta between years is the signal (capabilities going backwards are re-org wounds — surface them while they’re cheap), and the evidence-freshness failures double as a documentation-health metric. The meta-lesson for the whole module: the same anti-gaming architecture runs everywhere — BT9’s two-corpus gate for rules, BT11’s re-open rate for SLAs, BT19’s cadence-kept for sits, and here evidence-freshness for maturity. A program is trustworthy when its own numbers have the counterparts that would catch its lying.

Improvements to add

1. Render the profile as a radar-ready JSON for the leadership deck (per-function NIST CSF 2.0 grouping).
2. Track evidence freshness as a metric itself and alert when the assessment’s foundations age out.
3. Join with progmetrics: capabilities whose metrics regressed get their level demoted automatically pending review.
4. Add a “cost to next level” column to the gap register (estimated effort per level anchor) for real roadmap math.
5. Diff mode: emit year-over-year deltas as the report’s first page, regressions before improvements.

Detection — Defender View

Signal	Source	ID	Notes
MTTD per family	BT4/BT16/BT17 joins	quarterly	First-artifact clock
MTTR to containment	BT4 pipeline	quarterly	Split from closure
Validated coverage %	BT9 matrix	quarterly	90-day validation window
Triage FP rate	BT1 verdicts	per month	Paired with coverage
SLA on-time × re-open	BT11 exports	quarterly	The gaming pair
Resistant factor coverage	BT15 matrix	quarterly	Privileged class focus
Log-source health	BT3 pipeline	weekly	Compute-fail ≠ zero
Intel sighting feedback	BT10	quarterly	Loop closure
Time to first sitrep	BT19 log	per incident	Comms speed
Contract violations	progmetrics gate	quarterly	Zero expected
Maturity evidence age	maturitymap	annual	Documentation health
Gap burn-down	Roadmap	per quarter	The actual strategy

Evasion Techniques

Metrics get evaded by the defenders — and the adversaries benefit from every evasion. SLA laundering: closing and re-opening tickets to reset clocks, re-classifying severities before deadlines, closing without verification — the counterpart pair (on-time × re-open) and the assignment-time SLA rule (sladrift’s design) exist for this. Coverage inflation: counting rules as coverage, counting dashboards as detection, counting purchases as capabilities — validated-coverage with a 90-day window and evidence-gated maturity are the antidotes. Definition drift: quietly redefining MTTD (“from alert-creation” instead of “from first artifact”) makes the number improve without the program changing — metric contracts with versioned definitions make drift visible, and the drift-detector treats suspicious improvement as loudly as regression. Denominator games: excluding the worst quarter, the hardest family, or the failed source from the average — scorecards must pin their scopes in the contract, and scope changes need a logged decision (BT19’s log discipline applies to dashboards). Selective instrumentation: measuring the defended castles and not the forgotten closets — log-source health (BT3) and coverage maps (BT16/17) are the metrics that expose measurement itself as the blind spot. The meta-evasion is cultural: when a metric becomes a target, teams optimize the number; the only durable defense is the counterpart review where “how would we game this?” is asked quarterly, in the open, by name.

⚠️ What caught me: our MTTD trend looked fantastic for three straight quarters — 9 hours down to 2 — and in the leadership meeting I presented it as program excellence, right up until the new hire asked which clock started the timer. We had quietly switched from “first adversarial artifact” to “alert created,” and a new auto-triage rule was creating alerts from a batch job — the improvement was a definition, not a capability. I re-derived the honest number (it had moved from 9 to 6, still good, not heroically good) and since then every metric definition change is a logged decision with an approver, and the scorecard prints the definition version next to the trend. Metrics don’t just need counterparts; they need changelogs — because the person most likely to game your dashboard is the one who built it, with the best intentions.

Lab Practice

Build the measurement layer for the lab program you’ve been running across this series. Collect: export three quarters of synthetic-or-real data from the modules’ outputs — BT1 triage verdicts (FP rate), BT9 rule tests (validated coverage), BT11 remediation records (SLA + re-open). Contract: write the metric contracts for each (definition, provenance, behavior, counterpart) — one page, no more. Score: run progmetrics across the domains; deliberately plant one gamed number (an MTTD computed from alert-creation) and one contract violation (an unpaired SLA metric) — confirm both get caught. Assess: write the assessment file for five capabilities with real evidence paths from your lab artifacts; run maturitymap; fix the two unevidenced claims honestly rather than laundering them. Roadmap: pick three gaps, assign owners (you, this month), and define what the Q+1 scorecard must show if the work landed. Debrief: which metric did you want to game while building this lab — and what does that tell you about the ones you didn’t build?

Interview Questions

• Technical: Walk the five fields of a metric contract and explain why the counterpart field is the one that makes the others enforceable.
• Technical: Why must MTTD start at “first adversarial artifact in telemetry” rather than “alert created”, and what implementation detail makes that measurable?
• Scenario: A sudden 40% improvement in validated coverage arrives the same week as a detection-team re-org. Walk your verification before anyone celebrates.
• Scenario: The board wants one number for “how good is our security.” Defend your choice of number, its contract, and what you refuse to compress into it.
• Conceptual: Explain Goodhart’s law with a detection-program example, and design the counterpart that catches that specific gaming.
• Deep-dive: Design the measurement system for a 60-person security org: metric families, contracts, the quarterly scorecard pipeline, maturity assessment with evidence, and the governance that keeps it honest.
• Practical: You inherit a program with 40 dashboards, no contracts, and a leadership deck built on vibes. What do you retire, what do you rebuild first, and what do you tell the board in month one?

Related Modules

BT1 (SOC Operations & Alert Triage) · BT9 (Detection Engineering & Sigma) · BT11 (Vulnerability Management) · BT19 (Crisis Communication & Leadership) · T10 (Emulation Reporting & Metrics)