Detection Validation & Control Testing

Definition

Detection validation and control testing is the engineering practice of proving that each defensive control — a detection rule, an EDR policy, a blocklist, a hardening baseline — does what its owner claims, under conditions that resemble production. The method: execute a test case (a controlled, safe reproduction of the technique the control targets — the Atomic Red Team model), observe the control’s response through the same pipeline production uses, and score the result against the expectation declared in advance (PT1’s five-state vocabulary). Two families of claims get tested: detective controls (does the alert fire, how fast does it reach a human, does it survive to the SIEM — the full BT3 pipeline, not just the rule) and preventive controls (does the block actually block — the exploit fails, the macro doesn’t run, the unsigned driver is refused — with the block event itself logged). The craft’s center: a validation is only as good as its honesty conditions — telemetry verified before the test (otherwise a miss is ambiguous between “rule broken” and “pipeline blind”), expectations declared before execution (otherwise every result is explainable), and the production-path requirement (a rule that fires in the rule-tester but not through the real pipeline has validated nothing).

Why It Matters

Every defensive artifact in this handbook’s BT-series — every Sigma rule, every EDR policy, every blocklist — carries an implicit claim: “when X happens, I will do Y.” Unvalidated claims accumulate silently into a false sense of coverage: BT9’s two-corpus testing proves the rule logic against captured data, but only live validation proves the whole chain — the log source still shipping, the parsing still mapping fields, the rule still loading, the alert still routing, the analyst still seeing. Pipelines drift (a log source silently stops, a field rename breaks a parse, a rule gets disabled by a change ticket), and the drift is invisible until the technique that matters walks through the hole. Validation converts “we have coverage” from an assertion into a measurement, and — run continuously (PT4) — into a regression gate that catches drift within hours instead of at the next incident.

The failure modes are the classic validation traps. Testing the wrong layer: the atomic test on the analyst’s workstation fires in the EDR console and gets called validated — while the SIEM-side rule the SOC actually monitors is broken. The blind test: executing without verifying telemetry — a miss then means “rule broken” or “source down” with no way to tell. Expectation drift: the test “passes” because something fired — an unrelated noisy rule — and nobody checks it was the expected rule. Preventive-control optimism: the EDR blocks the payload in the test, therefore blocking works — without testing the bypass classes the adversary actually uses. And validation theater: testing only what already passes, so the green dashboard measures courage, not coverage.

Detailed Explanation

   TEST CASE (declared before execution)
   +-----------------------------------------------------+
   | technique: T1053.005        command: schtasks create|
   | tier: C (production-safe)   marker: purple-CY-Q4-x7 |
   | expected: rule SIG-0119 (expected-alert)            |
   | telemetry contract: winevent:4698 + sysmon:EID1     |
   | window: 15 min              repeat: 3x              |
   +--------------------------+--------------------------+
                              | execute (per tier gate)
                              v
   +-----------------------------------------------------+
   | OBSERVE THROUGH THE PRODUCTION PATH                 |
   | endpoint -> shipper -> parser -> SIEM -> rule ->    |
   | alert queue -> (analyst ack)  <-- every hop measured|
   +--------------------------+--------------------------+
                              v
   +-----------------------------------------------------+
   | CORRELATE (detectest)                               |
   | expected rule fired within window? -> PASS          |
   | fired late? -> PASS-with-latency (MTTD logged)      |
   | wrong rule fired? -> MISATTRIBUTION (fix routing)   |
   | nothing fired? -> blind-check telemetry first:      |
   |   contract present -> FAIL (rule broken)            |
   |   contract absent  -> BLOCKED (telemetry gap, BT3)  |
   +-----------------------------------------------------+

Test case design is half the craft. A good case is minimal (executes exactly the technique, no bonus behaviors that muddy attribution), safe (tier-gated; preventive-control tests especially need thought), marked (traceable end-to-end), and repeatable (three executions minimum — a single-pass detection could be luck; a rule that fires 2-of-3 is a different finding than 3-of-3). The test must not share infrastructure with real adversary tooling — the harness and the tradecraft stay separate so a compromised harness cannot masquerade as a test result.

Detective validation walks the production path. The rule firing in the vendor console is hop one; the alert appearing in the SIEM with correct field mappings is hop two; the case appearing in BT1’s queue with the right severity is hop three; the analyst acknowledging within the SLA is hop four. Each hop has a clock, and detectest scores the chain — because the leadership question (“would we catch it?”) is answered by the slowest hop, not the fastest. The most common silent failure is hop two: the rule fires, the parser drops the field the alert-title depends on, the alert arrives as unknown-host and the SOC ignores it — the rule works, the detection doesn’t.

Preventive validation tests the refusal. The control claims “blocks X”; the test executes X and observes: did the block event fire (the control’s own telemetry — EDR block events, firewall denies), did the attempted action fail (the technique’s effect absent — no scheduled task created, no process launched), and did the attempt generate detection (a block without an alert is silent prevention — acceptable, but it must be a declared expectation, not an accident). The bypass-class testing matters here: one refusal path proves one refusal path. Renamed binaries, LOLBin wrappers, and signed-component abuse are the standard second-wave tests for any blocking control.

Deep dive: the exclusion list — where validation goes to die

Every EDR and AV ships an exclusion mechanism (paths, processes, extensions the control ignores), and every enterprise accumulates them — performance fixes, compatibility waivers, developer exemptions. Each exclusion is, by construction, a region the validated control does not cover: the macro-block rule will never fire inside an excluded path, and the adversary who reads that layout inherits a free corridor. The discipline: exclusions live in a register with owner and justification; every exclusion gets its own test case (execute the technique inside the excluded path — confirm whatever compensating control watches that region actually fires); and the register is reviewed against the live agent config quarterly, because config drift adds exclusions nobody requested. An exclusion without a validated compensating control is not a performance optimization — it is a documented blind spot with an audit trail that proves you knew.

Common Techniques

Technique	What it does	Where it fails	Notes
Atomic test cases	Minimal, marked, repeatable reproductions	Bloated tests muddy attribution	Atomic Red Team model
Production-path scoring	Every hop measured (endpoint→SIEM→queue)	Vendor-console-only checks	The slowest hop answers
Blind-check gate	Telemetry verified before scoring a miss	Misses scored against dead pipes	detectest’s gate
Repeat consistency	3x execution, consistency scored	Single-pass luck	2-of-3 is a finding
Preventive refusal tests	Block event + effect-absent + attempt-alert	One bypass path tested	Second-wave bypass classes
Misattribution check	Correct rule fired, not just a rule	Anything-fires passes	Routing validation
Latency budgets	MTTD per hop vs declared budget	No budget declared	BT20 join
Control coverage join	Tests mapped to BT9 matrix states	Tests disconnected from matrix	Coverage honesty
Harness separation	Test infra ≠ tradecraft infra	Compromised-harness confusion	Security of the tooling
Gap declaration	Untestable controls declared honestly	Silent skips	Feeds BT3/BT20

Full Working Code

detectest — the validation harness: reads test-case definitions (CSV: test id, technique, command, expected_rule, tier, repeats, window_minutes) and the alerts export (CSV: rule, first_seen offsets from execution — any SIEM export normalized to this shape), correlates expected vs actual per test, applies the blind-check gate (a telemetry-presence column from the contract check), and emits per-test verdicts with MTTD — the validation record that feeds PT4’s regression suite and PT5’s reporting.

#!/usr/bin/env python3
"""detectest — correlate detection-test executions against alerts.

Inputs:
  tests.csv:  test_id,technique,expected_rule,tier,repeats,window_min,
              telemetry_ok (yes/no from the precheck gate)
  alerts.csv: test_id,rule,seconds_after_exec
    (one row per alert the pipeline produced during each test's window;
     seconds_after_exec is offset from that test's execution timestamp)
Verdicts: PASS / PASS-LATE / MISATTRIBUTION / FAIL / BLOCKED.
Exit 1 on any FAIL (CI gate for the validation cycle).
"""
import argparse
import csv
import sys
from collections import defaultdict
from pathlib import Path

REQUIRED_TESTS = ["test_id", "technique", "expected_rule", "tier",
                  "repeats", "window_min", "telemetry_ok"]
REQUIRED_ALERTS = ["test_id", "rule", "seconds_after_exec"]


def load(path, required, label):
    if not path.exists():
        sys.exit(f"fail-loud: {label} not found: {path}")
    with path.open(encoding="utf-8", errors="replace") as fh:
        reader = csv.DictReader(fh)
        missing = [c for c in required if c not in (reader.fieldnames or [])]
        if missing:
            sys.exit(f"fail-loud: {label} missing columns: {missing}")
        rows = list(reader)
    return rows


def main():
    ap = argparse.ArgumentParser(description="detection validation harness")
    ap.add_argument("--tests", type=Path, required=True)
    ap.add_argument("--alerts", type=Path, required=True)
    ap.add_argument("--out", type=Path, default=Path("detectest_out"))
    args = ap.parse_args()

    tests = load(args.tests, REQUIRED_TESTS, "tests")
    alerts = load(args.alerts, REQUIRED_ALERTS, "alerts")
    if not tests:
        sys.exit("fail-loud: zero test cases — nothing to validate is not "
                 "a validated estate")

    alerts_by_test = defaultdict(list)
    for a in alerts:
        try:
            off = float(a["seconds_after_exec"])
        except ValueError:
            sys.exit(f"fail-loud: alerts row non-numeric offset "
                     f"{a['seconds_after_exec']!r}")
        alerts_by_test[a["test_id"].strip()].append(
            (a["rule"].strip(), off))

    results, mttds = [], []
    for t in tests:
        tid = t["test_id"].strip()
        expected = t["expected_rule"].strip()
        try:
            repeats = int(t["repeats"])
            window = float(t["window_min"]) * 60
        except ValueError:
            sys.exit(f"fail-loud: {tid}: repeats/window not numeric")
        tel_ok = t["telemetry_ok"].strip().lower() == "yes"

        fired = alerts_by_test.get(tid, [])
        expected_hits = [off for rule, off in fired if rule == expected]
        other_hits = [(rule, off) for rule, off in fired if rule != expected]

        if not tel_ok:
            verdict, note = "BLOCKED", ("telemetry contract failed — "
                                        "miss is ambiguous (BT3 gap)")
        elif len(expected_hits) >= repeats:
            worst = max(expected_hits)
            if worst <= window:
                verdict = "PASS"
                mttds.append(sum(expected_hits) / len(expected_hits))
                note = f"fired {len(expected_hits)}/{repeats} within window"
            else:
                verdict = "PASS-LATE"
                mttds.append(worst)
                note = f"fired but slowest {worst:.0f}s > window {window:.0f}s"
        elif expected_hits:
            verdict = "FLAKY"
            mttds.append(sum(expected_hits) / len(expected_hits))
            note = (f"consistency {len(expected_hits)}/{repeats} — "
                    "rule or pipeline intermittent")
        elif other_hits:
            verdict = "MISATTRIBUTION"
            note = f"other rules fired: {other_hits[0][0]} — routing/expectation"
        else:
            verdict, note = "FAIL", ("expected rule never fired — "
                                     "rule or path broken")

        results.append({"test_id": tid, "technique": t["technique"].strip(),
                        "expected_rule": expected, "verdict": verdict,
                        "mttd_s": (f"{mttds[-1]:.0f}" if verdict in
                                   ("PASS", "PASS-LATE", "FLAKY") else "-"),
                        "note": note})

    results.sort(key=lambda r: ("BLOCKED", "FAIL", "MISATTRIBUTION",
                                "FLAKY", "PASS-LATE", "PASS").index(r["verdict"]))
    args.out.mkdir(parents=True, exist_ok=True)
    with (args.out / "validation_results.csv").open("w", newline="",
                                                    encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["test_id", "technique",
                                           "expected_rule", "verdict",
                                           "mttd_s", "note"])
        w.writeheader()
        w.writerows(results)

    counts = defaultdict(int)
    for r in results:
        counts[r["verdict"]] += 1
    total = len(results)
    tested = total - counts["BLOCKED"]
    rate = (counts["PASS"] + counts["PASS-LATE"]) / tested if tested else 0
    print(f"tests={total}  "
          f"PASS={counts['PASS']} PASS-LATE={counts['PASS-LATE']} "
          f"FLAKY={counts['FLAKY']} MISATTR={counts['MISATTRIBUTION']} "
          f"FAIL={counts['FAIL']} BLOCKED={counts['BLOCKED']}")
    if mttds:
        print(f"MTTD across passed: median "
          f"{sorted(mttds)[len(mttds)//2]:.0f}s  max {max(mttds):.0f}s")
    print(f"detection rate (tested only): {rate:.0%}  "
          f"({tested}/{total} testable)")
    for r in results:
        if r["verdict"] != "PASS":
            print(f"  [{r['verdict']:<14}] {r['test_id']} "
                  f"{r['technique']}: {r['note']}")
    print(f"results: {args.out / 'validation_results.csv'}")
    if counts["FAIL"] or counts["MISATTRIBUTION"]:
        sys.exit(1)
    print("validation cycle clean — fix queue empty of hard failures")


if __name__ == "__main__":
    main()

Code Walkthrough

The blind-check gate (lines 74-76) is the harness’s honesty anchor, and it runs first: a test whose telemetry contract failed precheck gets BLOCKED — never FAIL — because “the expected rule didn’t fire” against a dead pipeline is a statement about the pipeline, and recording it as a rule failure corrupts BT9’s matrix and BT20’s coverage number simultaneously. The verdict ladder (lines 78-99) then interprets the alert evidence strictly: repeats matched within the window is PASS (with the worst-offset checked against the window, line 83 — a rule that fired on execution 1 at 30s and never again does not pass on repeat-1’s evidence); consistent-but-slow is PASS-LATE (the detection works, the latency budget is the finding — BT20’s MTTD contract consumes it); partial consistency is FLAKY (its own verdict, because intermittent detections are operationally different from broken ones — the SOC loses trust in flaky rules even faster than absent ones); wrong-rule-only is MISATTRIBUTION (the technique did trip detection — just not the one owning it; a routing/expectation defect, not a rule defect); nothing is FAIL.

The exit discipline (lines 139-141) encodes the CI posture: FAIL and MISATTRIBUTION block the cycle — these mean the estate diverges from its claims. BLOCKED does not block (it’s a BT3 work item, not a detection defect) but it is subtracted from the detection-rate denominator (line 131 — tested = total - BLOCKED), so a cycle full of blind tests cannot flatter itself with an artificially high rate. The MTTD aggregation (lines 104-106) reports median and max across passed tests — median for the trend, max because the SOC’s SLA lives at the tail.

How It Reaches The Target

The monthly validation cycle runs against the lab and Tier-C production set: BT9’s rule inventory plus the preventive controls generate the test-case file; precheck verifies each case’s telemetry contract against BT3’s log-source health; execution week runs the cases (marked, tiered, three repeats); detectest correlates and scores. The output splits three ways. The estate’s truth: detection rate 74% of testable controls, median MTTD 38 seconds, two FLAKY rules, one MISATTRIBUTION (the T1059.001 test fired the generic script-block rule, not the family-specific one — the routing defect that buries real alerts in noise). The fix queue: the FLAKY pair gets pipeline-traced (a parser dropping events under load — BT3), the MISATTRIBUTION a routing fix, the BLOCKED technique its telemetry request. The regression seed: every PASS becomes a PT4 automated case — the next cycle re-runs it automatically, and drift (a rule disabled by a change ticket, a source that stops shipping) surfaces within one cycle instead of at the incident.

The production moment that justifies the whole apparatus: a real intrusion executes T1053.005 at 2 a.m. The rule that passed validation fires; the alert reaches the queue within the measured 38-second median; the analyst acknowledges inside the SLA — because every hop in that chain was measured, not assumed. And the intrusion that uses the bypass class the preventive tests covered (renamed binary) meets the second-wave control validated specifically against it. The estate’s claims and its reality converge — which is the entire product of this discipline, and the reason the next module (PT4) automates it into a gate that runs without anyone remembering to.

Exploitation Chain

1. Inventory — BT9 rule matrix + preventive control register produce the test-case backlog
2. Contract — precheck verifies telemetry per case; blind cases flagged before execution
3. Design — minimal, marked, repeatable (3x) cases per tier gate; preventive cases include bypass classes
4. Execute — T-series discipline: EMU- artifacts, markers, run log with timestamps
5. Correlate — detectest scores verdicts against expectations; MTTD per hop measured
6. Fix — FLAKY to pipeline trace, MISATTRIBUTION to routing, FAIL to rule authoring, BLOCKED to BT3
7. Re-test — fixes prove out; every PASS graduates into PT4’s automated regression suite
8. Report — detection rate (testable-only), MTTD median/max, fix queue state -> BT20 scorecard

Building Your Own Tool

The validation harness assumes telemetry; precheck is what makes the assumption testable. It reads the test plan’s telemetry requirements (per test: which sources must have delivered events in the last N hours) and the log-source health inventory (BT3’s export: source, last_event_age, daily_volume, status), verifies each contract, and flags every blind test before execution — the gate that keeps the coverage matrix honest.

Design decisions

Per-test contracts, not global health: a source can be healthy on average and dead for the specific host the test targets — the check runs against the narrowest scope the inventory supports. Age windows match the test: a contract requiring events within 4 hours is different from one requiring a daily batch; both are declared per test. Statuses are three-valued and loud: VERIFIED (contract met), STALE (source alive but older than the window — the pipeline may be fine, the data may be late; re-check before scoring), MISSING (no evidence the source exists at all — BT3 work item). Fail-loud on inventory schema drift, because a health export that half-parses is the exact silent failure this tool exists to catch.

Full source

#!/usr/bin/env python3
"""precheck — telemetry contract verification before detection testing.

Inputs:
  contracts.csv: test_id,source,scope,window_hours
    (source: inventory key; scope: host/zone qualifier or 'any';
     window_hours: max acceptable age of last event)
  inventory.csv: source,scope,last_event_hours,status
    (BT3 log-source health export)
Output: per-test verdicts (VERIFIED/STALE/MISSING) + go/no-go for the cycle."""
import argparse
import csv
import sys
from collections import defaultdict
from pathlib import Path

REQ_C = ["test_id", "source", "scope", "window_hours"]
REQ_I = ["source", "scope", "last_event_hours", "status"]


def main():
    ap = argparse.ArgumentParser(description="telemetry precheck")
    ap.add_argument("--contracts", type=Path, required=True)
    ap.add_argument("--inventory", type=Path, required=True)
    ap.add_argument("--out", type=Path, default=Path("precheck_out"))
    args = ap.parse_args()

    for path, req, label in ((args.contracts, REQ_C, "contracts"),
                             (args.inventory, REQ_I, "inventory")):
        if not path.exists():
            sys.exit(f"fail-loud: {label} not found: {path}")
    with args.contracts.open(encoding="utf-8") as fh:
        contracts = list(csv.DictReader(fh))
    with args.inventory.open(encoding="utf-8") as fh:
        inv_rows = list(csv.DictReader(fh))
    if not contracts:
        sys.exit("fail-loud: zero contracts — a test plan without telemetry "
                 "requirements is a blind plan")
    # inventory schema check
    for r in inv_rows:
        for c in REQ_I:
            if c not in r or r[c] is None:
                sys.exit(f"fail-loud: inventory missing column {c!r} — "
                         "schema drift breaks the gate")

    index = {}
    for r in inv_rows:
        try:
            age = float(r["last_event_hours"])
        except ValueError:
            sys.exit(f"fail-loud: inventory last_event_hours not numeric: "
                     f"{r['last_event_hours']!r}")
        index[(r["source"].strip().lower(),
               r["scope"].strip().lower())] = (age, r["status"].strip().lower())

    verdicts = []
    per_test = defaultdict(list)
    for c in contracts:
        tid = c["test_id"].strip()
        src = c["source"].strip().lower()
        scope = c["scope"].strip().lower()
        try:
            window = float(c["window_hours"])
        except ValueError:
            sys.exit(f"fail-loud: {tid}: window_hours not numeric")
        hit = index.get((src, scope))
        if hit is None and scope != "any":
            hit = index.get((src, "any"))       # fall back to unscoped entry
        if hit is None:
            v, note = "MISSING", f"no inventory entry for {src}/{scope}"
        else:
            age, status = hit
            if status in ("down", "disabled"):
                v, note = "MISSING", f"source status {status}"
            elif age <= window:
                v, note = "VERIFIED", f"last event {age:.1f}h <= {window:.0f}h"
            else:
                v, note = "STALE", (f"last event {age:.1f}h > window "
                                    f"{window:.0f}h — re-check before scoring")
        verdicts.append({"test_id": tid, "source": src, "scope": scope,
                         "verdict": v, "note": note})
        per_test[tid].append(v)

    args.out.mkdir(parents=True, exist_ok=True)
    with (args.out / "precheck_results.csv").open("w", newline="",
                                                  encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["test_id", "source", "scope",
                                           "verdict", "note"])
        w.writeheader()
        w.writerows(verdicts)

    print(f"contracts={len(verdicts)}  "
          f"VERIFIED={sum(1 for v in verdicts if v['verdict'] == 'VERIFIED')}  "
          f"STALE={sum(1 for v in verdicts if v['verdict'] == 'STALE')}  "
          f"MISSING={sum(1 for v in verdicts if v['verdict'] == 'MISSING')}")
    print("per-test readiness:")
    blocked = 0
    for tid in sorted(per_test):
        vs = per_test[tid]
        if "MISSING" in vs:
            go = "NO-GO  (blind test — fix telemetry or declare gap)"
            blocked += 1
        elif "STALE" in vs:
            go = "CAUTION (re-verify before scoring)"
        else:
            go = "GO"
        print(f"  {tid:<12} {'/'.join(vs):<28} {go}")
    print(f"results: {args.out / 'precheck_results.csv'}")
    if blocked:
        print(f"{blocked} test(s) blocked — set telemetry_ok=no for them in "
              "tests.csv so detectest scores them BLOCKED, not FAIL")
        sys.exit(1)


if __name__ == "__main__":
    main()

Walkthrough

The lookup cascade (lines 50-64) implements the narrowest-scope rule: exact (source, scope) first, falling back to the unscoped entry only when the scoped lookup misses — so a source that’s healthy fleet-wide but dead on the test host cannot pass a scoped contract. Status outranks age: a down/disabled source is MISSING regardless of its recorded age (the export may predate the outage), which is the inventory-vs-reality race this gate exists to catch. The three-valued verdict (VERIFIED/STALE/MISSING) refuses the binary trap: STALE means “re-verify at execution time” — data late is not data absent, and conflating them either blocks valid tests or green-lights blind ones.

The per-test rollup (lines 74-83) is what the executing team actually reads: NO-GO for any test with a MISSING contract (and the exit instructs exactly how to record it in the harness input — telemetry_ok=no — so detectest scores it BLOCKED rather than FAIL, keeping the two failure classes separated end-to-end), CAUTION for STALE, GO for clean. The exit code gates the cycle: a plan with blind tests either fixes its telemetry or explicitly declares the gaps — both are honest; executing anyway is the third option the tool exists to prevent.

Compilation and test procedure

author contracts from the test plan, pull BT3’s health export

$ head -3 contracts.csv test_id,source,scope,window_hours DT-0119,winevent:4698,host-lab-01,4 DT-0207,sysmon:EID1,host-lab-01,4

run the gate

$ python3 precheck.py —contracts contracts.csv —inventory health.csv contracts=14 VERIFIED=11 STALE=2 MISSING=1 per-test readiness: DT-0119 winevent:4698/any VERIFIED GO DT-0207 sysmon:EID1/any VERIFIED GO DT-0310 IdP-risk/any MISSING NO-GO (blind test — fix telemetry or declare gap) results: precheck_out/precheck_results.csv 1 test(s) blocked — set telemetry_ok=no for them in tests.csv so detectest scores them BLOCKED, not FAIL

the harness then scores blocked tests honestly

$ python3 detectest.py —tests tests.csv —alerts alerts.csv | head -3 tests=14 PASS=9 PASS-LATE=1 FLAKY=2 MISATTR=1 FAIL=0 BLOCKED=1 MTTD across passed: median 41s max 612s

Detection considerations

precheck’s verdicts are themselves detection telemetry: a source that flips VERIFIED → STALE between cycles without a corresponding change ticket is pipeline drift caught at the gate instead of at the incident — the delta feed belongs in BT3’s monitoring. The validation results join BT20’s scorecard (detection rate computed over testable controls only, with the BLOCKED count visible beside it), and the MISATTRIBUTION class deserves its own watch: a rising misattribution rate means alert routing is decaying — the SOC will start seeing “the wrong rule fired” as normal, which is how real alerts get mis-triaged. The regression seed (every PASS → PT4 case) is the compounding asset: coverage validated once is a claim; coverage re-validated weekly is a system.

Improvements to add

1. Auto-generate contracts from BT9’s rule metadata (each rule declares its sources) — the contract file becomes a build artifact, not a spreadsheet.
2. Add execution-time re-verification (re-query the inventory at test start, not just plan time) for STALE cases.
3. Correlate BLOCKED tests with BT3’s gap register automatically — one queue, one owner.
4. Track per-rule validation history and flag rules never re-validated in N cycles.
5. Emit the MTTD distribution (not just median/max) for BT20’s latency-contract tuning.

Detection — Defender View

Signal	Source	ID	Notes
Detection rate (testable)	detectest	per cycle	Honest denominator
MTTD median/max	detectest	per cycle	Latency contracts
FLAKY rule count	detectest	per cycle	Trust erosion
MISATTRIBUTION rate	detectest	per cycle	Routing decay
BLOCKED (blind) tests	precheck	per cycle	BT3 gap feed
Source VERIFIED→STALE flips	precheck delta	per cycle	Pipeline drift
Preventive refusal events	EDR/FW telemetry	per test	Block + alert pairs
Bypass-class pass rate	Second-wave tests	per control	Blocking truth
Regression suite size	PT4 registry	per cycle	Compounding asset
Validation coverage	BT9 matrix join	quarterly	Claims vs proof

Evasion Techniques

Adversaries evade validated controls by targeting what validation didn’t cover. The untested bypass class: validation proves the control refuses technique-as-tested; the adversary uses renamed binaries, LOLBin wrappers, or signed-component abuse — the second-wave test discipline (every blocking control gets its bypass-class tests) exists because first-wave passes are the adversary’s reconnaissance too. Environment mismatch: the control validated in the lab meets production’s different context (different agents, different exclusions — EDR exclusion lists are the classic divergence) — Tier C validation for the exclusion-sensitive controls catches what lab runs cannot. Timing evasion: detections validated at test tempo (minutes) miss the slow adversary (hours-days) — long-window validation cases for the beacon-class behaviors, and honesty about what can’t be compressed into a test window. Alert-routing blindness: the rule fires but into the misattributed queue (detectest’s MISATTRIBUTION class) — adversaries who study an org’s alert patterns (from a former employee, from publicized incident details) target the routing, not the rule. Validation fingerprinting: adversaries aware of an org’s validation cadence (test markers visible in threat intel sharing, or patterns learned from breaches) time operations between cycles — PT4’s continuous validation exists partly to shrink those windows to irrelevance. The excluded path: exclusions (AV/EDR folders, firewall exemptions for “trusted” tools) are unsegmented validation gaps by definition — each exclusion needs its own test case, because the adversary reads exclusion lists as an invitation.

⚠️ What caught me: we validated our EDR’s macro-blocking control thoroughly — test macro runs, block event, effect absent, alert fires. Green across the board, recorded as such. Six weeks later an incident walked in through an XLM macro in an .xls file — a form our test never covered, our control’s policy matched by extension and never fired, and the “validated” green had quietly meant “validated for VBA in .docm.” The fix was procedural, not technical: every control’s validation record now lists the exact variants tested (file formats, execution paths, user contexts), and anything not on the list is explicitly untested rather than implicitly covered. Validation proves the claim you wrote down — it has no obligation to cover the claim you assumed.

Lab Practice

Run a full validation cycle in the lab. Inventory: list your BT9 lab rules and one preventive control (a Defender ASR rule or AppLocker policy [verify per environment]). Contract: write precheck contracts for each test (which log sources must be alive); break one source deliberately (stop the shipper) and watch precheck flag it. Execute: run each case three times with markers — a T1053.005 scheduled task, a T1059.001 script block, and one preventive refusal. Correlate: run detectest; confirm verdicts match reality; deliberately point one test’s expected_rule at the wrong rule and observe MISATTRIBUTION fire. Fix: trace the FLAKY or FAIL to its actual hop (rule? parser? routing?), fix it, re-test, and record the MTTD improvement. Debrief: which hop in the production path was the slowest — and does that match where your real SOC’s latency actually lives?

Interview Questions

• Technical: Walk the verdict ladder — PASS, PASS-LATE, FLAKY, MISATTRIBUTION, FAIL, BLOCKED — and explain why BLOCKED is deliberately excluded from the detection-rate denominator.
• Technical: What does the blind-check gate prevent, and how does precheck’s three-valued verdict (VERIFIED/STALE/MISSING) differ from a binary health check?
• Scenario: A control validated green last month failed in a real incident through an untested variant. Walk the post-validation fix: what changes in the test design process, not just the control?
• Scenario: detectest reports MISATTRIBUTION on your highest-value rule — the technique fired a generic rule instead. Diagnose: is this a rule defect, a routing defect, or an expectation defect, and how do you tell?
• Conceptual: Why is a single-pass validation (one execution, one detection) weaker than a 3x-repeat protocol, and what does the 2-of-3 case tell you that 1-of-1 cannot?
• Deep-dive: Design the validation program for a 200-rule estate: test-case generation, telemetry gating, the execution cadence, preventive-control bypass testing, and the metrics for BT20.
• Practical: You have one week, one lab host, and 30 unvalidated rules. Which do you validate first, what do you explicitly defer, and how do you record the deferral?

Related Modules

PT1 (Purple Team Fundamentals & Program Design) · PT4 (Automation & Continuous Validation) · BT3 (SIEM Engineering & Log Pipelines) · BT9 (Detection Engineering & Sigma) · BT20 (Detection Program Metrics & Maturity)