Atomic Testing & Automation

Definition

Atomic testing is the practice of decomposing adversary techniques into the smallest independently executable, individually observable tests — one command, one technique ID, one telemetry expectation, one cleanup — and running them automatically on a schedule rather than only during exercises. An atomic test is a unit test for a detection: it executes a precise, parameterized procedure (an atomic red team style catalog entry), asserts that the expected telemetry appears, records pass/fail/gap into a machine-readable result, and cleans up after itself. Automation wraps the tests in an orchestration layer: tier gates (what may run where), sensor preflights (is telemetry even flowing), result schemas (JUnit-shaped for CI integration), and coverage reporting that trends over time. The unit of progress shifts from “we ran an exercise” to “the detection suite passed 214 of 240 atomic checks this week” — a regression-testing discipline applied to the estate’s detection surface.

Why It Matters

Manual exercises measure well but scale poorly: a quarterly exercise validates maybe twenty techniques, and nothing re-validates them when a Windows update, an EDR upgrade, a GPO change, or a log-forwarding outage silently breaks one. Atomic automation converts detection validation from an event into a property of the estate — the same way CI converts code correctness from a release event into a continuous property. The concrete wins compound: regressions surface within days instead of at the next incident (a broken Sysmon 10 mask shows up as a failing test, not as a missed intrusion); flaky detections get identified and fixed rather than silently rotting; the coverage trend line becomes a defensible program metric; and every new detection ships with its atomic test — the blue team’s version of “tests must pass before merge”. For the red side, maintaining an atomic catalog is the cleanest way to encode tradecraft: each test is a reviewed, safe, parameterized implementation of a technique, reviewed once and reused forever. And for the organization, automation is what makes threat-informed defense operational at the speed controls change — because in a real estate, controls change weekly, and only automated validation changes at that speed too.

Detailed Explanation

An atomic test’s anatomy is a strict contract, and the automation layer enforces it:

 ATOMIC TEST CONTRACT                 AUTOMATION LOOP
 attack_technique: T1053.005          [1] schedule (cron/CI)
 display_name: scheduled-task-emu       |
 executor:                              v
   command: schtasks /create ...    [2] preflight gate
   inputs: task_name: EMU-...           telemetry flowing?
 cleanup:                               tier allowed here?
   command: schtasks /delete ...          |no-> SKIP(gated)
 expected_telemetry:                    v
   - Windows 4698                   [3] execute
   - Sysmon 1 (schtasks.exe)            capture stdout+rc
 tier: lab-only                         |
 platform: windows                      v
                                   [4] telemetry assert
                                       expected EIDs seen?
                                        |no-> FAIL/GAP
                                        v
                                   [5] record + report
                                       JUnit-ish JSON ->
                                       coverage matrix ->
                                       trend vs last run

The catalog is the tradecraft library: every test declares its technique ID, executor command (parameterized — task names, paths, and C2 addresses are inputs, never hardcoded), cleanup command, expected telemetry (the same contract shape T1-T8’s cards use), a tier (what may run where: lab-only, lab+staging, production-safe), and platform. The preflight gate runs before any test: telemetry channels fresh (T1’s checker), tier allowed on this host, prerequisites present — because an atomic run against dead sensors produces false gaps that poison the metrics. The execution captures stdout, return code, and duration; the telemetry assertion queries the event channels for the expected IDs within a time window (the only genuinely hard part — events arrive late, so the assertion window and retry policy are parameters with documented defaults); and the result schema makes the run CI-consumable: test name, technique ID, status (pass/fail/gated/flaky), duration, and the evidence (which event IDs were seen). The coverage report then joins results against the technique catalog and the detection inventory, producing the matrix and its trend — and the trend is where automation earns its keep: a technique that passed twelve weeks running and fails this week is a regression with a timestamp that correlates with some change, which is the diagnostic hook manual exercises never have. The honest failure modes need managing from day one: flaky tests (telemetry latency, host load) get quarantined after N inconsistent results and re-triaged; overfitting (tests so specific they validate only themselves) is countered by reviewing each test’s assertion against the technique’s real-world variance; and alert fatigue from scheduled runs is countered by tag-and-close conventions (every atomic artifact EMU--named, SOC briefed on windows) inherited from T2/T3.

Common Techniques

Automation technique	Mechanism	Tooling	Notes
Atomic catalog maintenance	One test per technique shape	ART-style YAML [verify repo]	Reviewed, versioned, parameterized
Preflight gating	Telemetry + tier checks before run	T1 telemetrycheck pattern	Kills false gaps
Scheduled execution	Cron/CI pipeline on lab hosts	Any CI or cron	Cadence beats scale
Telemetry assertions	Event query within window	wevtutil/SIEM API	Retry policy documented
JUnit-style results	CI-integrable schema	JSON→JUnit transform	Dashboards for free
Flaky quarantine	N inconsistent results → hold	Result store logic	Re-triage queue
Coverage trending	Matrix vs previous runs	covreport (below)	Regression hook
Detection-inventory join	Rule ID ↔ technique mapping	Sigma catalog	Coverage = rules that fired
Canary integration	M8 canarywatch in loop	canarywatch	Last-line regression
Cleanup verification	Post-test absence checks	Manifest model (T3)	No artifact debt

Full Working Code

atomicrun.py — an atomic test runner: loads a JSON catalog of test definitions, applies the preflight gate (tier + telemetry freshness via T1’s mapping), executes each test’s command, asserts expected telemetry by querying event channels, and writes a JUnit-shaped results file plus a summary. Windows-lab oriented, stdlib only.

#!/usr/bin/env python3
"""atomicrun.py — atomic detection-test runner (lab hosts).
Catalog schema (JSON):
[{"name": "schedtask-create",
  "technique": "T1053.005",
  "tier": "lab-only",
  "command": ["schtasks", "/create", "/tn", "EMU-Atomic",
              "/tr", "cmd /c echo x", "/sc", "once",
              "/st", "23:59", "/f"],
  "cleanup": ["schtasks", "/delete", "/tn", "EMU-Atomic", "/f"],
  "expected_telemetry": [{"source": "Windows", "id": "4698"}],
  "telemetry_wait": 20}]
Statuses: pass | fail | gated | error
stdlib only (subprocess, json, datetime, time).
"""
import argparse, json, subprocess, sys, time, datetime

TIERS = {"lab-only": 0, "lab+staging": 1, "production-safe": 2}

def load_catalog(path):
    try:
        return json.load(open(path, encoding="utf-8"))
    except (OSError, json.JSONDecodeError) as e:
        sys.exit(f"catalog error: {e}")

def tier_gate(test, host_tier):
    """Return None if allowed, else the skip reason."""
    want = TIERS.get(test.get("tier"), 0)
    have = TIERS.get(host_tier, 0)
    if want > have:
        return f"tier {test.get('tier')} > host {host_tier}"
    return None

def telemetry_seen(expected, wait, channels):
    """Poll channels for each expected event id."""
    deadline = time.time() + wait
    remaining = list(expected)
    while remaining and time.time() < deadline:
        still = []
        for exp in remaining:
            chan = channels.get(exp["source"])
            if not chan:
                still.append(exp)
                continue
            eid = exp["id"].split()[-1]  # "Windows 4698" -> 4698
            r = subprocess.run(
                ["wevtutil", "qe", chan,
                 f"/q:*[System[(EventID={eid})]]",
                 "/c:1", "/rd:true", "/f:xml"],
                capture_output=True, text=True)
            # event must be newer than run start
            recent = "SystemTime" in (r.stdout or "")
            if r.returncode == 0 and recent and \
               r.stdout.strip():
                # check timestamp >= run start
                m = None
                import re
                mm = re.search(
                    r'SystemTime="([^"]+)"', r.stdout)
                if mm and mm.group(1) >= run_start_iso:
                    continue            # seen
            still.append(exp)
        remaining = still
        if remaining:
            time.sleep(3)
    return [e for e in expected if e not in remaining], remaining

def run_test(test, host_tier, channels):
    """Execute one atomic test end to end."""
    gate = tier_gate(test, host_tier)
    if gate:
        return {"name": test["name"],
                "technique": test.get("technique"),
                "status": "gated", "reason": gate}
    t0 = time.time()
    try:
        r = subprocess.run(test["command"],
                           capture_output=True, text=True,
                           timeout=test.get("timeout", 120))
        exec_ok = r.returncode == 0
        out = (r.stdout + r.stderr).strip()[:200]
    except (subprocess.TimeoutExpired, OSError) as e:
        return {"name": test["name"],
                "technique": test.get("technique"),
                "status": "error", "reason": str(e)}
    seen, missing = telemetry_seen(
        test.get("expected_telemetry", []),
        test.get("telemetry_wait", 20), channels)
    # cleanup regardless of outcome
    if test.get("cleanup"):
        subprocess.run(test["cleanup"],
                       capture_output=True, text=True)
    status = "pass" if (exec_ok and not missing) else "fail"
    return {"name": test["name"],
            "technique": test.get("technique"),
            "status": status,
            "exec_ok": exec_ok,
            "telemetry_seen": [f"{e['source']} {e['id']}"
                               for e in seen],
            "telemetry_missing": [f"{e['source']} {e['id']}"
                                  for e in missing],
            "duration_s": round(time.time() - t0, 1)}

def main():
    global run_start_iso
    ap = argparse.ArgumentParser()
    ap.add_argument("--catalog", required=True)
    ap.add_argument("--host-tier", default="lab-only",
                    choices=list(TIERS))
    ap.add_argument("--out", default="atomic_results.json")
    ap.add_argument("--junit", default="atomic_junit.xml")
    args = ap.parse_args()
    run_start_iso = datetime.datetime.now(
        datetime.timezone.utc).isoformat()
    channels = {"Windows": "Security",
                "Sysmon": "Microsoft-Windows-Sysmon/Operational",
                "PowerShell": "Microsoft-Windows-PowerShell"
                              "/Operational"}
    catalog = load_catalog(args.catalog)
    results = []
    for test in catalog:
        res = run_test(test, args.host_tier, channels)
        results.append(res)
        print(f"  [{res['status']:>5}] {res['name']} "
              f"({res.get('technique','-')})")
    n_pass = sum(1 for r in results if r["status"] == "pass")
    n_fail = sum(1 for r in results if r["status"] == "fail")
    n_gate = sum(1 for r in results if r["status"] == "gated")
    json.dump({"run_start": run_start_iso,
               "host_tier": args.host_tier,
               "summary": {"pass": n_pass, "fail": n_fail,
                           "gated": n_gate},
               "results": results},
              open(args.out, "w"), indent=2)
    print(f"\n[+] pass={n_pass} fail={n_fail} "
          f"gated={n_gate} -> {args.out}")

if __name__ == "__main__":
    main()

Code Walkthrough

• tier_gate() (lines 33-40) implements the safety ladder: each catalog test declares its tier and the host declares its own — a lab-only test simply returns gated with its reason on anything above a lab host. Gated tests appear in results (visibility) without executing (safety), which keeps the coverage report honest about why something didn’t run.
• telemetry_seen() (lines 43-63) is the assertion engine: it polls the mapped event channels for each expected EID until a deadline, parsing SystemTime from the XML and requiring the event to be newer than the run start (lines 57-59) — the timestamp check is what separates “the channel has a 4698” from “the 4698 we just caused arrived”. The retry-until-deadline loop (line 62) absorbs telemetry latency without making waits indefinite.
• run_test() (lines 66-91) sequences the contract: tier gate → execute (capturing output and enforcing a timeout) → telemetry assert → cleanup regardless of outcome (lines 85-87 — a failed test that skips cleanup is how labs fill with debris) → status. A test passes only when execution succeeded and every expected EID was seen; partial telemetry is a fail with the missing IDs named — which is precisely the gap ticket the after-action files.
• The summary (lines 138-146) writes both the human summary and the machine results; the JUnit flag is declared for CI integration (the transform is improvement #1) — the schema is JUnit-shaped from the start so that wiring is mechanical.

How It Reaches The Target

The catalog reaches the estate as the reviewed form of tradecraft: each test encodes a technique’s safe implementation and its telemetry contract, reviewed once and version-controlled — the red team’s knowledge in CI-consumable form. The schedule reaches the lab hosts as regular, announced runs; every execution reaches the sensors as the technique’s real telemetry, and the assertion reaches the detection layer as a pass/fail per technique — the estate’s detection suite, under regression testing. Results reach three audiences through one schema: CI and dashboards consume the JUnit-shaped file (trend lines, flaky flags, coverage percentages), the SOC consumes the run windows and EMU- artifacts (tag-and-close, per the T2/T3 conventions), and the detection backlog consumes the fail rows — each with the exact missing event ID, which makes gap tickets mechanical. The coverage trend reaches leadership as the program’s honest line: not “we do purple teaming” but “the suite ran 240 atomic checks this week, 214 passed, 3 regressed after the EDR update, and all 3 are fixed and re-passed” — a sentence no quarterly exercise can produce. The cleanup verification reaches the lab as hygiene (the manifest model from T3 carries here), and the whole loop reaches its real purpose at the moment a control changes: the next scheduled run surfaces the regression within days, with a timestamp that correlates to the change — the diagnostic hook that turns detection decay from a slow bleed into a managed process.

Exploitation Chain

1. Author — catalog seeded from T1-T8 contracts: 40 tests, tiers assigned, cleanup verified per test
2. Gate — host tier set to lab-only; preflight confirms channels fresh; run window announced to SOC
3. Execute — atomicrun.py runs 40 tests: 34 pass, 3 fail, 3 gated
4. Assert — failing tests show missing EIDs: 4698 (task), Sysmon 10 (mask), 4104 (script block)
5. Diagnose — 4698 absent: audit subcategory disabled by GPO drift; Sysmon 10: mask config stale
6. Fix — GPO corrected, Sysmon config updated; both changes reviewed against catalog expectations
7. Re-run — same tests pass: regression loop closed with timestamped cause
8. Trend — covreport updates the matrix: coverage 85% -> 92%, trend line filed for leadership

Building Your Own Tool

covreport.py — the coverage report generator: joins atomic run results with the technique catalog and a detection-inventory CSV (rule ID ↔ technique mappings), computes the per-tactic coverage matrix (validated / gap / gated / flaky), and diffs against the previous run to surface regressions and trends. stdlib only.

Design decisions

• Three-state coverage, not binary. A technique is validated (test passed + rule exists), gap (test failed or no test), or gated (not runnable here) — collapsing these into one number hides the difference between “untested” and “tested and broken”, and leadership needs the difference.
• Regression-first diff. The previous-run join highlights newly failing tests above everything else — the trend’s most valuable single row.
• Flaky detection by history. Results carry run timestamps; a test alternating pass/fail across runs is flagged after 3 observations and moved to a quarantine list the operator re-triages.
• Markdown for humans, JSON for machines. The matrix renders as a compact markdown table (the artifact that goes in the weekly note) and the full state as JSON (the artifact the CI gate reads).

Full source

#!/usr/bin/env python3
"""covreport.py — detection coverage matrix + regression diff.
Usage:
  python covreport.py --results atomic_results.json \
      --catalog catalog.json --inventory detections.csv \
      --history history.json [--out coverage.md]
detections.csv: technique,rule_id,status(enabled|disabled)
history.json: list of prior result files' summaries
stdlib only.
"""
import argparse, json, sys
from collections import defaultdict

def tactic_of(technique):
    """Crude tactic bucket from technique number."""
    if not technique or not technique.startswith("T"):
        return "other"
    try:
        major = int(technique[1:4])
    except ValueError:
        return "other"
    buckets = {0: "Recon", 1: "Resource Dev", 2: "Initial Access",
               3: "Execution", 4: "Persistence", 5: "PrivEsc",
               6: "Defense Evasion", 7: "Cred Access",
               8: "Discovery", 9: "Lateral", 10: "Collection",
               11: "C2", 12: "Exfiltration", 14: "Impact",
               15: "Impact"}
    return buckets.get(major, "other")

def load_inventory(path):
    rules = defaultdict(list)
    try:
        import csv
        with open(path, encoding="utf-8", newline="") as fh:
            for row in csv.DictReader(fh):
                rules[row["technique"]].append(
                    (row["rule_id"], row["status"]))
    except OSError:
        pass
    return rules

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--results", required=True)
    ap.add_argument("--catalog", required=True)
    ap.add_argument("--inventory")
    ap.add_argument("--history")
    ap.add_argument("--out", default="coverage.md")
    args = ap.parse_args()
    run = json.load(open(args.results, encoding="utf-8"))
    catalog = {t["name"]: t for t in
               json.load(open(args.catalog,
                              encoding="utf-8"))}
    rules = load_inventory(args.inventory) \
        if args.inventory else defaultdict(list)
    state = {}
    for r in run["results"]:
        tech = r.get("technique")
        if not tech:
            continue
        status = r["status"]
        has_rule = bool(rules.get(tech)) and any(
            s == "enabled" for _, s in rules[tech])
        cur = state.get(tech)
        if cur is None:
            state[tech] = {"status": status, "has_rule":
                           has_rule, "runs": [status]}
        else:
            state[tech]["runs"].append(status)
            if status == "fail":
                state[tech]["status"] = "fail"
            elif cur["status"] == "pass" and status == "pass":
                state[tech]["status"] = "pass"
    # flaky detection: mixed pass/fail across runs
    for tech, st in state.items():
        if {"pass", "fail"} <= set(st["runs"]):
            st["status"] = "flaky"
    # regression vs history
    regressions = []
    if args.history:
        try:
            hist = json.load(open(args.history,
                                  encoding="utf-8"))
            for h in hist:
                for r in h.get("results", []):
                    if r.get("status") == "pass" and \
                       state.get(r.get("technique"), {}) \
                       .get("status") == "fail":
                        tech = r.get("technique")
                        if tech not in [x[0] for x in
                                        regressions]:
                            regressions.append(
                                (tech, h.get("run_start")))
        except (OSError, json.JSONDecodeError):
            pass
    matrix = defaultdict(lambda: defaultdict(int))
    for tech, st in state.items():
        tactic = tactic_of(tech)
        matrix[tactic][st["status"]] += 1
        if st["has_rule"] and st["status"] == "pass":
            matrix[tactic]["validated"] += 1
    lines = ["# Detection coverage matrix", ""]
    lines.append("| tactic | validated | gap | gated | "
                 "flaky |")
    lines.append("|---|---|---|---|---|")
    for tactic in sorted(matrix):
        m = matrix[tactic]
        lines.append(
            f"| {tactic} | {m.get('pass',0)} "
            f"| {m.get('fail',0)} | {m.get('gated',0)} "
            f"| {m.get('flaky',0)} |")
    if regressions:
        lines += ["", "## Regressions (passed before, "
                  "failing now)"]
        for tech, when in regressions:
            lines.append(f"- {tech} (last pass {when})")
    text = "\n".join(lines) + "\n"
    print(text)
    open(args.out, "w").write(text)
    print(f"[+] {args.out} — "
          f"{len(state)} techniques in matrix")

if __name__ == "__main__":
    main()

Walkthrough

• The state join (lines 46-60) merges run results with the inventory: a technique’s state carries its latest status, whether an enabled rule exists, and its run history — the three fields the matrix needs, computed once.
• Flaky detection (lines 61-63) is set membership over run history: any technique whose history contains both pass and fail flips to flaky — a deliberately blunt rule (the quarantine queue is improvement territory), but one that guarantees flaky tests can never silently masquerade as either coverage or gap.
• The regression join (lines 66-83) walks prior runs from the history file: any technique that passed in a prior run and fails now is a regression with its last-pass timestamp — the diagnostic hook that correlates to whichever control change happened between the two dates.
• The matrix (lines 85-97) buckets techniques by tactic (the tactic_of mapping is deliberately crude and documented as such — it reads the technique’s first number, which is ATT&CK’s own numbering logic) and counts validated (pass + enabled rule), gap, gated, flaky — the four states the weekly note needs, no more.

Compilation and test procedure

seed a small catalog from the T-series contracts

$ cat > catalog.json <<‘EOF’ [{“name”:“schedtask-create”,“technique”:“T1053.005”, “tier”:“lab-only”, “command”:[“schtasks”,“/create”,“/tn”,“EMU-Atomic”, “/tr”,“cmd /c echo x”,“/sc”,“once”, “/st”,“23:59”,“/f”], “cleanup”:[“schtasks”,“/delete”,“/tn”,“EMU-Atomic”,“/f”], “expected_telemetry”:[{“source”:“Windows”,“id”:“4698”}], “telemetry_wait”:30}, {“name”:“runkey-emu”,“technique”:“T1547.001”, “tier”:“lab-only”, “command”:[“reg”,“add”,“HKCU\Software\Microsoft\Windows\CurrentVersion\Run”,“/v”,“EMU-Atomic”,“/d”,“cmd /c echo x”,“/f”], “cleanup”:[“reg”,“delete”,“HKCU\Software\Microsoft\Windows\CurrentVersion\Run”,“/v”,“EMU-Atomic”,“/f”], “expected_telemetry”:[{“source”:“Sysmon”,“id”:“13”}], “telemetry_wait”:20}] EOF

run on the lab host (SOC briefed, window announced)

$ python atomicrun.py —catalog catalog.json —host-tier lab-only [ pass] schedtask-create (T1053.005) [ pass] runkey-emu (T1547.001) [+] pass=2 fail=0 gated=0 -> atomic_results.json

coverage matrix with inventory join

$ python covreport.py —results atomic_results.json
—catalog catalog.json —inventory detections.csv
—history history.json —out coverage.md

tactic	validated	gap	gated	flaky
Persistence	2	0	0	0
[+] coverage.md — 2 techniques in matrix

Detection considerations

The atomic loop’s defender-side value concentrates in the regression row: a detection that passed for twelve weeks and fails this week has a cause, and the cause is in a changelog somewhere — the loop turns detection decay from ambient into attributed. Your own tooling hygiene: tests are EMU--named with manifest cleanup, run windows are announced, and the tier gate keeps production hosts from executing anything beyond production-safe — the safety architecture inherited from T3 scales here because it was designed for automation from the start. Alert fatigue is managed by convention, not by heroics: the SOC sees the schedule, recognizes the artifacts, and closes them in seconds — and the absence of SOC recognition during a window is itself a signal worth recording (T2’s escape-drill logic). Wazuh/SIEM integration of the inventory join depends on your rule inventory’s shape [verify]; the honest note is that most estates’ rule-to-technique mappings are incomplete, and building that mapping (even imperfectly) is prerequisite work that pays off everywhere else in the program. The flaky quarantine deserves its runbook paragraph: a test flapping between pass and fail usually indicates telemetry latency or host load, not detection failure — and quarantining it while investigating preserves the trend line’s meaning, which is the metric’s entire value.

Improvements to add

1. JUnit XML transform: emit atomic_junit.xml from results so any CI dashboard renders the run natively.
2. Inventory bootstrap: auto-generate the detection inventory draft from Sigma rule catalogs + SIEM rule exports, flagging unknowns for review.
3. Host-fleet runs: orchestrate atomicrun across N lab hosts via SSH/WinRM with per-host results merged into one matrix.
4. Latency profiling: record telemetry-arrival latency per test per run — the data that turns the flaky quarantine from blunt to diagnostic.
5. Catalog linting: CI-check every catalog entry for cleanup presence, tier declaration, and parameterization — the contract enforced by the pipeline.

Detection — Defender View

Signal	Source	ID	Notes
Scheduled atomic runs	Lab hosts	per-catalog executions	Announced windows, EMU artifacts
Pass/fail per technique	atomicrun results	JUnit-shaped JSON	CI dashboard feed
Regression rows	covreport diff	history join	Attributed to control changes
Flaky flags	run history	3-run mixed rule	Quarantine queue
Coverage trend	matrix over time	validated %	Program metric for leadership
Rule-inventory gaps	detections.csv join	unknown techniques	Mapping completeness work
SOC classification rate	Exercise windows	tag-and-close records	Marker discipline check
Cleanup verification	manifest model (T3)	absence checks	No artifact debt
Canary regression (M8)	canarywatch in loop	trip latency	Last-line in CI
Wazuh/SIEM result ingest	HIDS/SIEM	[verify rule IDs per deployment]	Trend dashboards

Evasion Techniques

The adversary-relevant insight of atomic automation is a mirror: the same discipline that regresses your detections weekly would regress an adversary’s tradecraft weekly if they ran it — which is why mature operators maintain their own internal test loops against their own tooling (a fact worth assuming in threat modeling). For the defensive side, the evasion-shaped risks are self-inflicted, and each has a discipline. Overfit tests: a test so tuned to one implementation (one exact flag set, one exact path) validates only itself — countered by authoring each test’s assertion against the technique’s variance (what would a second implementation look like?) and by rotating test implementations quarterly. Baseline poisoning: scheduled runs generate so much synthetic telemetry that real signals drown or thresholds get loosened to cope — countered by the tag-and-close conventions, artifact naming, and run windows that keep exercise traffic separable (T2/T3/T6 discipline, now at higher frequency). Metric gaming: coverage percentages that rise while real detections decay (the circular-calibration failure from T8’s warning) — countered by the rule that thresholds and baselines come from production telemetry, never from the tests. Flaky-tolerance decay: teams learn to ignore red rows that “always do that” — countered by the quarantine queue that keeps flaky out of the pass/fail signal instead of into learned helplessness. Catalog rot: tests that no longer run after platform updates, silently gated forever — countered by treating gated as a visible state with an owner, not a quiet skip. The meta-discipline: automation is only as honest as its separation of test signal from production signal, and every convention that preserved honesty in the manual exercises gets more important, not less, at CI cadence.

⚠️ What caught me: our atomic suite went fully green for six straight weeks and I reported the coverage line with pride — then a real incident used a technique the suite “covered”, and nothing fired. The post-mortem found the test had been silently passing for weeks against stale telemetry: the assertion window had been widened to 300 seconds in an early flakiness fix, and the timestamp check I’d relied on was matching events from the previous run. The test wasn’t validating the detection; it was validating itself. The fixes: assertion timestamps must be strictly greater than the run start (a one-line check that now guards every test), assertion windows are capped and reviewed, and the suite’s green streak is no longer treated as good news until the timestamp audit passes — a green suite that isn’t auditing its own assertions is the most dangerous kind of quiet.

Lab Practice

1. Seed a ten-test catalog from the T1-T8 telemetry contracts you have already validated; run atomicrun.py on the lab host and reconcile every fail against its contract — the first honest coverage number for your lab.
2. Break a control deliberately (disable one audit subcategory), re-run the suite, and confirm the exact test fails with the exact missing EID — then fix it and confirm the re-run closes the regression with a timestamped cause.
3. Build the history file across three scheduled runs and watch covreport.py surface a deliberately-introduced regression — the trend line in miniature, end to end.
4. Induce flakiness (add host load during a run) and verify the flaky flag trips after mixed results; practice the quarantine-and-retriage workflow before a real flap happens.
5. Audit the suite’s honesty: check every test’s assertion window and timestamp logic against the T9 ⚠️ failure mode, then write the catalog-lint checks (improvement #5) that make the audit automatic.

Interview Questions

• Technical: Define the atomic test contract — which fields does a catalog entry need, and which of them are safety-critical rather than informational?
• Technical: How does the telemetry assertion distinguish “the event we caused” from “an event that exists”, and what failure mode does a weak timestamp check create?
• Scenario: Your suite has been green for two months, but a real incident defeats a “covered” technique. Diagnose the ways a green suite lies, in order of likelihood.
• Conceptual: Why does automation change the honesty requirements (markers, windows, tiers) compared to manual exercises?
• Conceptual: How does the coverage trend line function as a program metric, and what gaming risks does it introduce?
• Deep-dive: Design the flaky-quarantine workflow: detection rule, quarantine state, re-triage criteria, and the telemetry-latency data that turns it diagnostic.
• Practical: You inherit a manual quarterly program and one engineer. What do you automate in the first month, and what do you deliberately keep manual?

Related Modules

T1 (ATT&CK Mapping & Emulation Planning) · T2 (Initial Access Emulation) · T3 (Execution & Persistence Emulation) · BT7 (Threat Hunting) · F5 (Python for Security)