Automation & Continuous Validation

Definition

Automation and continuous validation is the engineering layer that converts purple work from events into infrastructure: every detection validated once (PT2) becomes an automated test case, every fix becomes a regression gate, and the whole suite runs on a schedule against the live estate — so detection drift (a rule disabled by a change ticket, a log source that stops shipping, a parser broken by a field rename) is caught by the machine within a cycle, not by the next incident. The architecture mirrors what software engineering learned decades ago: tests as code (versioned, reviewable, executable), CI/CD integration (the validation suite gates detection deployment the way unit tests gate application code — BT9’s rules cannot merge without their validation passing), autonomous execution (scheduled runners executing Tier-A/B cases in the lab and staging ranges — CALDERA-class platforms for the emulation layer [verify current platform capabilities]), and drift detection as a first-class signal (the estate’s state diffed cycle-over-cycle: new detections untested, previously-passing tests now failing, telemetry contracts degrading). The craft’s center: continuous validation is the estate’s immune system for change — every improvement the purple program makes is preserved forever by a test that runs whether or not anyone remembers why.

Why It Matters

Detection estates decay at the speed of their change velocity, and the decay is invisible. A SOC shipping 20 rules a month (BT9’s healthy pace) makes ~240 changes a year — each one a chance to break a neighbor (the new rule’s parser change breaks an old one’s field mapping), disable a source (the log profile “optimized” for cost), or shift routing (the new alert taxonomy re-maps severities). Manual validation — even disciplined quarterly cycles — samples this decay at 4 points a year; continuous validation samples it continuously, and the difference is the difference between finding the broken detection at merge time (a failed CI gate, five minutes of annoyance) and finding it at 3 a.m. during an incident (an MTTD measured in hours, an RCA that reads like an indictment). The economics compound: every automated case costs minutes to run forever and preserves a fix whose original cost was hours-to-weeks of purple work.

The failure modes are automation-shaped. The flaky fleet: tests that fail intermittently for environmental reasons (timing, lab resource contention) get muted one by one until the suite “passes” and validates nothing — flakiness is a finding (PT2’s FLAKY verdict) that must be fixed or explicitly quarantined with an owner, never silently ignored. The stale suite: cases written for detections that were since retired or rewritten — the suite passes against ghosts while the current estate rots; the registry needs the same lifecycle discipline as the detections themselves (BT9’s retirement doctrine, applied to tests). Automation as assault: autonomous execution without the tier architecture — a scheduled runner that executes Tier-C techniques against production every night without markers, change control, or the SOC’s awareness is an insider threat with good intentions. The CI theater: gates that exist but can be bypassed with an --force flag and a senior person’s approval — the bypass count is a governance metric (BT20) precisely because “we have CI” and “CI gates merges” are different claims. And runner sprawl: execution logic scattered across cron jobs, laptops, and one engineer’s home directory — the validation infrastructure itself needs the repo-discipline it enforces on others.

Detailed Explanation

   THE CONTINUOUS VALIDATION PIPELINE
   BT9 repo (rules as code)          test registry (cases as code)
        | merge request                     |
        v                                   v
   +-----------------+             +------------------+
   | CI GATE (merge) |             | REGISTRY         |
   | rule lint (BT2) |             | case -> rule,    |
   | two-corpus test |             | tier, telemetry, |
   | validation case |<------------| expected verdict |
   | MUST pass       |   new cases | lifecycle status |
   +-----------------+             +--------+---------+
                                            | schedule (nightly/weekly)
                                            v
                              +--------------------------+
                              | RUNNER (autonomous)      |
                              | tier gates enforced,     |
                              | markers rotated,         |
                              | lab (nightly) staging    |
                              | (weekly) production-     |
                              | safe (on-change)         |
                              +------------+-------------+
                                           v
                              +--------------------------+
                              | RESULTS -> detectest     |
                              | verdicts, MTTD, drift    |
                              | diffs vs last cycle      |
                              +------------+-------------+
                                           v
                     +---------------------+---------------------+
                     v                     v                     v
              REGRESSION (fail      DRIFT (new untested    TELEMETRY (precheck
              was-pass -> now-      detections, retired    flips, source gaps)
              fail = alert!)        rules with live tests) -> BT3 queue
                     +---------------------+---------------------+
                                           v
                                BT20 scorecard + BT9 fix queue

Tests as code is the foundational discipline: every validation case lives in the repo with the detections it guards — technique, command, tier, telemetry contract, expected verdict, and provenance (which purple cycle produced it, linking back to the finding). The registry (the test inventory) is itself versioned and diffable, so “what changed in the validation suite” is a reviewable question. Case lifecycle mirrors detection lifecycle: active (running on schedule), quarantined (flaky, with owner + expiry — quarantine without expiry is mute), retired (detection retired — the test retires with it, explicitly).

The CI gate is where automation pays its rent. The merge pipeline for a new or modified detection runs: rule lint (BT2), two-corpus validation (BT9), and — the purple layer — the validation case for that technique executed against the staging range before the merge completes. A rule that cannot demonstrate a firing detection in staging does not merge; the emergency bypass exists, is logged, and appears in BT20’s governance metrics. The same gate pattern extends to configuration: EDR policy changes, firewall rule changes (BT13’s segaudit gates), and parser changes all run their validation cases on merge.

The runner executes on schedule with the safety architecture enforced by the tooling: tier gates checked per case (a Tier-C case cannot run outside its declared window), markers rotated per cycle (PT1’s discipline — the runner reads the current cycle’s marker set from the manifest), precheck gates each case’s telemetry (PT2), and results flow through detectest’s verdict ladder. The drift diff closes the loop: this cycle’s results vs last cycle’s — new-detections-without-tests (BT9 shipped faster than purple validated — the coverage honesty metric), was-pass-now-fail (the regression alert — page-worthy), telemetry flips (precheck deltas to BT3). Every output lands in BT20’s scorecard, making the automation’s own health as measurable as the estate it guards.

purple-automation/ # the validation repo layout ├── registry/ # cases as code (CSV per domain) │ ├── endpoint_cases.csv │ ├── cloud_cases.csv │ └── quarantine_register.csv ├── runners/ │ ├── nightly.sh # schedule entry, tier gates, markers │ ├── staging_weekly.sh │ └── on_change.sh # CI merge gate hook ├── manifests/ │ └── cycle_markers.csv # rotated per cycle (PT1 discipline) ├── results/ # detectest outputs, per cycle │ └── 2026-W42/ └── README.md # bypass procedure lives HERE, reviewed

Common Techniques

Technique	What it does	Where it fails	Notes
Tests-as-code registry	Versioned, diffable case inventory	Ghost cases for dead rules	Lifecycle = detections
CI merge gates	Detections can’t merge unvalidated	Bypass culture	Bypass count = metric
Scheduled runners	Nightly/weekly autonomous execution	Runner sprawl, laptop cron	Infra in the repo too
Tier-gated automation	Tooling enforces safety per case	The nightly production assault	Markers rotated per cycle
Drift diffing	Estate state vs last cycle	Diff without action	was-pass→now-fail pages
Regression suites	Every fix preserved forever	Quarantine-without-expiry	The compounding asset
Flaky management	Quarantine with owner + expiry	Silent mutes	Flaky is a finding
Platform integration	CALDERA-class emulation runners	Tool sprawl	[verify capabilities]
Config validation	EDR/FW/parser changes gated too	Only rules gated	Whole-estate honesty
Scorecard integration	Automation health in BT20	Automation unmetered	The meta-metric

Full Working Code

autockt — the continuous-validation scheduler and drift detector: reads the test registry (CSV: case_id, technique, tier, expected_rule, status, last_result, schedule), the latest results export (detectest output), and the detection repo’s manifest (which rules exist), then computes: cases due to run, regression alerts (was-pass → now-fail), drift (active rules without cases, cases for retired rules, quarantines past expiry), and the go/no-go for the validation cycle — exiting non-zero on regressions so the scheduler’s CI context pages a human.

#!/usr/bin/env python3
"""autockt — continuous validation scheduler + drift detector.

Inputs:
  registry.csv: case_id,technique,tier,expected_rule,status,schedule,last_result
                (status: active|quarantined|retired; schedule: nightly|weekly|on-change;
                 last_result: PASS|PASS-LATE|FLAKY|MISATTRIBUTION|FAIL|BLOCKED|-)
  results.csv:  case_id,verdict,mttd_s   (this cycle's detectest output)
  rules.csv:    rule_id,status           (BT9 repo manifest: active|retired)
Outputs: due-cases list, regression alerts, drift register, cycle go/no-go.
Exit 1 on regressions or critical drift (scheduler CI context pages a human).
"""
import argparse
import csv
import sys
from collections import defaultdict
from pathlib import Path

REQ_R = ["case_id", "technique", "tier", "expected_rule", "status",
         "schedule", "last_result"]
REQ_S = ["case_id", "verdict", "mttd_s"]
REQ_D = ["rule_id", "status"]
VERDICTS = ("PASS", "PASS-LATE", "FLAKY", "MISATTRIBUTION", "FAIL", "BLOCKED")


def load(path, req, label):
    if not path.exists():
        sys.exit(f"fail-loud: {label} not found: {path}")
    with path.open(encoding="utf-8", errors="replace") as fh:
        reader = csv.DictReader(fh)
        missing = [c for c in req if c not in (reader.fieldnames or [])]
        if missing:
            sys.exit(f"fail-loud: {label} missing columns: {missing}")
        rows = list(reader)
    return rows


def main():
    ap = argparse.ArgumentParser(description="continuous validation scheduler")
    ap.add_argument("--registry", type=Path, required=True)
    ap.add_argument("--results", type=Path, required=True)
    ap.add_argument("--rules", type=Path, required=True)
    ap.add_argument("--due-schedule", default="nightly",
                    help="which schedule class is due now (default nightly)")
    args = ap.parse_args()

    reg = load(args.registry, REQ_R, "registry")
    results = load(args.results, REQ_S, "results")
    rules = load(args.rules, REQ_D, "rules")
    if not reg:
        sys.exit("fail-loud: empty registry — automation without cases is "
                 "a cron job looking for a purpose")

    result_map = {r["case_id"].strip(): r for r in results}
    rule_status = {r["rule_id"].strip(): r["status"].strip().lower()
                   for r in rules}

    due, regressions, drift, quarantines = [], [], [], []
    seen_rules = set()
    for c in reg:
        cid = c["case_id"].strip()
        status = c["status"].strip().lower()
        schedule = c["schedule"].strip().lower()
        expected = c["expected_rule"].strip()

        if status == "quarantined":
            quarantines.append(cid)
        if status == "retired":
            continue
        if not schedule:
            sys.exit(f"fail-loud: {cid}: active case without schedule")
        if schedule == args.due_schedule:
            due.append(cid)

        # drift: case guards a rule that no longer exists or is retired
        if expected not in rule_status:
            drift.append({"kind": "GHOST-CASE", "id": cid,
                          "note": f"expected rule {expected!r} not in repo "
                                  "manifest — retire or fix the case"})
        elif rule_status[expected] == "retired":
            drift.append({"kind": "GHOST-CASE", "id": cid,
                          "note": f"rule {expected!r} retired — case must "
                                  "retire with it"})
        seen_rules.add(expected)

        # regression: last_result PASS-family, this cycle FAIL-family
        last = c["last_result"].strip().upper()
        now = result_map.get(cid, {}).get("verdict", "").strip().upper()
        if last in ("PASS", "PASS-LATE") and now in ("FAIL", "MISATTRIBUTION"):
            regressions.append({"case_id": cid,
                                "rule": expected,
                                "note": f"{last} -> {now}: was detected, "
                                        "now missed — page the owner"})

    # drift: active rules with no case guarding them
    for rid, status in rule_status.items():
        if status == "active" and rid not in seen_rules:
            drift.append({"kind": "UNTESTED-RULE", "id": rid,
                          "note": "active detection with no validation "
                                  "case — purple backlog"})

    args_out = Path("autockt_out")
    args_out.mkdir(parents=True, exist_ok=True)
    with (args_out / "due_cases.csv").open("w", newline="",
                                           encoding="utf-8") as fh:
        w = csv.writer(fh)
        w.writerow(["case_id"])
        for cid in due:
            w.writerow([cid])
    with (args_out / "drift.csv").open("w", newline="",
                                       encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["kind", "id", "note"])
        w.writeheader()
        w.writerows(drift)

    print(f"registry={len(reg)} cases  results={len(results)}  "
          f"rules={len(rules)}")
    print(f"due now ({args.due_schedule}): {len(due)} cases")
    print(f"regressions: {len(regressions)}")
    for r in regressions:
        print(f"  [REGRESSION] {r['case_id']} ({r['rule']}): {r['note']}")
    kinds = defaultdict(int)
    for d in drift:
        kinds[d["kind"]] += 1
    print(f"drift: GHOST-CASE={kinds['GHOST-CASE']} "
          f"UNTESTED-RULE={kinds['UNTESTED-RULE']}")
    for d in drift[:8]:
        print(f"  [{d['kind']}] {d['id']}: {d['note'][:70]}")
    print(f"quarantined cases: {len(quarantines)} "
          f"({', '.join(quarantines[:5]) or '-'})")
    print(f"registers: {args_out}/")
    if regressions:
        sys.exit(1)
    print("no regressions — cycle may proceed")


if __name__ == "__main__":
    main()

Code Walkthrough

The three-way join (lines 47-53) loads registry, current-cycle results, and the detection repo manifest — the manifest join is what makes drift detection possible: without knowing which rules actually exist, a case for a deleted rule passes silently forever. The per-case walk (lines 58-97) runs five checks: schedule matching (line 66 — which cases are due this run class), quarantine accounting (line 63 — counted, reported, never silently skipped), ghost-case drift (lines 71-80 — a case whose expected rule is missing from the repo manifest or retired; the test retires with its detection or it’s guarding a ghost), regression detection (lines 82-87 — last_result in the PASS family and this cycle FAIL/MISATTRIBUTION; the was-pass→now-fail transition is the page-worthy signal because it means a working detection broke), and the untested-rule sweep (lines 89-94 — active rules with no case in the registry; BT9 shipping faster than purple validates, made visible).

The exit discipline (lines 126-128) is the CI posture: regressions fail the cycle — and since autockt runs inside the scheduler’s CI context, a regression pages a human before the estate drifts further. Quarantines and drift do not block (they’re work queues, not defects), but both print in the summary so the quarterly review sees them — the automation’s own hygiene report.

How It Reaches The Target

The pipeline in operation: BT9 merges a new rule for T1053.005 on Tuesday. The merge gate (BT2 lint + BT9 two-corpus + the purple validation case for T1053.005 in staging) passes; the rule deploys; and the same validation case registers into the nightly runner (PT2’s regression seed, realized). Wednesday 02:00 the runner executes the nightly class: markers rotated for the cycle, precheck verifies telemetry, the case runs, detectest scores it — PASS, MTTD 31 seconds, logged. Friday, a change ticket “optimizes” the log profile and drops 4698 forwarding from two subnets. Saturday 02:00 the runner executes: the T1053.005 case’s telemetry contract fails precheck — BLOCKED — and the drift diff flips the case’s history: was-PASS, now-BLOCKED with a telemetry cause. The BT3 queue receives the drift register before the SOC’s Monday coffee: source down since Friday night, two subnets blind, validation caught it in 24 hours.

The regression case is the one that pages: a parser refactor on Thursday renames the field the T1547.001 rule depends on. Saturday’s run: was-PASS → FAIL. The regression alert fires with the case ID and rule, the owner gets paged, and the fix (parser mapping restored) re-runs the case green the same day — the entire incident being a failed CI gate nobody outside the program noticed. What the adversary experiences: the estate they probed last quarter is still exactly as alerting as it was — because the change velocity that usually erodes detection coverage now flows through a gate that won’t let it. And BT20’s scorecard carries the automation’s own numbers: suite size, regression rate, quarantine age, untested-rule count — the meta-metrics that say whether the immune system is healthy.

Exploitation Chain

1. Seed — every PT2 validation and PT3 fix registers a case: technique, tier, expectation, provenance
2. Registry — cases-as-code in the repo, lifecycle-matched to detections (active/quarantined/retired)
3. Gate — merge pipeline: BT2 lint + BT9 corpus + validation case in staging, or logged bypass
4. Schedule — runners execute nightly/weekly/on-change with tier gates, rotated markers, precheck
5. Score — detectest verdicts + MTTD per case; results flow to the registry
6. Diff — autockt: regressions (was-pass→now-fail), ghost cases, untested rules, telemetry flips
7. Alert — regressions page owners via CI exit code; drift feeds BT3/BT9 queues
8. Report — suite health (size, regression rate, quarantine age, coverage) into BT20’s scorecard

Building Your Own Tool

Automation accumulates quarantines the way queues accumulate backlog — and a quarantine without an expiry is a mute button. regress is the quarantine watchdog: it reads the quarantine register (case, reason, owner, quarantined_date, expiry) and the current results, enforces expiry (expired quarantines fail the cycle), and reports the quarantine age distribution — the metric that says whether flaky management is a process or a landfill.

Design decisions

Expiry is law: a quarantine past its expiry (default 14 days) fails the cycle — the team must either fix the flake, extend with justification (a new expiry + reason), or retire the case; silence is not an option the tool permits. Re-entry requires evidence: a quarantined case returns to active only when its results show consecutive passes (the register’s reentry_evidence field), so un-muting is a decision backed by data, not hope. Age buckets (0-7d, 7-14d, expired) make the distribution readable at a glance — a healthy program’s quarantine population is young and small. Fail-loud on register schema drift; date parsing fails per-row with context.

Full source

#!/usr/bin/env python3
"""regress — quarantine watchdog for the validation suite.

Input register.csv: case_id,reason,owner,quarantined_date,expiry,
                    reentry_evidence
Output: age buckets, expired list (cycle fails), re-entry eligibility."""
import argparse
import csv
import sys
from datetime import date, datetime
from pathlib import Path

REQ = ["case_id", "reason", "owner", "quarantined_date", "expiry",
       "reentry_evidence"]
DEFAULT_EXPIRY_DAYS = 14


def parse_d(raw):
    for fmt in ("%Y-%m-%d", "%Y/%m/%d"):
        try:
            return datetime.strptime(raw.strip(), fmt).date()
        except ValueError:
            continue
    return None


def main():
    ap = argparse.ArgumentParser(description="quarantine watchdog")
    ap.add_argument("register", type=Path)
    ap.add_argument("--consecutive-required", type=int, default=3,
                    help="passes needed to re-enter active (default 3)")
    ap.add_argument("--out", type=Path, default=Path("regress_out"))
    args = ap.parse_args()

    if not args.register.exists():
        sys.exit(f"fail-loud: register not found: {args.register}")
    with args.register.open(encoding="utf-8", errors="replace") as fh:
        rows = list(csv.DictReader(fh))
    for r in rows:
        missing = [c for c in REQ if c not in r]
        if missing:
            sys.exit(f"fail-loud: register missing columns: {missing}")
    if not rows:
        sys.exit("fail-loud: empty quarantine register — either pristine "
                 "or broken; verify with a smoke case")

    today = date.today()
    buckets = {"0-7d": 0, "7-14d": 0, "EXPIRED": 0}
    expired, eligible = [], []
    for r in rows:
        cid = r["case_id"].strip()
        qd = parse_d(r["quarantined_date"])
        ex = parse_d(r["expiry"])
        if qd is None:
            sys.exit(f"fail-loud: {cid}: bad quarantined_date "
                     f"{r['quarantined_date']!r}")
        if ex is None:
            sys.exit(f"fail-loud: {cid}: bad expiry {r['expiry']!r} — "
                     "quarantine without expiry is a mute button")
        age = (today - qd).days
        if today > ex:
            buckets["EXPIRED"] += 1
            expired.append({"case_id": cid, "owner": r["owner"].strip(),
                            "age_days": age,
                            "note": "expired — fix, extend with "
                                    "justification, or retire"})
        elif age <= 7:
            buckets["0-7d"] += 1
        else:
            buckets["7-14d"] += 1
        try:
            passes = int(r["reentry_evidence"] or 0)
        except ValueError:
            sys.exit(f"fail-loud: {cid}: reentry_evidence not numeric")
        if passes >= args.consecutive_required and today <= ex:
            eligible.append({"case_id": cid,
                             "note": f"{passes} consecutive passes — "
                                     "eligible for active"})

    args.out.mkdir(parents=True, exist_ok=True)
    with (args.out / "expired.csv").open("w", newline="",
                                         encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["case_id", "owner", "age_days",
                                           "note"])
        w.writeheader()
        w.writerows(expired)

    print(f"quarantine population: {len(rows)}  buckets: {buckets}")
    print(f"re-entry eligible: {len(eligible)}")
    for e in eligible:
        print(f"  {e['case_id']}: {e['note']}")
    if expired:
        print(f"EXPIRED ({len(expired)}) — cycle fails until resolved:")
        for e in expired:
            print(f"  {e['case_id']} (owner {e['owner']}, "
                  f"{e['age_days']}d): fix, extend, or retire")
        print(f"register: {args.out / 'expired.csv'}")
        sys.exit(1)
    print("quarantine hygiene clean — no expired entries")


if __name__ == "__main__":
    main()

Walkthrough

The expiry law (lines 46-49) is the tool’s whole point: every quarantine row carries an expiry date, and today > expiry fails the cycle with the owner attached — the three exits (fix the flake, extend with a new justified expiry, retire the case) are the only resolutions, and silence resolves nothing. The parse gates (lines 38-44, 51-54) fail loud on bad dates and missing evidence counts — a register that half-parses would otherwise let exactly the oldest entries (the ones typed sloppiest) escape review. Re-entry eligibility (lines 56-60) requires consecutive_required passes recorded in the register’s evidence field — the runner writes the count, so un-muting is mechanical rather than social. The age buckets (lines 41-46, printed at line 72) turn the population into one glance: healthy programs show young quarantines in small numbers; the landfill pattern (large 7-14d and EXPIRED buckets) is visible before the next quarterly review asks about it.

Compilation and test procedure

weekly hygiene run inside the validation CI

$ python3 regress.py quarantine_register.csv quarantine population: 6 buckets: {‘0-7d’: 2, ‘7-14d’: 3, ‘EXPIRED’: 1} re-entry eligible: 1 CASE-0117: 3 consecutive passes — eligible for active EXPIRED (1) — cycle fails until resolved: CASE-0093 (owner deteng-2, 19d): fix, extend, or retire register: regress_out/expired.csv

resolve CASE-0093 (extend with justification), re-run -> clean

$ python3 regress.py quarantine_register.csv | tail -1 quarantine hygiene clean — no expired entries

the scheduler’s full cycle: scheduler + harness + watchdog

$ autockt —registry r.csv —results res.csv —rules m.csv —due-schedule nightly &&
detectest —tests t.csv —alerts a.csv &&
regress.py quarantine_register.csv registry=142 cases results=58 rules=131 due now (nightly): 58 cases regressions: 0 drift: GHOST-CASE=2 UNTESTED-RULE=4 quarantined cases: 5 (CASE-0031, CASE-0044, CASE-0093, CASE-0117, CASE-0122) no regressions — cycle may proceed

Detection considerations

The automation’s own telemetry joins BT20: suite size (is the regression asset growing with the estate?), regression rate per quarter (estate stability — spiking regression rate means change velocity is outrunning validation), quarantine age distribution (flaky management hygiene), UNTESTED-RULE count (purple backlog honesty), and bypass counts from the CI gates (governance). Two failure tells to watch: a suite that never grows while the estate does (automation theater — the gate exists but new detections skip it), and quarantine population that only ever grows (the mute-by-inertia pattern regress exists to stop). The runner infrastructure itself needs the BT17 treatment: containerized, audited, and segmented — an autonomous execution platform with credentials is a tier-0 asset wearing a tool’s clothing.

Improvements to add

1. Auto-register cases on merge: the CI gate that validates a new rule proposes its registry entry in the same PR.
2. Add execution-cost tracking per case (runtime, lab resources) so the nightly schedule optimizes like a build matrix.
3. Emit a per-team validation debt view (UNTESTED-RULE by owning team) for the quarterly roadmap.
4. Integrate CALDERA-class platforms for the emulation-layer cases via their APIs [verify current API surface].
5. Cross-check runner infrastructure against BT17’s podspec admission policies — the automation estate follows the same rules it enforces.

Detection — Defender View

Signal	Source	ID	Notes
Regression alerts	autockt	per cycle	Page-worthy
Drift: ghost cases	autockt	per cycle	Test hygiene
Drift: untested rules	autockt	per cycle	Purple backlog
Quarantine age	regress	weekly	The mute-button guard
Telemetry flips	precheck delta	per cycle	BT3 gap feed
CI bypass count	Gate logs	per quarter	Governance metric
Suite size trend	Registry	quarterly	Asset growth
MTTD per case	detectest	per cycle	Latency drift
Runner infra health	Heartbeat	per day	Automation honesty
Estate regression rate	autockt history	quarterly	Change-velocity truth

Evasion Techniques

Adversaries (and decay) evade continuous validation through its blind spots. Schedule transparency: nightly runners at fixed times are observable — an adversary with internal access times operations for the 22 hours between runs, or (better) during the runner’s own noise; randomized schedule jitter and continuous-access-evaluation controls shrink the window. The bypassed gate: organizational evasion — an emergency deploy that skips CI “just this once” becomes the pattern; bypass counts in BT20 with named approvers exist so the governance metric makes the pattern visible before it’s culture. Case-shape fingerprinting: automated cases run the same command with the same artifacts on the same hosts — adversaries who fingerprint the runner’s signature (hostnames, artifact hashes) distinguish test from real and from the manual purple cycles; marker rotation plus host/artifact rotation per cycle are the counters. The excluded corridor: automation validates the standard paths; EDR exclusions, service accounts, and management-plane paths (BT2/BT13’s blind-spot catalog) stay untested — the exclusion register’s test-case requirement applies to automation too. Slow-decay evasion: changes that degrade detection below thresholds without breaking it (a source that starts dropping 5% of events) pass binary PASS/FAIL checks — latency and consistency metrics (MTTD trends, FLAKY rates) are the drift detectors that catch erosion a boolean cannot. Infrastructure compromise inverted: the runner is a credential-holding, widely-connected machine on a schedule — its compromise is the adversary’s automation; the BT17 hardening (admission policies, minimal service accounts, audited changes) is what keeps the immune system from becoming the infection vector.

⚠️ What caught me: our continuous validation ran green for six straight months, and I presented it as the program’s crown jewel — until a real incident showed the T1059.001 detection failing, and the RCA found the validation case had been quarantined since spring for lab-resource flakiness. The quarantine had no expiry, the owner had left, and the mute was so old it had institutional standing (“that one’s always been skipped”). Green dashboards and dead coverage, simultaneously. The fix was regress: quarantine with expiry or it fails the cycle, re-entry on evidence not vibes. The deeper lesson I now repeat in every design review: automation doesn’t remove the need for skepticism — it industrializes the need. Every silent skip in the machinery is a decision somebody made without recording it, and the machine will defend that decision forever unless you build the tool that surfaces it.

Lab Practice

Automate the validation you ran manually in PT2. Registry: convert your PT2 test cases into registry entries (case_id, tier, schedule, expected_rule, provenance = which cycle produced them). Gate: wire the validation into your rules repo’s CI (or a local pre-merge script) — attempt to merge a deliberately broken rule and watch the gate refuse. Schedule: run autockt nightly against your lab (cron/systemd timer [verify per distro]); after a week, break something deliberately (disable a log source) and confirm the drift register catches it within one cycle. Watchdog: introduce two quarantines (one fresh, one expired) and run regress — confirm the expiry fails the cycle and the re-entry logic requires evidence. Measure: collect the four meta-metrics (suite size, regression rate, quarantine age, untested rules) into your BT20 scorecard. Debrief: how long did the broken-source drift take to surface — and what was that latency under your previous manual cadence?

Interview Questions

• Technical: Walk the continuous-validation pipeline — registry, merge gate, scheduled runner, drift diff — and identify which component prevents each of: ghost cases, silent mutes, and untested detections.
• Technical: Why is a was-pass→now-fail regression page-worthy while a BLOCKED case is a work queue item? What different failure does each represent?
• Scenario: Your nightly suite has been green for a quarter, but an incident reveals a broken detection. Diagnose: what automation blind spot allowed this, and which two metrics would have exposed it?
• Scenario: Engineering demands a CI bypass for an emergency detection deploy. Walk the decision: what gets logged, who approves, and how does the bypass count appear in BT20?
• Conceptual: Why does quarantine-without-expiry constitute a mute button, and what three resolutions does regress force?
• Deep-dive: Design continuous validation for a 200-rule estate: registry schema, CI integration, runner architecture with safety enforcement, drift detection, and the meta-metrics for BT20.
• Practical: You have one engineer-month to automate validation for the highest-value 30 detections. What do you build first — the registry, the gate, or the runner — and what stays manual?

Related Modules

PT2 (Detection Validation & Control Testing) · PT3 (Adversary Emulation Campaigns) · BT9 (Detection Engineering & Sigma) · BT16 (Cloud Detection & Response) · BT17 (Container & Kubernetes Security Ops)