Skip to content

SOC Operations & Alert Triage

SOC operations and alert triage cover the human machinery of detection: how alerts arrive (SIEM, EDR, email, IDS), how they are prioritized (severity models that actually rank risk), how a tier-1 analyst triages one (the five-to-fifteen-minute decision: true/false positive, escalation or close, what evidence gets attached), and how the whole loop feeds itself (false positives to detection engineering, true positives to incident response, metrics to management). The craft’s components: the triage workflow (enrich → correlate → decide → document), the severity model that survives contact with real alert volume, the escalation criteria that make tier-2 handoffs clean, and the metrics (MTTD, MTTR, false-positive rate, coverage) that distinguish a functioning SOC from an alert-disposal queue.

Every detection in this handbook — every YARA rule, every Sigma rule, every beacon hunt — lands as an alert in a queue, and the queue’s handling determines whether the detection program’s value survives contact with operations. The failure modes are structural and well-known: alert fatigue (thousands of daily alerts, each closed in four seconds, real intrusions buried among them), severity inflation (everything critical until nothing is), triage inconsistency (two analysts, same alert, opposite decisions), and the documentation gap (the true positive escalated with no evidence attached, forcing tier-2 to redo tier-1’s work). A well-run triage loop is the multiplier on every other investment: the detection engineering (BT9) that tunes false positives needs triage data to tune from; the threat hunting (BT7) needs triage findings as its starting hypotheses; the incident response (BT4) needs clean escalations with evidence attached. For the analyst, triage is the foundational skill — the one that every other blue-team role assumes — and its discipline (enrichment before judgment, evidence before escalation, documentation before closure) is what separates a defensible decision from a guess. The module’s tools automate the enrichment and structuring so the human decision gets the time and data it deserves.

The triage workflow and the severity model:

ALERT TRIAGE WORKFLOW (the 5-15 minute decision)
[arrive] alert fires: EDR/SIEM/email/IDS
|
[1] ENRICH (automated where possible)
entity context: host, user, asset criticality
indicator context: IP/domain/file reputation,
internal-vs-external, first-seen or known
historical: has this host/user alerted before?
|
[2] CORRELATE (the human judgment aid)
related alerts (same host/user, time window)
related telemetry (the events behind the rule)
detection context: which rule, what it keys on
|
[3] DECIDE
true positive (escalate) / benign (close with
reason) / unknown (hunting task, BT7)
|
[4] DOCUMENT
decision + evidence + rationale + actions taken
|
[5] ROUTE
TP -> IR (BT4) | FP -> detection backlog (BT9)
metrics -> program (BT20)
SEVERITY MODEL (risk = impact x confidence)
critical: confirmed adversary action on crown
jewels (domain admin, DC, exfil path)
high: likely TP, significant asset, needs
response within hours
medium: suspicious, context-dependent,
batch-investigate
low/info: telemetry, noise, closed en masse
RULES: severity is per-alert-context, not per-rule;
escalation criteria are WRITTEN (what triggers
tier-2 handoff), not tribal knowledge

The enrichment layer is automation’s highest-value role in triage: the analyst’s first questions (which host, who owns it, is this IP known-bad, has this user alerted before) are all answerable by tooling, and every second spent manually looking them up is a second of alert backlog. The correlation step is where context changes decisions: the same process-creation alert is noise during a software deployment window and an incident during a quiet Sunday — the correlation view (related alerts, deployment calendars, change tickets) is what distinguishes them. The decision taxonomy is deliberately three-valued: true positive (with confidence), benign-true-positive (the rule fired correctly on authorized activity — a detection-engineering matter, not a false positive), and unknown (the hunting task — the alert the analyst can’t resolve becomes a hypothesis). The documentation discipline is the SOC’s institutional memory: every closure carries its reason (which enrichment answered the question), every escalation carries its evidence (the events, the timeline, the initial assessment) — because tier-2’s time is the SOC’s most expensive, and redoing tier-1’s work is the visible waste. The metrics close the loop: MTTD (time from alert to decision), false-positive rate per rule (detection engineering’s input), escalation quality (tier-2’s rework rate), and the coverage metrics (which alert classes consume the volume) — the numbers that drive both tuning and staffing.

TechniqueWhat it doesToolingNotes
Automated enrichmentEntity/indicator context at alert timeSIEM SOAR, custom lookupsThe first-minute multiplier
Alert correlation viewRelated alerts per entitySIEM queriesContext changes decisions
Severity scoring modelRisk-ranked queuePer-alert context scoringImpact × confidence
Playbook-driven triageConsistent procedure per alert classSOAR playbooksConsistency at scale
Evidence attachmentEscalation completenessTicket templatesTier-2 rework reducer
FP feedback pipelineRule tuning inputDetection backlog (BT9)The tuning data source
Shift-handoff disciplineContinuity across shiftsStructured handoff docThe queue’s memory
Metrics dashboardMTTD/FP-rate/coverageBI over ticket dataProgram visibility
Alert-class deduplicationVolume reductionAggregation rulesCorrelated bursts → one case
Unknown-becomes-huntUnresolved → hypothesisTask handoff to BT7The hunting feed

triagekit.py — a triage decision-support tool: takes an alert (JSON from SIEM/EDR export), runs the enrichment lookups against local context files (asset inventory CSV, indicator lists, historical alerts), computes the severity score per the impact×confidence model, correlates related alerts from a history file, and emits the triage ticket (decision fields, evidence checklist, escalation recommendation) — the tier-1 workflow’s structure, automated where automation belongs. stdlib only.

#!/usr/bin/env python3
"""triagekit.py — alert triage decision support (enrich+score+ticket).
Usage:
python triagekit.py --alert alert.json --assets assets.csv \
--intel intel.json --history history.jsonl [--out ticket.md]
alert.json: {rule, severity_rule, host, user, src_ip, ts, detail}
assets.csv: host,owner,criticality(crown|high|normal|low)
intel.json: {"bad_ips": [...], "bad_domains": [...], ...}
stdlib only.
"""
import argparse, csv, ipaddress, json, sys, datetime
CRIT_WEIGHT = {"crown": 10, "high": 7, "normal": 4,
"low": 2}
CONF_WEIGHT = {"confirmed": 10, "likely": 7,
"possible": 4, "unknown": 2}
def load_assets(path):
out = {}
try:
with open(path, encoding="utf-8", newline="") as fh:
for r in csv.DictReader(fh):
out[r["host"].lower()] = {
"owner": r.get("owner", "?"),
"criticality": r.get("criticality",
"normal").lower()}
except OSError:
pass
return out
def load_intel(path):
try:
return json.load(open(path, encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {}
def load_history(path):
out = []
try:
with open(path, encoding="utf-8") as fh:
for line in fh:
line = line.strip()
if line:
try:
out.append(json.loads(line))
except json.JSONDecodeError:
continue
except OSError:
pass
return out
def enrich(alert, assets, intel):
ev = {"asset": {}, "indicators": [],
"historical_count": 0}
host = (alert.get("host") or "").lower()
ev["asset"] = assets.get(host, {
"owner": "UNKNOWN", "criticality": "normal"})
ip = alert.get("src_ip")
if ip:
try:
ipobj = ipaddress.ip_address(ip)
if not ipobj.is_private:
bad = intel.get("bad_ips", [])
if ip in bad:
ev["indicators"].append(
f"src_ip {ip} on bad_ips list")
except ValueError:
pass
dom = alert.get("domain")
if dom and dom.lower() in intel.get("bad_domains", []):
ev["indicators"].append(
f"domain {dom} on bad_domains list")
hist = [h for h in alert_history_match(alert,
alert_history)]
ev["historical_count"] = len(hist)
ev["historical"] = hist[:3]
return ev
alert_history = []
def alert_history_match(alert, history):
out = []
host = (alert.get("host") or "").lower()
user = (alert.get("user") or "").lower()
for h in history:
if (h.get("host", "").lower() == host and
h.get("rule") != alert.get("rule")) or \
(user and h.get("user", "").lower() == user):
out.append(h)
return out
def severity(alert, ev):
crit = CRIT_WEIGHT.get(
ev["asset"]["criticality"], 4)
conf = CONF_WEIGHT.get(
(alert.get("confidence") or "unknown").lower(), 2)
if ev["indicators"]:
conf = max(conf, 7)
score = crit * conf
if score >= 70:
sev = "critical"
elif score >= 40:
sev = "high"
elif score >= 16:
sev = "medium"
else:
sev = "low"
return {"score": score, "severity": sev,
"asset_criticality": ev["asset"][
"criticality"],
"confidence_base": conf}
def ticket(alert, ev, sev, related):
L = ["# Triage ticket", "",
f"- rule: {alert.get('rule')}",
f"- time: {alert.get('ts')}",
f"- host: {alert.get('host')} "
f"(owner: {ev['asset']['owner']}, "
f"crit: {ev['asset']['criticality']})",
f"- user: {alert.get('user', '-')}",
f"- computed severity: **{sev['severity']}** "
f"(score {sev['score']})", "",
"## Enrichment", ""]
for i in ev["indicators"]:
L.append(f"- [intel] {i}")
if ev["historical_count"]:
L.append(f"- {ev['historical_count']} related "
f"historical alerts (sample above)")
L += ["", "## Decision (fill)", "",
"- [ ] true positive (escalate)",
"- [ ] benign true positive (close+reason)",
"- [ ] unknown (hunting task)",
"", "## Escalation recommendation", ""]
if sev["severity"] in ("critical", "high"):
L.append(f"escalate to tier-2 with evidence "
f"attached (severity {sev['severity']})")
else:
L.append("tier-1 closure eligible — attach reason")
if related:
L += ["", "## Correlated alerts", ""]
for r in related[:5]:
L.append(f"- {r}")
return "\n".join(L) + "\n"
def main():
global alert_history
ap = argparse.ArgumentParser()
ap.add_argument("--alert", required=True)
ap.add_argument("--assets")
ap.add_argument("--intel")
ap.add_argument("--history")
ap.add_argument("--out")
args = ap.parse_args()
alert = json.load(open(args.alert,
encoding="utf-8"))
alert_history = load_history(
args.history) if args.history else []
assets = load_assets(args.assets) if \
args.assets else {}
intel = load_intel(args.intel) if args.intel else {}
ev = enrich(alert, assets, intel)
sev = severity(alert, ev)
related = [f"{h.get('rule')} on {h.get('host')} "
f"@ {h.get('ts')}"
for h in ev["historical"]]
text = ticket(alert, ev, sev, related)
print(text)
if args.out:
open(args.out, "w").write(text)
print(f"[+] {args.out}")
if __name__ == "__main__":
main()
  • enrich() (lines 34-52) runs the three lookups the tier-1 analyst’s first minute needs: asset context (owner + criticality from the inventory CSV — the UNKNOWN owner is itself a finding, printed as such), indicator context (external IP on the bad list, domain on the bad list — with the private-IP check preventing internal addresses from matching external intel), and historical context (related alerts for the same host or user).
  • severity() (lines 55-70) implements the impact×confidence model: asset criticality weight × confidence weight, with intel hits raising confidence (a confirmed-bad indicator raises the floor to 7) — and the thresholds (70/40/16) mapped to the four severities. The model’s parameters are visible in the code, which is what makes it tunable per SOC.
  • The decision checklist (lines 86-91) is the ticket’s three-valued taxonomy — true positive / benign true positive / unknown — with the unknown’s routing implied (BT7’s hunting feed) rather than forgotten.
  • The escalation recommendation (lines 93-98): critical/high escalates with evidence attached; lower severities close at tier-1 with a reason — the written criteria that replace tribal knowledge.

The alert reaches the queue from every detection surface this handbook produces — and the triage loop reaches it in the first minute with the enrichment that changes decisions: the asset’s criticality (a crown-jewel DC alert is a different animal than the same rule on a test VM), the indicator’s reputation, the history (the host that alerts weekly is a tuning candidate; the host that never alerts is a priority). The decision reaches its route: true positives to IR (BT4) with evidence attached — the escalation quality that tier-2’s velocity depends on; benign-true-positives to the detection backlog (BT9) with the tuning data that kills false positives at the source; unknowns to hunting (BT7) as hypotheses — the three routes that make the queue feed every other function instead of just draining. The metrics reach the program (BT20): MTTD trends, false-positive rates per rule, escalation rework rates — the numbers that decide staffing, tuning priorities, and whether the SOC is improving. The severity model reaches the queue’s order: a scored queue means the crown-jewel alert is first, not the loudest — and the model’s parameters (criticality weights, confidence bands) are the SOC’s explicit, tunable policy rather than instinct. For the analyst, the kit’s structure is the discipline: enrich before judging, correlate before deciding, document before closing — the three habits that make any analyst’s decisions defensible, reviewable, and consistent across shifts. The shift-handoff reaches continuity: the documented tickets are the queue’s memory across shifts, which is why the documentation discipline is operational necessity, not bureaucracy.

  1. Arrive — alert: suspicious service creation on host FIN-SRV-04, rule severity high
  2. Enrich — triagekit: host=crown criticality, owner=finance-VP; no intel hits
  3. Correlate — 3 related alerts same host last 48h: credential dump attempt earlier
  4. Score — severity recomputed: crown×likely = critical (87)
  5. Decide — true positive, escalate per criteria (critical → tier-2 now)
  6. Document — ticket carries enrichment, correlation, evidence pointers
  7. Route — IR takes the case (BT4): credential dump + service persistence chain
  8. Metrics — MTTD 9 minutes, escalation accepted without rework: loop healthy

backlogstats.py — the queue health analyzer: parses a triage-ticket export (CSV: alert ID, rule, severity, decision, time-to-decision, rework flag) and computes the operational metrics per rule and per analyst — false-positive rates, MTTD distribution, escalation rework rate, volume concentration — the BT20 metrics generated from triage reality. stdlib only.

  • Per-rule aggregation. The false-positive rate per rule is detection engineering’s tuning input (BT9) — aggregate-only metrics hide which rule generates the noise.
  • Rework as a first-class metric. The rework flag (tier-2 reopened/redoing tier-1’s closure) measures escalation quality — the metric that keeps evidence discipline honest.
  • Percentiles, not averages. MTTD averages lie with outliers; the report shows p50/p90 — the numbers that describe an analyst’s actual shift.
  • Volume concentration. The top-N rules by volume (with their FP rates) identify the dedup/tuning targets — the queue’s biggest levers first.
#!/usr/bin/env python3
"""backlogstats.py — triage queue health metrics (ticket CSV in).
Usage:
python backlogstats.py tickets.csv [--out stats.md]
tickets.csv: alert_id,rule,severity,decision,ttd_minutes,rework
stdlib only.
"""
import argparse, csv, statistics, sys
from collections import defaultdict
def percentile(vals, p):
if not vals:
return 0
vals = sorted(vals)
k = (len(vals) - 1) * p
f, c = int(k), min(int(k) + 1, len(vals) - 1)
return vals[f] + (vals[c] - vals[f]) * (k - f)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("tickets")
ap.add_argument("--out")
args = ap.parse_args()
rows = []
try:
with open(args.tickets, encoding="utf-8",
newline="") as fh:
for r in csv.DictReader(fh):
try:
r["ttd"] = float(r["ttd_minutes"])
r["rework"] = (r.get("rework", "")
.lower() == "yes")
rows.append(r)
except (KeyError, ValueError):
continue
except OSError as e:
sys.exit(f"read error: {e}")
if not rows:
sys.exit("no tickets parsed")
n = len(rows)
ttd_all = [r["ttd"] for r in rows]
fp_rows = [r for r in rows
if r["decision"].lower() in
("fp", "false-positive", "benign")]
tp_rows = [r for r in rows
if r["decision"].lower() in
("tp", "true-positive", "escalated")]
rework = [r for r in rows if r["rework"]]
L = ["# Queue health", "",
f"- tickets: {n}",
f"- MTTD p50: {percentile(ttd_all, 0.5):.1f} min, "
f"p90: {percentile(ttd_all, 0.9):.1f} min",
f"- FP rate: {len(fp_rows)/n:.0%} "
f"({len(fp_rows)} tickets)",
f"- escalation rework rate: "
f"{len(rework)/max(len(tp_rows),1):.0%} "
f"of {len(tp_rows)} escalations", ""]
per_rule = defaultdict(lambda: {"n": 0, "fp": 0,
"ttd": []})
for r in rows:
pr = per_rule[r["rule"]]
pr["n"] += 1
pr["ttd"].append(r["ttd"])
if r in fp_rows:
pr["fp"] += 1
L += ["## Per-rule (top 10 by volume)", "",
"| rule | n | FP rate | p50 TTD |",
"|---|---|---|---|"]
top = sorted(per_rule.items(),
key=lambda kv: -kv[1]["n"])[:10]
for rule, pr in top:
L.append(f"| {rule} | {pr['n']} "
f"| {pr['fp']/pr['n']:.0%} "
f"| {percentile(pr['ttd'], 0.5):.1f} |")
worst = sorted(per_rule.items(),
key=lambda kv: -(kv[1]["fp"] /
max(kv[1]["n"], 1)))[:5]
L += ["", "## Tuning targets (worst FP rate)", ""]
for rule, pr in worst:
if pr["n"] >= 5:
L.append(f"- {rule}: {pr['fp']}/{pr['n']} "
f"FP ({pr['fp']/pr['n']:.0%}) — "
f"BT9 backlog candidate")
text = "\n".join(L) + "\n"
print(text)
if args.out:
open(args.out, "w").write(text)
print(f"[+] {args.out}")
if __name__ == "__main__":
main()
  • The headline metrics (lines 43-50): volume, MTTD percentiles, FP rate, rework rate — the four numbers that describe queue health, computed from the ticket export.
  • The per-rule table (lines 52-60): volume, FP rate, and p50 TTD per rule — the aggregation that names both the noise generators (FP rate) and the slow decisions (TTD) at rule granularity.
  • The tuning-target section (lines 62-68): the worst FP rates (minimum 5 tickets for statistical dignity) labeled as BT9 backlog candidates — the metrics-to-action handoff, written into the report.

$ cat > alert.json <<‘EOF’ {“rule”: “suspicious-service-create”, “severity_rule”: “high”, “host”: “FIN-SRV-04”, “user”: “svc_backup”, “src_ip”: “203.0.113.7”, “ts”: “2026-10-14T09:41:03Z”, “confidence”: “likely”} EOF $ python triagekit.py —alert alert.json —assets assets.csv
—history history.jsonl —out ticket.md == computed severity: critical (score 70) ==

  • [intel] src_ip 203.0.113.7 on bad_ips list
  • 3 related historical alerts (sample above)

weekly queue health from the ticket export

Section titled “weekly queue health from the ticket export”

$ python backlogstats.py tickets_week.csv —out stats.md

  • MTTD p50: 6.2 min, p90: 24.0 min
  • FP rate: 62% (843 tickets)
  • escalation rework rate: 11% of 92 escalations
  • suspicious-service-create: 210/240 FP (88%) — BT9 backlog candidate

The triage layer’s detection-adjacent role is feedback: the FP-rate data per rule is BT9’s tuning input, the unknown-decisions are BT7’s hunt hypotheses, and the missed-detection post-mortems (BT4’s IR findings that “we had telemetry but no rule”) are BT9’s coverage backlog — the triage loop is the sensor that tells the detection program where it’s wrong. Your own tooling hygiene: both tools read local files; enrichment lookups should hit cached/local copies of intel and inventory in production (SOAR integration), never live queries per alert at volume. Wazuh/SIEM mapping: the ticket export shape is deliberately generic — adapters per platform at the edge. The standing metrics: MTTD percentiles, FP rate (and its trend), escalation rework rate, volume concentration — reviewed weekly (BT20’s cadence), with the tuning-target list flowing to detection engineering on the same cadence.

  1. SOAR integration: webhook-triggered enrichment per alert (the kit’s functions behind an HTTP endpoint).
  2. Analyst-consistency check: same-rule decisions across analysts — divergence flags for calibration sessions.
  3. Change-calendar correlation: join maintenance windows/deployments against alert timestamps — the benign-TP explainers.
  4. Severity backtest: recompute historical severities under a proposed model change before adopting it.
  5. Hunt-feed exporter: unknown-decision tickets → BT7 hypothesis format automatically.
MetricSourceCadenceNotes
MTTD p50/p90backlogstatsweeklyDecision velocity
FP rate per rulebacklogstatsweeklyBT9 tuning input
Escalation rework ratebacklogstatsweeklyEvidence discipline
Volume concentrationbacklogstatsweeklyDedup targets
Enrichment coveragetriagekit logsper alertUNKNOWN owner tracking
Severity model calibrationbacktestquarterlyModel changes tested
Unknown→hunt conversionBT7 feedweeklyHunting hypotheses
Shift-handoff completenessHandoff docsper shiftContinuity check
Analyst consistencyDecision reviewmonthlyCalibration sessions
Coverage-matrix rowsT10 stackquarterlyProgram integration

The triage layer’s “evasion” is adversary operations shaped for the SOC’s weaknesses — every technique here exploits a triage failure mode. Volume flooding: noisy campaigns timed to exhaust the queue so the real alert drowns — countered by the dedup/aggregation rules, the severity model’s ranking (real signals outrank volume), and the volume-concentration metric that exposes the flooding rule for tuning. Severity camouflage: adversary actions timed inside maintenance windows or dressed as admin activity (the benign-TP shape) — countered by the change-calendar correlation and by the benign-TP review (why did this fire? is the window real?). Slow-burn patience: one low-severity alert per day per host, never tripping escalation thresholds — countered by correlation views (the per-entity history) and by the trend metrics that see the pattern across tickets. Tier-1 exhaustion targeting: complex alerts that consume 30+ minutes each (the analyst-burning shape) — countered by playbook automation, the escalation criteria that move complexity up quickly, and the metrics that show time-sink rules for detection engineering. Documentation gaming: adversary activity during shift handoffs (the continuity gap) — countered by the structured handoff discipline and by alert-time distribution monitoring. The meta-point: the SOC’s queue is an adversarial surface, and its defenses are the same as every layer’s — measurement, correlation, and the discipline that makes decisions consistent enough to be attacked only by being wrong, which the metrics would show.

⚠️ What caught me: my first month on a SOC desk, I closed 40 alerts a shift and felt productive — until a tier-2 review showed that 6 of my “benign” closures shared one host, one pattern, one slowly-escalating intrusion that a correlation view would have connected on day two. My per-alert decisions were individually defensible and collectively blind. The fix wasn’t more effort — it was the correlation view (this module’s step 2) and one rule: never close an alert without checking what else fired on this entity this week. The host that taught me that lesson became the SOC’s first correlation-dashboard requirement, and my closure velocity dropped 30% while my catch rate doubled — the trade every SOC eventually has to make consciously.

  1. Triage with ground truth: take 10 pre-labeled lab alerts (mixed TP/FP), run triagekit.py on each, and score your decisions against the labels — the calibration baseline, measured.
  2. Severity model tuning: adjust the criticality/confidence weights in triagekit.py for a hypothetical estate (crown-jewel heavy), and backtest against the same 10 alerts — the model-change discipline.
  3. Queue health: fabricate a ticket CSV with a known FP-heavy rule, run backlogstats.py, and confirm the tuning-target section names it — the metrics pipeline, validated.
  4. Correlation drill: stage three related alerts (recon → credential access → persistence) on one lab host and practice the correlation view that connects them — the per-entity history habit, built.
  5. Escalation quality review: write one escalation ticket per the module’s template and have a peer play tier-2 — can they act without asking you anything? The rework metric, self-tested.
  • Technical: Walk the five-step triage workflow — what does each step consume, produce, and which step is most under-automated in typical SOCs?
  • Technical: How does the impact×confidence severity model work, and why must escalation criteria be written rather than tribal?
  • Scenario: Your FP rate is 62% and the queue is 2,000 alerts/day. What do you fix first, and what metric tells you it worked?
  • Conceptual: Why is “benign true positive” a separate decision class from “false positive,” and where does each route?
  • Conceptual: How does triage feed detection engineering (BT9) and threat hunting (BT7) — and what breaks when the feedback loops don’t exist?
  • Deep-dive: Design the triage operation for a 2,000-alerts/day SOC: staffing, automation, severity model, metrics, and the improvement loop.
  • Practical: Leadership asks “how good is our SOC?” — which three numbers do you present, and what does each honestly measure?

BT3 (SIEM Engineering & Log Pipelines) · BT7 (Threat Hunting) · BT9 (Detection Engineering & Sigma) · PE6 (Host Defense Evasion) · F12 (Cyber Kill Chain & MITRE ATT&CK)