Cloud Detection & Response
Definition
Section titled “Definition”Cloud detection and response is the surveillance of the control plane — the API surface where every cloud resource is created, modified, and destroyed — plus automated response at the same speed. The architectural shift from on-prem: there is no network perimeter to watch, hosts are ephemeral, and the only complete ground truth is the audit log stream (AWS CloudTrail, Azure Activity Logs + Sign-in Logs, GCP Cloud Audit Logs) that records every API call as who did what from where to what. Detection therefore means building on that stream the same discipline BT9 built on endpoint telemetry: baselines, rare-event analysis, identity-aware detections (BT15’s signals join here), and coverage measurement. Response means the inverse insight: in cloud, containment actions are API calls too — disable the access key, detach the policy, revoke sessions, snapshot the instance for forensics, isolate via security-group change — so response can run at machine speed through playbooks instead of waiting for a human to reach a keyboard. The craft’s center: the cloud’s control plane is simultaneously the attacker’s target (credentials, IAM, compute, storage) and your response weapon, and the program that instruments one and not the other is half-blind. Shared responsibility reframed for detection: the provider logs everything, but nobody but you looks for the adversary in it.
Why It Matters
Section titled “Why It Matters”Cloud intrusions run at API speed. A leaked access key (committed to a public repo, scraped from a build log, phished via an OAuth consent) goes from discovery to full-account compromise in minutes: enumerate permissions, escalate through an over-privileged role, create persistence (new federated identity or access key on a second principal), stage data, egress. No malware lands, no endpoint alert fires, and the traditional IR clock — detect, assemble, decide, act — is geometrically too slow. The compensation is structural: every step of that chain writes an audit record, so the cloud gives defenders the most complete forensic surface that has ever existed, provided two conditions hold — the logs are centralized and retained (the default 90-day CloudTrail event history is not a forensics program), and detections actually run on them (BT3’s pipeline, BT9’s engineering).
The failure modes are the standard cloud postures. Logging theater: management events on, data events off — so the object-level reads that constitute exfiltration (T1530) are invisible. Retention rot: audit logs expiring before the average dwell-time discovery, destroying the retro-hunt. Alert spam without identity context: rate-based alerts on ConsoleLogin that fire 200 times a day and get muted — while the one new federated principal created at 3 a.m. sails through unwatched. Response without revocation: an analyst confirms compromise and then… watches the attacker’s session stay alive because nobody remembers that password reset doesn’t kill keys or sessions (BT15’s revocation doctrine applies double in cloud). And drift: a new account spun up last quarter by another team, outside the logging coverage map, is where the incident will happen.
Detailed Explanation
Section titled “Detailed Explanation” CONTROL PLANE (the real perimeter) IAM / roles / keys compute storage federation | | | | +--------+-----------+----+-----+-----------+ v v +------------------------+------------------------+ | AUDIT LOG STREAMS | CloudTrail / Activity | | (management + data | Logs / Cloud Audit | | events, all regions, | Logs | | all accounts) | | +-----------+------------+------------------------+ | BT3 pipeline: normalize, enrich, | join identity (BT15) & threat intel (BT10) v +------------------------+ +---------------------------+ | DETECTIONS (BT9) | | RESPONSE PLAYBOOKS | | rare API for principal | | disable key / detach pol | | IAM changes, new feder | --> | revoke sessions, isolate | | egress from storage | | snapshot for forensics | | root activity, trail | | (machine-speed via API) | | tampering (T1562.001) | +---------------------------+ +------------------------+ | | v v +------------------------+ +------------------------+ | BT4 IR: evidence chain,| | MEASURE (BT20) | | comms, root cause | | coverage, MTTR, drift | +------------------------+ +------------------------+Telemetry architecture decides everything downstream. The non-negotiables: audit logging enabled in every region and every account (org-level trail / management-group policy, not per-team judgment), data events on for storage that matters (S3 object-level, blob access), identity logs joined with resource logs (Azure Sign-in + Activity; the sign-in alone misses what the session did), and retention pushed to the SIEM/data lake with a retention measured in years — incidents surface months after the fact. Multi-account estates centralize into a logging account; detection content inherits BT9’s lifecycle rather than living as console one-offs.
Detection patterns on the control plane sort into a small number of high-yield families. Rare-event-for-principal: an API call a principal has never made (the auditdiff approach below) — the workhorse, because it self-adapts per account and needs no threat feed. Identity-plane events: new access keys, new federated providers, role trust-policy changes, permission grants (T1098) — each is low-frequency and high-signal. Defense impairment: trail stopped, log group deleted, guardduty detector disabled, retention shortened (T1562.001) — always alert, zero tolerance. Data plane: bulk object reads from storage (T1530), egress patterns against DLP baselines. Console vs API asymmetry: a principal that has only ever used the console suddenly running CLI from a new ASN.
Response is playbook engineering (BT4’s process, encoded). Every play has: trigger (a detection), scope queries (what did this identity touch — the audit log makes scope computable: list every API call by that principal in the window), containment actions as API calls (disable keys, revoke sessions, detach policies, isolate compute via security groups), evidence actions (snapshot volumes, export the log window with chain-of-custody hashing — BT5 discipline), and a measured MTTD/MTTR feeding BT20. The fastest play is automatic: disable the leaked key the scanner found — the approval gate is a message, not a meeting.
Common Techniques
Section titled “Common Techniques”| Technique | What it does | Where it fails | Notes |
|---|---|---|---|
| Org-wide audit logging | Every account/region, mgmt + data events | Data events off = blind exfil | Non-negotiable baseline |
| Rare-API-for-principal | Self-adapting per-identity baseline | Cold-start on new principals | auditdiff’s core |
| Identity-plane detection | Keys, federation, trust policy changes | Trust changes seen as routine | Low-frequency, high-signal |
| Impairment alerts | Trail/logging/disabled guards | Muted after FP storm | Zero-tolerance family |
| Storage egress analytics | Bulk reads from object storage | Versioning/masking gaps | T1530 coverage |
| Automated containment | API-based key/session/policy revocation | Revocation gaps (sessions!) | Playbooks, not runbooks |
| Forensic snapshots | Volume/log snapshots on trigger | Cost alarms kill automation | Evidence before teardown |
| Guardrails (preventive) | SCPs/policy blocking dangerous actions | Guardrail sprawl | Prevent ≠ detect; need both |
| Coverage map | Accounts/regions/projects instrumented | Shadow accounts outside map | BT20 metric |
| Federated identity watch | New OIDC/SAML providers alerted | Legit onboarding noise | Peak-signal detection |
Full Working Code
Section titled “Full Working Code”auditdiff — rare-event detection for cloud audit logs: builds a baseline of (principal, event) frequency from a history window, then scores a current window — new principal-event pairs, first-seen source networks, and a curated sensitive-event list get weighted scores, and the output is a triage-ready register with the exact calls that made each principal stand out. Works on any CloudTrail-shaped JSONL (eventName + principal + source IP), which covers the three major clouds’ exports with light normalization.
#!/usr/bin/env python3"""auditdiff — rare-event detection over cloud audit logs (CloudTrail-shaped).
Input: two JSONL files of audit records: baseline.jsonl — a quiet reference window (e.g. prior 30 days) current.jsonl — the window to triage (e.g. last 24h)Record fields used: eventName, principal, source_ip (normalize your exportto these; Azure/GCP exports map with a one-pass jq or 5-line shim).Output: triage register scored by novelty + sensitivity."""import argparseimport csvimport jsonimport sysfrom collections import Counterfrom pathlib import Path
# control-plane calls that are high-signal regardless of frequencySENSITIVE = { "CreateUser", "CreateAccessKey", "DeleteAccessKey", "AttachUserPolicy", "AttachRolePolicy", "PutRolePolicy", "PutUserPolicy", "CreateRole", "UpdateAssumeRolePolicy", "CreateFederationToken", "CreateOpenIDConnectProvider", "CreateSAMLProvider", "PutBucketPolicy", "DeleteTrail", "StopLogging", "UpdateTrail", "DeleteDetector", "DisableGuardDuty", "PutBucketVersioning", "DeleteBucketLifecycle", "CreatePolicyVersion", "SetDefaultPolicyVersion", "AddUserToGroup", "CreateLoginProfile", "UpdateLoginProfile",}NEW_PAIR_WEIGHT = 3.0 # principal-event pair never seen in baselineSENSITIVE_WEIGHT = 4.0 # sensitive-list call in current windowNEW_ASN_WEIGHT = 1.5 # first-seen /8 network for principalREQUIRED = ("eventName", "principal", "source_ip")
def load(path: Path, label: str): if not path.exists(): sys.exit(f"fail-loud: {label} not found: {path}") rows, bad = [], 0 with path.open(encoding="utf-8", errors="replace") as fh: for n, line in enumerate(fh, 1): line = line.strip() if not line: continue try: rec = json.loads(line) except json.JSONDecodeError: bad += 1 continue missing = [k for k in REQUIRED if not rec.get(k)] if missing: bad += 1 continue rows.append({k: str(rec[k]) for k in REQUIRED}) return rows, bad
def asn(ip: str): parts = ip.split(".") if len(parts) == 4: return ".".join(parts[:3]) # /24-ish network bucket return ip
def main(): ap = argparse.ArgumentParser(description="rare-event triage for audit logs") ap.add_argument("baseline", type=Path, help="reference window JSONL") ap.add_argument("current", type=Path, help="window to triage JSONL") ap.add_argument("--out", type=Path, default=Path("auditdiff_out")) args = ap.parse_args()
base_rows, bad1 = load(args.baseline, "baseline") cur_rows, bad2 = load(args.current, "current") if not cur_rows: sys.exit("fail-loud: zero parseable current records — export broken?") if not base_rows: sys.exit("fail-loud: zero parseable baseline records — " "novelty needs a reference window")
base_pairs = Counter((r["principal"], r["eventName"]) for r in base_rows) base_asns = {} for r in base_rows: base_asns.setdefault(r["principal"], set()).add(asn(r["source_ip"]))
cur_pairs = Counter((r["principal"], r["eventName"]) for r in cur_rows) cur_asns = {} for r in cur_rows: cur_asns.setdefault(r["principal"], set()).add(asn(r["source_ip"]))
findings = [] for (principal, event), count in cur_pairs.items(): score, reasons = 0.0, [] if (principal, event) not in base_pairs: score += NEW_PAIR_WEIGHT reasons.append(f"new behavior: {event} never in baseline") if event in SENSITIVE: score += SENSITIVE_WEIGHT reasons.append("sensitive control-plane call") for net in cur_asns.get(principal, ()): if net not in base_asns.get(principal, set()): score += NEW_ASN_WEIGHT reasons.append(f"new source network {net}") break if score > 0: findings.append({"principal": principal, "event": event, "count": count, "score": round(score, 2), "reasons": "; ".join(reasons)})
findings.sort(key=lambda f: -f["score"]) args.out.mkdir(parents=True, exist_ok=True) with (args.out / "triage.csv").open("w", newline="", encoding="utf-8") as fh: w = csv.DictWriter(fh, fieldnames=["score", "principal", "event", "count", "reasons"]) w.writeheader() w.writerows(findings)
hot = [f for f in findings if f["score"] >= 5] print(f"baseline pairs={len(base_pairs)} current pairs={len(cur_pairs)} " f"findings={len(findings)} hot(>=5)={len(hot)}") print(f"unparseable/skipped: baseline={bad1} current={bad2}") for f in findings[:10]: print(f" [{f['score']:>5}] {f['principal']} {f['event']} x{f['count']}") print(f" {f['reasons']}") print(f"register: {args.out / 'triage.csv'}")
if __name__ == "__main__": main()Code Walkthrough
Section titled “Code Walkthrough”The SENSITIVE set (lines 22-28) is curated doctrine, not a dump: it contains exactly the calls that create persistence (keys, login profiles), escalate (policy attachments, policy versions), federate (OIDC/SAML providers), or impair defense (trail stop, detector disable) — one list, maintained like detection content. The weights (lines 29-31) encode triage arithmetic: a sensitive call that is also new-for-principal scores 7 — instant queue-topper; a new-but-benign call scores 3 and waits; a familiar call from a new network scores 1.5 and lands in the hunt bucket rather than the page bucket.
load() (lines 34-54) fails soft per line but loud in aggregate, and requires the three normalized fields — the module’s contract is that your export shim (jq, Logstash, or a five-line script) maps CloudTrail’s userIdentity.arn, Azure’s claims.name, or GCP’s authenticationInfo.principalEmail onto principal before this tool runs; normalization belongs at the pipeline edge, not in every consumer. asn() (lines 57-62) buckets to a /24 — coarse enough to survive residential DHCP churn, fine enough to catch a new hosting provider. The diff loop (lines 88-107) computes novelty per (principal, event) pair against the baseline Counter, adds the sensitivity and network-novelty bonuses, and emits reasons as text — the analyst reads why the score exists without re-querying anything. Output is the sorted register plus a hot-list print (lines 108-125); the >= 5 hot threshold (line 113) is the “someone looks in the next hour” cut, tuned to your on-call reality.
How It Reaches The Target
Section titled “How It Reaches The Target”The leak-to-compromise chain, instrumented end to end. A developer’s AWS key commits to a public repo; within an hour an automated scraper tests it: GetCallerIdentity (who am I?), then enumeration of attached policies, then — finding an over-privileged role — AttachRolePolicy or a trust-policy change, a new access key on a second principal for persistence, and data staged from an S3 bucket. Every one of those calls lands in CloudTrail; the pipeline normalizes into auditdiff’s shape; the register lights up: CreateAccessKey by a principal whose baseline contains only s3:GetObject — score 7, hot. The playbook fires before a human reads the register: disable the leaked key (automatic — the scanner found it, the trigger is unambiguous), then the human work starts from a contained position.
Scope is computable, which is the quiet revolution of cloud IR: one query — every audit event by principal IN (leaked-key identity, new-key identity, affected role) in the window — returns the attacker’s complete action history with timestamps, source IPs, and exact API parameters. Evidence preserves as log-window export plus EBS snapshots with hashes (BT5 chain); identity cleanup follows BT15’s revocation doctrine (keys and sessions and the persistence artifacts — delete the added key, the trust-policy change, the federation provider if created); the post-incident review feeds BT9 (why did the role allow AttachRolePolicy to anyone?) and BT11 (why was the over-privileged role on a laptop?). The measure loop closes in BT20: time-from-leak-to-key-disable (here, minutes), coverage map updated, and the preventive fix (SCP blocking policy-attach on that OU, secret scanning in CI) graduates from detection lesson to guardrail — which is how the program’s preventive layer grows from real incidents instead of from theory.
Exploitation Chain
Section titled “Exploitation Chain”- Initial access — leaked access key from a repo/build log, phished OAuth consent, or bought credentials
- Discovery — GetCallerIdentity, permission enumeration, resource listing (T1526)
- Privilege escalation — AttachRolePolicy / trust-policy change / PassRole into an over-privileged role
- Persistence — new access key on a second principal, new federated provider, or a consent grant (T1098)
- Collection — bulk object reads from storage (T1530), snapshots copied cross-region
- Impairment (optional) — StopLogging/DeleteTrail if the attacker reads the room (T1562.001)
- Detection — auditdiff hot register: sensitive + never-seen calls by a quiet principal
- Response — automatic key disable, computable scope from audit history, snapshot evidence, identity cleanup (BT15)
Building Your Own Tool
Section titled “Building Your Own Tool”The rare-event detector finds the moment; iamchain finds the path — how a principal could get from its current permissions to admin-class actions through the escalation primitives (policy attach, role trust, PassRole). It reads an IAM snapshot JSON and answers the configuration question BT9’s detections can’t: which identities are one wrong hire away from domain admin?
Design decisions
Section titled “Design decisions”Graph over heuristics: nodes are principals and managed policies; edges are the escalation actions themselves (AttachUserPolicy/AttachRolePolicy grant a node; CreatePolicyVersion/SetDefaultPolicyVersion rewrite a node; UpdateAssumeRolePolicy + PassRole patterns reach roles). BFS from every user to the “admin actions” target set (the SENSITIVE list from auditdiff, reused deliberately — same doctrine in both tools). Paths report edges, not just reachability, because the remediation is always “remove edge X”. Depth-capped at 3 hops — deeper chains are configuration archaeology, not triage. Fail-loud on schema drift so a half-written export can’t produce a false “no paths” comfort.
iam-snapshot.json # the export iamchain consumes { “users”: [ {“name”: “ci-bot”, “attached”: [“S3FullAccess”], “can_pass_roles”: [“deploy-role”]} ], “roles”: [ {“name”: “deploy-role”, “trust”: [“ci-bot”], “attached”: [“PowerUserAccess”]}, {“name”: “admin-role”, “trust”: [“deploy-role”], “attached”: [“AdministratorAccess”]} ] }
Full source
Section titled “Full source”#!/usr/bin/env python3"""iamchain — find IAM privilege-escalation paths in an IAM snapshot.
Usage: python3 iamchain.py iam-snapshot.json [--depth 3]Output: every user -> admin-actions path, as edges to remove."""import argparseimport jsonimport sysfrom collections import dequefrom pathlib import Path
ADMIN_ACTIONS = {"AdministratorAccess", "admin", "*"}ESCALATION = { "AttachUserPolicy", "AttachRolePolicy", "PutUserPolicy", "PutRolePolicy", "CreatePolicyVersion", "SetDefaultPolicyVersion", "UpdateAssumeRolePolicy", "CreateAccessKey",}
def load(path: Path): if not path.exists(): sys.exit(f"fail-loud: snapshot not found: {path}") try: snap = json.loads(path.read_text(encoding="utf-8")) except json.JSONDecodeError as e: sys.exit(f"fail-loud: bad JSON ({e})") for key in ("users", "roles"): if key not in snap: sys.exit(f"fail-loud: snapshot missing '{key}'") return snap
def is_admin(policy_name: str): return any(a in policy_name.lower() for a in ("administrator", "admin", "*"))
def main(): ap = argparse.ArgumentParser(description="IAM escalation path finder") ap.add_argument("snapshot", type=Path) ap.add_argument("--depth", type=int, default=3) args = ap.parse_args()
snap = load(args.snapshot) users = {u["name"]: u for u in snap["users"]} roles = {r["name"]: r for r in snap["roles"]}
paths = [] for uname, user in users.items(): # BFS state: (node, kind, path_edges) start = [(uname, "user", [])] seen = {(uname, "user")} queue = deque(start) while queue: node, kind, edges = queue.popleft() if len(edges) > args.depth: continue attached = [] if kind == "user": attached = users[node].get("attached", []) elif kind == "role": attached = roles[node].get("attached", []) if any(is_admin(p) for p in attached): paths.append((uname, edges)) continue # escalation edge: user/role can pass or assume a role targets = [] if kind == "user": targets += [(r, "role", "PassRole/AssumeRole") for r in users[node].get("can_pass_roles", []) if r in roles] if kind == "role": targets += [(r, "role", "PassRole/AssumeRole") for r in roles[node].get("can_pass_roles", []) if r in roles] # role trust: who can assume this role (already assumed) - # instead: roles trusting THIS node let it reach them targets += [(r["name"], "role", "AssumeRole(trust)") for r in roles.values() if node in r.get("trust", [])] for tname, tkind, edge in targets: if (tname, tkind) not in seen: seen.add((tname, tkind)) queue.append((tname, tkind, edges + [(node, edge, tname)]))
if not paths: print("no escalation paths found within depth " f"{args.depth} — verify the snapshot is complete") return print(f"escalation paths to admin: {len(paths)}") for uname, edges in paths: chain = " -> ".join(f"{a} --[{b}]--> {c}" for a, b, c in edges) or "(already admin)" print(f" {uname}: {chain}") print("remediation = remove the labeled edges (trust/PassRole/attachment)")
if __name__ == "__main__": main()Walkthrough
Section titled “Walkthrough”load() (lines 20-31) fails loud on both malformed JSON and missing sections — a partial export must never yield a confident “no paths”. is_admin() (lines 34-37) treats any policy whose name contains administrator/admin/* as the target class — snapshot exports vary in naming, and the coarse match errs toward more findings, which is the safe direction for a config audit. The BFS (lines 55-96) tracks (node, kind) in seen to avoid cycles through shared roles, carries the edge list in the queue state so each found path is self-describing, and depth-caps at the CLI flag (line 50) — triage depth, not archaeology. Two escalation edge families exist: explicit can_pass_roles (the PassRole/AssumeRole primitive, lines 71-79) and role trust entries (line 80-83 — if deploy-role trusts ci-bot, the bot reaches the role’s permissions). Output prints every path as its edge chain (lines 100-104) with the remediation phrasing built in: each [label] in the chain is a specific trust relationship or attachment to delete, which turns the audit finding into a one-line ticket.
Compilation and test procedure
Section titled “Compilation and test procedure”run against the sample snapshot shipped in the module docs
Section titled “run against the sample snapshot shipped in the module docs”$ python3 iamchain.py iam-snapshot.json —depth 3 escalation paths to admin: 1 ci-bot: ci-bot —[PassRole/AssumeRole]—> deploy-role —[AssumeRole(trust)]—> admin-role remediation = remove the labeled edges (trust/PassRole/attachment)
depth-1 view for the sprint planning meeting (edge count only)
Section titled “depth-1 view for the sprint planning meeting (edge count only)”$ python3 iamchain.py iam-snapshot.json —depth 1 escalation paths to admin: 0 no escalation paths found within depth 1 — verify the snapshot is complete
schema drift must fail loud, not comfort
Section titled “schema drift must fail loud, not comfort”$ echo ’{“users”: []}’ > broken.json $ python3 iamchain.py broken.json; echo “exit=$?” fail-loud: snapshot missing ‘roles’ exit=1
Detection considerations
Section titled “Detection considerations”iamchain is preventive analysis, but its output seeds detection directly: every edge it reports should have a corresponding auditdiff sensitivity (AttachRolePolicy, UpdateAssumeRolePolicy are already in SENSITIVE), so a path that cannot be removed this sprint is at least watched. Two operational traps: snapshot timing (paths found in a Monday export may be gone by Wednesday — automate the export daily and the tool runs on fresh truth), and completeness (roles outside the snapshot’s scope produce false comfort — the coverage map from the main module applies to configuration exports exactly as it does to logs).
Improvements to add
Section titled “Improvements to add”- Ingest resource-based policies (bucket policies, KMS key policies) as additional edges — cross-principal access through resources is a classic blind spot.
- Weight paths by blast radius (admin targets touching logging/billing separate from compute) to sequence remediation.
- Diff snapshots day-over-day and alert on new edges — configuration drift as a detection stream.
- Emit a per-user “one action from admin” list — the queue for the quarter’s IAM cleanup.
- Validate against AWS Access Analyzer findings when running on AWS, joining external analysis with internal graph.
Detection — Defender View
Section titled “Detection — Defender View”| Signal | Source | ID | Notes |
|---|---|---|---|
| Trail/logging stopped | CloudTrail/Activity | StopLogging | T1562.001, zero tolerance |
| New access key / login profile | Audit stream | CreateAccessKey | Persistence primitive |
| New federation provider | Audit stream | CreateSAMLProvider/OIDC | Peak-signal event |
| Trust policy change | Audit stream | UpdateAssumeRolePolicy | iamchain edge live |
| Rare API for principal | auditdiff | score >= 5 | Hot register |
| Storage bulk read | Data events | T1530 | Egress family |
| Root account activity | Audit stream | ConsoleLogin (root) | Should be near-zero |
| GuardDuty/detector disable | Audit stream | DeleteDetector | Impairment family |
| Console→CLI asymmetry | Sign-in + audit | new ASN + API burst | Hijacked session tell |
| Cloud CLI from endpoint | Sysmon | EID 1 | aws.exe/az.exe on servers |
Evasion Techniques
Section titled “Evasion Techniques”Adversaries adapt to control-plane visibility deliberately. Region hopping: activity in regions the logging coverage map missed (new AWS regions default-off in older trail configs) — the org-wide trail doctrine exists because of this. Federation blending: instead of creating visible new users, attackers mint tokens through a legitimate SAML/OIDC provider or assume roles through existing trust chains — the audit log shows your federated infrastructure being used, which rare-event baselines catch only if principal identity is normalized carefully (role session names, not just role ARNs). Living off the data plane: management-event-only logging leaves object reads invisible; attackers stage exfil as ordinary-looking S3 traffic, or use sync tools whose API pattern matches the owner’s own scripts. Impairment timing: StopLogging fires after persistence is in place — defenders must detect on the impairment event itself (zero tolerance) and treat the logging gap as a compromise indicator, not an outage. Ephemeral compute: spot instances and serverless functions run the noisy step (credential stuffing, scanning) for minutes and vanish; the audit record persists but the source does not — response playbooks must not depend on the attacker’s infrastructure surviving. Root avoidance: everything goes through IAM principals because defenders watch root like a hawk — which inverts nicely: root near-zero is a baseline, and any root event is worth a look.
⚠️ What caught me: our cloud detections were tuned, our trail was org-wide, and we still spent three weeks investigating an incident whose first malicious API call was actually ten weeks old — because the logging account’s retention was 30 days, the detections only ran on the live stream, and nobody had ever loaded history into the lake. The attacker’s CreateAccessKey sat in an S3 bucket we technically owned but observably ignored. Retro-hunt is not a nice-to-have in cloud: the audit log’s completeness is worthless if your detection surface is only the last 24 hours. We now load history continuously and every new detection rule backfills against 12 months before it goes live — three retro-fires on its first week taught us more than six months of forward alerts.
Lab Practice
Section titled “Lab Practice”Build a two-account lab (a “workload” account and a “logging” account, or one account with two profiles if budget-limited). Instrumentation: enable an org trail with management events; enable data events on one S3 bucket; verify the stream lands in the logging account. Baseline: run a scripted hour of normal activity (a CLI profile doing GetObject/ListBuckets) to build baseline.jsonl. Attack simulation: with a second key, perform the chain — GetCallerIdentity, CreateAccessKey, AttachUserPolicy, one bulk GetObject sweep; capture as current.jsonl. Detect: run auditdiff; confirm the sensitive+new pairs top the register with score 7. Respond: write the containment playbook as actual CLI steps — disable key, delete the added key, revert the policy — and time yourself; that time is your manual MTTR baseline. Audit: run iamchain against a snapshot of the lab’s IAM; remove one escalation edge and re-run. Debrief: which detection fired that a host-based stack would have missed entirely, and what does that imply about the endpoint-only SOC?
Interview Questions
Section titled “Interview Questions”- Technical: Why is the control-plane audit stream — not host telemetry — the primary detection surface in cloud, and what does data-events-on add that management events miss?
- Technical: Walk the auditdiff scoring model — why does a sensitive call that is new-for-principal score 7, and what does the /24 ASN bucket trade off?
- Scenario: CloudTrail shows StopLogging from an unexpected principal at 3 a.m., then silence. Walk your first 30 minutes: what do you assume, what do you check, what do you revoke?
- Scenario: An auditdiff hot finding shows AttachRolePolicy from a principal whose baseline is read-only S3. The identity owner says “that’s our CI bot, it’s fine.” Diagnose properly — what do you check before accepting that?
- Conceptual: Why does password reset not equal containment for cloud identities, and what does complete revocation require (BT15’s doctrine applied)?
- Deep-dive: Design cloud detection & response for a 40-account AWS estate: logging architecture, detection families with examples, response playbook automation with approval gates, and the BT20 metrics.
- Practical: You inherit a cloud estate with default 90-day history, data events off, and alerts only on root login. What do you fix in week one, and what do you schedule?
Related Modules
Section titled “Related Modules”BT3 (SIEM Engineering & Log Pipelines) · BT4 (Incident Response Process) · BT9 (Detection Engineering & Sigma) · BT15 (Identity & MFA Defense) · H9 (IoT Cloud & Mobile Attack Surface)