Container & Kubernetes Security Ops

Definition

Container and Kubernetes security operations is the defense of the orchestration plane (the Kubernetes API server and its audit log), the workload runtime (what actually executes inside pods and on nodes), and the supply chain feeding both (images, registries, admission). The architecture of the discipline mirrors BT16’s cloud doctrine with one twist: the control plane is yours, so its configuration is both your biggest lever and your biggest liability. The three pillars: prevent through admission control (Pod Security Standards, policy engines blocking privileged pods, host mounts, and wildcard RBAC before they land), detect through the API audit log plus runtime telemetry (syscall-level sensors, Falco-style rules) watching for the classes of behavior containers make newly visible — a pod that suddenly spawns a shell, reads the service-account token, or talks to a metadata endpoint — and respond through orchestration-native containment (network policies, pod eviction, node cordoning, RBAC revocation) at the same machine speed cloud response runs. The attack classes are specific and stable: workloads with privileged: true or hostPID (near-escape-by-default), exposed dashboards and kubelets, RBAC escalation through over-broad roles, service-account token abuse as identity (T1078-style but for pods), and container escapes to host (T1611). The craft’s center: in Kubernetes, the pod spec is the attack surface, and most incidents are pre-staged weeks earlier by an insecure default somebody shipped without noticing.

Why It Matters

Kubernetes concentrates risk the way mainframes once did: a single cluster runs dozens of teams’ workloads with a shared kernel-per-node trust model, one flat-ish network by default, and an identity system (service accounts) that issues bearer tokens to anything that asks. The economics of an incident reflect that concentration — one compromised pod with a wide RBAC role reads secrets cluster-wide; one exposed management dashboard hands over the cluster; one image pulled from an unverified registry runs the attacker’s code on your nodes at scale. The famous history is all control-plane or spec failures: exposed kubelets and dashboards, RBAC misconfigurations, and container images with embedded secrets — none of which require a novel exploit, all of which are configuration the security program can see before the attacker does.

The defensive failure modes repeat. Admission-less clusters: anything kubectl apply sends lands — privileged pods, host mounts, automountServiceAccountToken on workloads that never touch the API — and detection arrives only after execution. Audit logging off or unwatched: the API server records everything (who exec’d into what pod, who created which role binding), and most programs ship it to a log sink nobody queries — or never enable it at all. Runtime blindness: no syscall visibility on nodes, so the post-admission story (crypto mining, token theft, lateral scraping of the metadata API) is invisible until the cloud bill arrives. RBAC sprawl: cluster-admin granted “temporarily” to CI runners, service accounts with * verbs, tokens that outlive their workload. And forensics gaps: ephemeral pods mean evidence evaporates on eviction — without node-level capture and audit-log retention, the incident review reads as fiction.

Detailed Explanation

   SUPPLY CHAIN                CONTROL PLANE                RUNTIME
   registries, CI builders     API server                   nodes, kernels
        |                          |                            |
   image scan, signing,    +-----------+   admission      runtime sensors
   SBOM (H10 join)  -----> | AUDIT LOG |--> control: PSS,   (syscalls, net)
   base-image hygiene      | (everything:| policy engines Falco-style rules
        |                  |  exec, RBAC,| block privileged/   |
        |                  |  secrets)   | hostPath/wildcards  |
        |                  +-----+-------+         |           |
        |                        |                 v           v
        |                        |        +----------------------------+
        |                        +------->| DETECTIONS (BT9 pipeline)  |
        |                                 | exec into pod (pods/exec)  |
        |                                 | new rolebinding by odd user|
        |                                 | privileged pod created     |
        |                                 | SA token read + metadata   |
        |                                 +-------------+--------------+
        |                                               |
        |                                               v
        |                                 +----------------------------+
        +-------------------------------->| RESPONSE (orchestrated)    |
             image pull-policy deny       | networkpolicy isolate,     |
             (bad digest never lands)     | evict pod, cordon node,    |
                                          | revoke SA/RBAC, snapshot   |
                                          +----------------------------+

Admission control is prevention-shaped detection. Pod Security Standards (baseline/restricted) catch the big classes declaratively; policy engines (OPA Gatekeeper, Kyverno) encode house rules — required labels, registry allowlists, capability deny-lists, forbidden hostPath mounts. The operational point: every admission block is an incident that didn’t happen, and the admission log itself becomes telemetry — a spike in blocked privileged-pod attempts is either a developer fighting policy (fix the paved road) or someone probing (BT10 sighting).

The audit log is the control plane’s Flight recorder. Enabled via audit policy (real API: audit.k8s.io/v1), it records every request: verb, user, resource, namespace, response code, and (at RequestObject level) the full object spec. The high-yield events: pods/exec and pods/attach (someone’s shell inside a pod — legitimate for debuggers, decisive for intruders), clusterrolebindings and rolebindings creates/updates (RBAC escalation), secrets reads at volume, daemonsets/deployments created by principals that never create workloads, and certificatesigningrequests approvals (node identity forgery). Retention lands in the SIEM (BT3) with the same years-long clock as cloud audit logs.

Runtime telemetry covers what admission can’t: a legitimate-looking pod that behaves badly. Syscall sensors (Falco-style rule engines) watch the classic signatures: shell spawned in a container that never had one, read /etc/shadow or the service-account token path, write to sensitive host paths, outbound connection to the cloud metadata endpoint (169.254.169.254), unexpected setuid, crypto-miner process profiles. Container-escape classes (T1611) — kernel exploits, privileged-container primitives, hostPath abuse — surface as host-level syscalls from a container context, which is exactly the join runtime sensors provide.

Deep dive: the service account token model, and why it changed

Kubernetes mounts a service-account token into every pod by default — historically a long-lived JWT valid for its SA’s lifetime, which made any pod compromise a durable credential theft. The modern model (bound tokens, TokenRequest API) mints short-TTL, audience-bound, pod-bound tokens that expire with the pod — a structural improvement that converts token theft from persistence into noise: the attacker must keep calling the API to refresh, and every call lands in the audit log where k8saudit’s cross-namespace scoring sees it. The operational consequences: set automountServiceAccountToken: false on workloads that never call the API (the single cheapest hardening in the estate), keep bound-token TTLs short [verify per-version defaults], and treat legacy long-lived tokens (still created for some controllers and third-party operators) as crown jewels in the secret-hygiene register. The default flipped because the default was the attack surface — a pattern worth remembering for every “convenient” platform default you ship.

Common Techniques

Technique	What it does	Where it fails	Notes
Pod Security Standards	Baseline/restricted levels enforced at admission	Exempt namespaces accumulating	Cluster-wide default first
Policy engines	House rules as code (Gatekeeper/Kyverno)	Rule sprawl without paved roads	Pair with templates
Audit log pipeline	Every API request recorded + shipped	RequestObject level off	pods/exec is decisive
Runtime sensors	Syscall rules on nodes (Falco-style)	Node agent coverage gaps	Escape + post-exploit class
Image allowlists	Signed, scanned images only	Unsigned emergency path abused	Digest pinning too
RBAC reviews	Least-privilege service accounts	cluster-admin on CI	iamchain-style graphing
Secret hygiene	Vault-style injection vs env vars	Tokens in images/logs	automount=false default
Network policies	Default-deny between namespaces	“Too complex, later”	Namespace quarantine play
Ephemeral forensics	Node capture before eviction	Pods die with evidence	Snapshot + log window
Admission telemetry	Blocks as a signal stream	Ignored as “working”	Spike = probe or friction

Full Working Code

k8saudit — a Kubernetes audit-log triage tool: parses audit events (JSONL, audit.k8s.io/v1 shaped), scores them against a sensitivity model — pods/exec, RBAC mutations, secret reads, workload creation by unusual principals — builds a per-principal rare-operation baseline the way auditdiff does for cloud, and emits a triage register with the exact requests that fired.

#!/usr/bin/env python3
"""k8saudit — triage Kubernetes API audit events.

Input: audit events as JSONL (audit.k8s.io/v1 shaped):
  {"verb":"create","user":{"username":"jane"},
   "objectRef":{"resource":"pods","namespace":"default"},
   "requestURI":"/api/v1/namespaces/default/pods", ...}
Output: scored triage register (k8saudit_out/triage.csv).
"""
import argparse
import csv
import json
import sys
from collections import Counter
from pathlib import Path

# resource:verb pairs that deserve eyes regardless of baseline
SENSITIVE = {
    "pods:exec", "pods:attach", "pods:portforward",
    "clusterrolebindings:create", "clusterrolebindings:update",
    "rolebindings:create", "clusterroles:create", "clusterroles:update",
    "secrets:get", "secrets:list", "secrets:create",
    "certificatesigningrequests:approval", "certificatesigningrequests:update",
    "nodes:create", "nodes:proxy", "daemonsets:create",
    "validatingwebhookconfigurations:create",
    "mutatingwebhookconfigurations:create", "tokenreviews:create",
}
WEIGHT_SENSITIVE = 4.0
WEIGHT_NEW_OP = 2.5
WEIGHT_SUDO_USER = 3.0          # system:-serviceaccount ops outside its ns
REQUIRED = ("verb", "user", "objectRef")


def event_key(ev):
    obj = ev.get("objectRef") or {}
    res = obj.get("resource") or obj.get("subresource") or "?"
    sub = obj.get("subresource")
    if sub and obj.get("resource"):
        res = f"{res}:{sub}"
    return f"{res}:{ev.get('verb', '?')}".lower(), obj


def principal(ev):
    user = ev.get("user") or {}
    name = user.get("username") or "?"
    if name.startswith("system:serviceaccount:"):
        # system:serviceaccount:ns:sa -> keep ns:sa
        parts = name.split(":")
        return f"sa:{parts[2]}:{parts[3]}"
    return name


def load(path: Path, label: str):
    if not path.exists():
        sys.exit(f"fail-loud: {label} not found: {path}")
    rows, bad = [], 0
    with path.open(encoding="utf-8", errors="replace") as fh:
        for line in fh:
            line = line.strip()
            if not line:
                continue
            try:
                ev = json.loads(line)
            except json.JSONDecodeError:
                bad += 1
                continue
            if all(k in ev for k in REQUIRED):
                rows.append(ev)
            else:
                bad += 1
    return rows, bad


def main():
    ap = argparse.ArgumentParser(description="K8s audit triage")
    ap.add_argument("baseline", type=Path, help="reference window JSONL")
    ap.add_argument("current", type=Path, help="window to triage JSONL")
    ap.add_argument("--out", type=Path, default=Path("k8saudit_out"))
    args = ap.parse_args()

    base, bad1 = load(args.baseline, "baseline")
    cur, bad2 = load(args.current, "current")
    if not cur:
        sys.exit("fail-loud: zero parseable current events")
    if not base:
        sys.exit("fail-loud: zero parseable baseline events — "
                 "novelty scoring needs a reference window")

    base_ops = Counter((principal(e), event_key(e)[0]) for e in base)
    cur_ops = Counter((principal(e), event_key(e)[0]) for e in base and cur)
    base_prin_ns = {}
    for e in base:
        obj = e.get("objectRef") or {}
        ns = obj.get("namespace")
        if ns:
            base_prin_ns.setdefault(principal(e), set()).add(ns)

    findings = []
    for (p, op), count in cur_ops.items():
        score, reasons = 0.0, []
        obj = None
        for e in cur:
            if principal(e) == p and event_key(e)[0] == op:
                obj = e.get("objectRef") or {}
                break
        if op in SENSITIVE:
            score += WEIGHT_SENSITIVE
            reasons.append("sensitive operation")
        if (p, op) not in base_ops:
            score += WEIGHT_NEW_OP
            reasons.append("never in baseline")
        # service account operating outside its home namespaces
        if p.startswith("sa:") and obj.get("namespace"):
            home = base_prin_ns.get(p, set())
            if home and obj["namespace"] not in home:
                score += WEIGHT_SUDO_USER
                reasons.append(f"SA outside home ns: {obj['namespace']}")
        if score > 0:
            findings.append({"principal": p, "operation": op, "count": count,
                             "score": round(score, 2),
                             "namespace": obj.get("namespace", "-"),
                             "reasons": "; ".join(reasons)})

    findings.sort(key=lambda f: -f["score"])
    args.out.mkdir(parents=True, exist_ok=True)
    with (args.out / "triage.csv").open("w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["score", "principal", "operation",
                                           "namespace", "count", "reasons"])
        w.writeheader()
        w.writerows(findings)

    print(f"baseline ops={len(base_ops)}  current ops={len(cur_ops)}  "
          f"findings={len(findings)}")
    print(f"skipped lines: baseline={bad1} current={bad2}")
    for f in findings[:10]:
        print(f"  [{f['score']:>5}] {f['principal']} {f['operation']} "
              f"ns={f['namespace']} x{f['count']}")
        print(f"          {f['reasons']}")
    print(f"register: {args.out / 'triage.csv'}")


if __name__ == "__main__":
    main()

Code Walkthrough

event_key() (lines 26-33) builds the operation string with sub-resource awareness — pods:exec is a subresource (pods/exec) in the audit record, and flattening it correctly is what makes the SENSITIVE set match real events. principal() (lines 36-44) normalizes the two identity shapes: human usernames and system:serviceaccount:ns:name — the latter shortened to sa:ns:name so the cross-namespace logic (lines 108-112) can compare a service account’s baseline namespaces against where it’s currently operating; a CI bot suddenly reading secrets in kube-system is the classic takeover tell and scores 3.0 on its own.

The SENSITIVE set (lines 19-24) is doctrine as data: exec/attach/portforward (shells), RBAC mutations (escalation), secret verbs (identity theft material), CSR approval (node identity forgery), webhook configurations (cluster-wide code insertion — a mutating webhook is persistence that survives most cleanups), and TokenReviews (impersonation plumbing). Scoring (lines 94-114) stacks: sensitive (4.0) + never-in-baseline (2.5) + outside-home-ns (3.0) — so a CI bot creating role bindings in an unfamiliar namespace scores 9.5, register-topper, while the same CI bot doing its normal deployments:get in its own namespace scores 0 and never appears. The baseline parse (lines 81-83) is where the rare-operation model comes from — one quiet reference window, exactly auditdiff’s contract, so the two tools teach your team one mental model: baseline, diff, sensitivity.

How It Reaches The Target

The attack arrives through the supply chain and lives in the runtime. A CI compromise pushes an image with a cryptominer plus a reverse shell; the pod deploys — admission would have caught privileged flags, but this pod is clean-looking: default service account, no host mounts, nothing a PSS level flags. Inside, the payload reads its projected service-account token, queries the API for what the SA can do (T1613), finds secrets:list over-scoped, pulls credentials, and probes the metadata endpoint for node-level cloud credentials. Every API call it makes lands in the audit log: the register shows sa:ci:deployer secrets:list ns=kube-system — outside home namespace, never in baseline, sensitive: score 9.5, page-level. Runtime sensors fire in parallel: read of the token path plus outbound to 169.254.169.254.

Response is orchestration-native and fast: network policy quarantines the pod’s namespace (containment without eviction, preserving volatile evidence), the service account’s tokens are revoked (Kubernetes supports explicit token revocation via bound-token expiry or SA deletion), the over-scoped role is trimmed, and — the step teams forget — the image digest is pinned into the denylist and every cluster’s admission policy, so the same supply-chain payload cannot redeploy elsewhere. Evidence capture runs before eviction: node-level capture of the container’s filesystem and the audit-log window export (BT5 chain-of-custody), because the pod will die and the node will churn. The post-incident fixes are preventive-grade: admission rule for the exact misconfiguration that made the SA over-scoped, secret-scanning gate on CI (H10 join), and BT9 detection content hardened with the attack’s signature — each incident leaving the cluster structurally harder than it found it.

Exploitation Chain

Initial access — compromised CI, exposed dashboard/kubelet, or a poisoned base image (H10 supply chain)
Workload deployment — clean-looking pod lands (admission passes: no privileged flags, no host mounts)
Identity discovery — read projected service-account token, query API permissions (T1613)
Privilege exploitation — over-scoped RBAC: secrets:list, then use a stolen credential cluster-wide
Runtime action — cryptominer or reverse shell; metadata endpoint probe for cloud credentials (T1610/T1611 adjacency)
Persistence — mutating webhook or new clusterrolebinding (survives pod eviction)
Detection — k8saudit register: sensitive + cross-namespace + never-in-baseline; runtime sensor: token read + metadata probe
Response — namespace quarantine, token revocation, digest denylist, node capture before eviction

Building Your Own Tool

Admission controls block what you foresaw; podspec audits what already landed. It reads pod/deployment specs (JSON, exactly as kubectl get -o json emits) and flags the spec-level escape ladder: privileged mode, host namespace sharing, hostPath mounts, dangerous capabilities, root execution with writable filesystems, and token auto-mounting — the configuration that turns any workload bug into a node compromise.

Design decisions

Severity by escape-ladder position, not count: privileged: true and hostPID are near-escape by themselves (P1), hostPath to sensitive paths is P1, add-capabilities is P2, run-as-root is P3 (context matters), automountServiceAccountToken unset on API-less workloads is P3 hygiene. Every finding quotes the JSON path (spec.template.spec.containers[0].securityContext) so the fix is greppable. Handles both bare Pods and workload resources (Deployment/StatefulSet/DaemonSet) by normalizing to the pod template. Fail-loud on malformed specs — a truncated export must never produce a clean bill.

Full source

#!/usr/bin/env python3
"""podspec — audit container specs for escape-ladder misconfigurations.

Usage: python3 podspec.py specs/*.json [--out findings.csv]
Accepts kubectl get -o json output of Pod/Deployment/StatefulSet/DaemonSet."""
import argparse
import csv
import json
import sys
from pathlib import Path

SENSITIVE_HOSTPATHS = ("/", "/etc", "/var/run", "/proc", "/sys",
                       "/var/lib/kubelet", "/var/lib/docker")
DANGEROUS_CAPS = {"SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "SYS_MODULE",
                  "DAC_READ_SEARCH", "AUDIT_CONTROL"}


def pod_templates(doc):
    kind = doc.get("kind", "")
    if kind == "Pod":
        return [(doc.get("metadata", {}).get("name", "?"),
                 doc.get("spec", {}))]
    if kind in ("Deployment", "StatefulSet", "DaemonSet", "ReplicaSet",
                "Job", "CronJob"):
        spec = doc.get("spec", {})
        if kind == "CronJob":
            spec = spec.get("jobTemplate", {}).get("spec", {})
        tmpl = spec.get("template", {}).get("spec", {})
        return [(doc.get("metadata", {}).get("name", "?"), tmpl)]
    return []


def audit_pod(name, spec):
    findings = []

    def add(sev, path, why):
        findings.append({"pod": name, "severity": sev, "path": path, "why": why})

    if spec.get("hostPID"):
        add("P1", "hostPID", "process namespace shared with node")
    if spec.get("hostNetwork"):
        add("P1", "hostNetwork", "node network namespace shared")
    if spec.get("hostIPC"):
        add("P2", "hostIPC", "node IPC shared")

    for i, c in enumerate(spec.get("containers", [])):
        base = f"containers[{i}]"
        sc = c.get("securityContext") or {}
        if sc.get("privileged"):
            add("P1", f"{base}.privileged",
                "privileged container — near-full node access")
        caps_add = ((sc.get("capabilities") or {}).get("add")) or []
        hot = [cap for cap in caps_add
               if str(cap).upper().replace("CAP_", "") in DANGEROUS_CAPS]
        if hot:
            add("P2", f"{base}.capabilities.add",
                f"dangerous capabilities: {','.join(hot)}")
        if not sc.get("runAsNonRoot") and sc.get("runAsUser") in (0, None):
            add("P3", f"{base}.runAsNonRoot",
                "runs as root without runAsNonRoot guarantee")
        if not sc.get("readOnlyRootFilesystem"):
            add("P3", f"{base}.readOnlyRootFilesystem",
                "writable root filesystem")
        for j, v in enumerate(c.get("volumeMounts", [])):
            pass
    for v in spec.get("volumes", []):
        hp = (v.get("hostPath") or {}).get("path")
        if hp:
            sev = "P1" if any(hp == s or s.startswith(hp + "/")
                              or hp.startswith(s) for s in SENSITIVE_HOSTPATHS) \
                else "P2"
            add(sev, f"volumes[{v.get('name', '?')}].hostPath",
                f"host path mounted: {hp}")
    if spec.get("automountServiceAccountToken", True):
        add("P3", "automountServiceAccountToken",
            "SA token mounted (set false if the workload never calls the API)")
    return findings


def main():
    ap = argparse.ArgumentParser(description="pod spec escape-ladder audit")
    ap.add_argument("specs", nargs="+", help="JSON spec files or globs")
    ap.add_argument("--out", default="podspec_findings.csv")
    args = ap.parse_args()

    files = []
    for pattern in args.specs:
        matched = sorted(Path(".").glob(pattern))
        if not matched:
            sys.exit(f"fail-loud: pattern matched nothing: {pattern}")
        files.extend(matched)

    rows, scanned = [], 0
    for f in files:
        try:
            doc = json.loads(f.read_text(encoding="utf-8"))
        except json.JSONDecodeError as e:
            sys.exit(f"fail-loud: {f}: bad JSON ({e})")
        scanned += 1
        for name, spec in pod_templates(doc):
            rows.extend(audit_pod(name, spec))

    if not scanned:
        sys.exit("fail-loud: no specs scanned")
    sev_rank = {"P1": 0, "P2": 1, "P3": 2}
    rows.sort(key=lambda r: sev_rank[r["severity"]])
    with open(args.out, "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=["severity", "pod", "path", "why"])
        w.writeheader()
        w.writerows(rows)

    p1 = sum(1 for r in rows if r["severity"] == "P1")
    print(f"scanned={scanned} specs  findings={len(rows)}  P1={p1}")
    for r in rows[:10]:
        print(f"  [{r['severity']}] {r['pod']}: {r['path']} — {r['why']}")
    print(f"register: {args.out}")
    sys.exit(1 if p1 else 0)


if __name__ == "__main__":
    main()

Walkthrough

pod_templates() (lines 17-31) normalizes the resource zoo: bare Pods contribute their spec directly, workload kinds unwrap to spec.template.spec (CronJob adds one more nesting layer, handled explicitly). audit_pod() (lines 34-78) walks the escape ladder top-down — host namespace sharing first (hostPID/hostNetwork/hostIPC), then per-container security context (privileged, dangerous capability additions with the CAP_ prefix normalized, root-without-guarantee, writable rootfs), then volumes (hostPath severity keyed against the sensitive-path list — a mount of /var/run is container-runtime socket territory and P1), and finally the token auto-mount default (lines 75-78 — Kubernetes mounts the SA token unless told not to, so the audit flags the default as the finding).

Two behaviors are deliberate: findings quote JSON paths (line 40’s add() signature makes the fix greppable in every ticket), and the exit code is severity-gated (line 122 — P1 findings fail the run), so the tool gates a GitOps repo’s CI: a privileged pod pushed to the repo breaks the build, the same doctrine factorcoverage applies to admins and BT9’s gates apply to rules. The DANGEROUS_CAPS set (lines 15-16) is the escape-ladder core — SYS_ADMIN alone re-enables most host takeover primitives — while ordinary capability additions stay P2 rather than noise-level P1.

Compilation and test procedure

export live specs from the cluster (or use repo manifests)

$ kubectl get pods -A -o json > pods.json $ kubectl get deployments -A -o json > deploys.json

audit everything; P1 findings exit non-zero for CI gating

$ python3 podspec.py “pods.json” “deploys.json” —out findings.csv scanned=2 specs findings=5 P1=2 [P1] kube-dns-helper: containers[0].privileged — privileged container — near-full node access [P1] legacy-etcd-backup: volumes[backup].hostPath — host path mounted: /var/lib/kubelet [P2] monitor-agent: containers[0].capabilities.add — dangerous capabilities: SYS_ADMIN [P3] web-frontend: containers[0].runAsNonRoot — runs as root without runAsNonRoot guarantee [P3] web-frontend: automountServiceAccountToken — SA token mounted (set false if the workload never calls the API) register: findings.csv

CI gate behavior: fix or fail the build

$ python3 podspec.py pods.json >/dev/null; echo “gate exit=$?” gate exit=1

Detection considerations

podspec runs on manifests and live exports; the delta between runs is drift telemetry (a new privileged workload is a finding whether or not admission caught it — if it landed, something bypassed policy, which is itself the alert). Pair with runtime sensors for post-deployment behavior: the spec says what a pod may do, syscall telemetry says what it is doing — a clean spec with token-read + metadata-probe behavior is the exact pattern from this module’s attack chain. In the SIEM, join admission blocks, podspec findings, and k8saudit register hits into one namespace-level risk view (BT3’s pipeline); a namespace accumulating all three is where the next incident is being staged.

Improvements to add

Accept the full resource list from kubectl get all -A -o json and audit every kind with a pod template in one pass.
Compare against the admission policy’s own rules and report gaps — specs that would pass today but violate the intended standard.
Add init-container and ephemeral-container coverage explicitly (escape ladders love init containers).
Emit per-namespace risk scores so cleanup sprints have a ranked target list.
Join with image provenance (H10 SBOM data) to correlate risky specs with unscanned images.

Detection — Defender View

Signal	Source	ID	Notes
exec/attach into pods	K8s audit log	pods:exec	Shell-in-pod, decisive
New role bindings	K8s audit log	rolebindings:create	RBAC escalation
Secret mass read	K8s audit log	secrets:list	Identity theft prep
Privileged pod attempt	Admission log	PSS/policy block	Probe or friction
Token path read	Runtime sensor	syscall rule	SA token theft
Metadata endpoint probe	Runtime sensor	169.254.169.254	Cloud credential pivot
Escape-class syscalls	Runtime sensor	host-write from container	T1611 family
Webhook config change	K8s audit log	mutatingwebhookconfigurations	Persistence primitive
New CSR approval	K8s audit log	certificatesigningrequests	Node identity forgery
Spec drift (new P1)	podspec delta	CI gate	Bypassed admission
Image digest mismatch	Registry webhook	allowlist violation	Supply chain guard
Miner process profile	Runtime sensor	process/cpu shape	Post-exploit classic

Evasion Techniques

Adversaries adapt to each layer. Admission bypass through legitimacy: workloads arrive via operators/CRDs whose generated pods inherit permissive defaults, or through namespaces admins exempted from policy years ago — the exemption register strikes again. Audit-log quietness: attackers whose API activity matches the workload’s normal pattern (the CI bot that legitimately reads secrets in its own namespace) stay under rare-event baselines; cross-namespace movement is the tell to watch, and the reason k8saudit scores it. Runtime sensor evasion: static binaries, direct syscalls, LD_PRELOAD-free payloads, and time-boxed execution (mine at 40% CPU under the profile threshold) — Falco-class rules need behavior baselines, not just signature lists. Token hygiene inversion: bound tokens with short TTLs force attackers to live-call the API (noisy) — so they pivot to stealing long-lived secrets from etcd-adjacent paths or image-baked credentials, which is why secret hygiene (H10) is detection-relevant, not just preventive. Ephemeral forensics denial: attackers count on pods dying — evidence-first response (capture before evict) is the counter, and node-level audit retention is what makes the counter possible. CRD-layer hiding: persistence lives in operators and webhooks rather than deployments, surviving naive “delete the malicious pod” cleanup — the audit log’s webhook-configuration events are the durable tell.

⚠️ What caught me: we enforced Pod Security Standards cluster-wide, had audit logging on, and still found a cryptominer that had lived for three weeks — because it ran in a namespace exempted from PSS since a 2023 migration nobody remembered, and its spec was clean: no privileged flags, no host mounts, just an over-permissive service account I’d granted secrets:list cluster-wide during a “temporary” CI migration. The miner never triggered a signature rule; it was the audit register’s cross-namespace secret read that finally surfaced it. The lesson burned in: exemptions are the attack surface, service accounts are identities (BT15’s doctrine extends to pods), and “temporary” RBAC is permanent until someone with a register checks.

Lab Practice

Build a single-node cluster (minikube/kind [verify current setup steps per tool]) and run the full loop. Prevention: enable Pod Security Standards on a namespace; try deploying a privileged pod and watch admission reject it; install a policy engine and write one custom rule (no hostPath). Detection: enable audit logging with a policy that logs RequestObject for pods/exec and RBAC changes; generate activity — a legit kubectl exec debug session, then a scripted “attacker” pod reading secrets cross-namespace — and run k8saudit over the audit stream; confirm the attacker pod’s secrets:list scores top of register. Runtime: install a Falco-style sensor and fire the classic signatures (shell in container, metadata endpoint curl) — note which rules fire that the audit log alone missed. Audit: export pod specs and run podspec; fix every P1 and confirm the exit-code gate flips to 0. Response drill: quarantine a namespace with a NetworkPolicy, revoke a service account’s tokens, and time the whole containment — that time is your MTTR baseline. Debrief: which of the three pillars (admission, audit, runtime) caught what the other two missed?

Interview Questions

Technical: Walk the three detection surfaces in Kubernetes — admission logs, API audit log, runtime sensors — and name one attack class only each can see.
Technical: Why is a service account a first-class identity in cluster forensics, and how does k8saudit’s cross-namespace scoring encode that?
Scenario: The k8saudit register shows a CI service account reading secrets in kube-system — never in its baseline. Walk your first hour: containment, evidence, and the one question you ask the platform team.
Scenario: A developer needs kubectl exec for debugging and your admission policy is fine with it — but the security register flags every exec. Design the workflow that serves both.
Conceptual: Why do mutating webhooks make strong persistence primitives, and why does deleting the malicious pod not remove them?
Deep-dive: Design container security ops for a 12-cluster estate: admission policy as code, audit pipeline, runtime sensor rollout, secret hygiene, and the response playbooks you’d automate first.
Practical: You inherit a cluster with audit logging off, default service accounts everywhere, and no runtime sensors. What do you enable in week one, and in what order?

BT3 (SIEM Engineering & Log Pipelines) · BT4 (Incident Response Process) · BT9 (Detection Engineering & Sigma) · BT16 (Cloud Detection & Response) · H10 (Supply Chain & Update Security)