audytx vs Checkov, Trivy, KICS & Terrascan

Table 1 — IAM privilege-escalation: precision & recall

Corpus: BishopFox iam-vulnerable — 31 documented AWS IAM privilege-escalation paths, one Terraform file per path. A tool "detects" a path if it fires at least one HIGH/CRITICAL finding on the file implementing it. audytx and Checkov are scored on HIGH; KICS and Trivy on HIGH+CRITICAL.

¹ Checkov run offline (no Bridgecrew API) emits no per-check severity, so this is total failed checks, not HIGH-only — generous for Checkov's recall, conservative for its precision.
² Terrascan exceeded the 5-minute timeout on this corpus (large module graph).
audytx and Checkov are the only two tools that detect all 31 paths. audytx does it with ~2× the precision and half the alert volume. Most of audytx's 104 "FP" here are legitimate secondary detections (e.g. AWS_OPS_038 firing on the same privesc file the primary rule already claimed) plus the corpus's own intentional FP-test fixtures — TP matching counts only one finding per path, which structurally undercounts audytx precision.

Table 2 — False positives on 21 clean production modules

This is the wedge. Each corpus is a well-regarded, actively-maintained AWS community Terraform module with an expected high-severity count of 0. Lower is better — every HIGH finding here is noise a reviewer has to triage. Raw counts shown as measured (nothing subtracted).

³ Terrascan fires 348 HIGH alerts on a single module (terraform-aws-s3-bucket) — a mass-rule blowup that alone accounts for 77% of its total.
audytx's 33 = ~36× fewer than Checkov (1,193), 14× fewer than Terrascan (454), and 5× fewer than Trivy (175) — and the fewest raw false positives of all five, just ahead of KICS (34), which reaches that only by detecting 3% of the privesc paths (Table 1) to audytx's 100%. Several of audytx's 33 trace to documented justified exceptions (real issues in the modules' own example code, tracked in clean-modules.yaml); the rest are new rules (EKS secrets-encryption, deprecated Lambda runtimes) correctly flagging issues in the modules' examples/ — see the honest column below.

Table 3 — Recall corpora (raw HIGH counts)

Six additional corpora with deliberately insecure configurations. We have no path-level ground truth here beyond iam-vulnerable, so these are raw HIGH counts, not scored precision/recall — more is not automatically better, since a chunk of any tool's count is noise. Shown for completeness.

⁴ Trivy timed out on eks-blueprints (5-min limit); Terrascan DNF on iam-vulnerable (Table 1).

How the precision gap happens — a worked example

The Table 2 result is not fewer rules — it's cross-resource reasoning. audytx pre-computes relationship graphs and suppresses findings that context proves benign, showing the rationale instead of dropping them silently. Here's the mechanism on one fixture (testbed #11), illustrative of why the clean-module counts diverge so far.

Multiply this across DLQ identity, Lambda invocation graphs, encryption variants, data lifetime, network exposure, IAM trust/policy reachability, tag environment and IMDSv2 inheritance — 17 reasoning axes in the live engine — and you get the 48-vs-1,193 gap in Table 2.

Methodology

Everything needed to reproduce the run, exactly as it was performed.

Corpus: 28 AWS-Terraform repos/modules in the public audytx-testbed, each on a bench/<name> branch. 5-minute timeout per tool per corpus (timeouts = DNF). No suppression files for any tool. Scoring is scripts/score.py (Python-3 stdlib only) — given the same inputs it produces a byte-identical scorecard every run. TP matching: a finding counts once per ground-truth path if its file and resource/rule reference that path; extra findings on the same path count as FP (conservative for audytx).

Where audytx is weaker — the honest column

Reproduce it yourself

The corpus, the ground truth, and the scorer are public. You do not have to take our numbers on faith.

# 1. Clone the public benchmark corpus
git clone https://github.com/victorsinha/audytx-testbed
cd audytx-testbed

# 2. Run any competitor on a corpus (example: Checkov on a clean module)
checkov -d corpus/terraform-aws-rds --framework terraform -o json | jq '.summary'

# 3. audytx numbers come from the live Code Scanning SARIF on each bench branch
gh api "repos/victorsinha/audytx-testbed/code-scanning/analyses?ref=refs/heads/bench/terraform-aws-rds" \
  --jq '[.[] | select(.tool.name=="audytx")] | sort_by(.created_at) | last | .id'

# 4. Re-score everything deterministically
python3 scripts/score.py results ground-truth

Full write-up — every table, footnote and caveat — lives in docs/benchmark-v1.md in the engine repo.