Historical benchmark — June 3, 2026, engine v0.2.4. audytx now detects all 31 documented IAM privilege-escalation paths at full recall; the IAM limitation described below was closed after this run. See the current results on /comparison and /iam-security.

Public benchmark · 2026-06-03

audytx vs Checkov: 7 real Terraform repos, full receipts

Seven real public AWS Terraform repos, no cherry-picked fixtures — four secure-by-default modules to measure noise, three intentionally-vulnerable repos to measure detection. Every finding below traces to the Code Scanning SARIF, not the truncated comment — knives out, including where audytx loses.

4×

less noise than Checkov across the four secure-by-default modules

real public AWS Terraform repos — 4 precision, 3 recall, zero fixtures

0–5

high-severity audytx findings per secure module vs Checkov's dozens

~108

of Checkov's 129 s3-bucket findings are DR, cost, style, or stale — not security

How it ran

Methodology

Same files, same live pull request, both tools automated — no suppression files, audytx v0.2.4.

Real repos, vendored

AWS .tf only, vendored onto long-lived PRs in the public audytx-testbed repo.

audytx GitHub App

Scans the PR and uploads SARIF to Code Scanning. Counts come from the structured output, not the truncated summary comment.

Stock Checkov

Default ruleset, pip-latest (2026-06), same files. Full failed-check count from JSON output.

Trivy config

v0.71, trivy config, default ruleset, run locally against each corpus directory.

The receipts

Scoreboard

Precision repos: lower is better (findings are candidate false positives on secure code). Recall repos: detection across the intentional-vuln surface.

Repo	Role	audytx (high)	Checkov	Trivy
Precision — secure-by-default modules (lower = less noise)
terraform-aws-vpc	noise	6 (0)	25	25
terraform-aws-s3-bucket	noise	20 (3)	129	53
terraform-aws-eks	noise	31 (5)	88	73
terraform-aws-secure-baseline	noise	29 (4)	107	0
Recall — intentionally-vulnerable repos (higher = better detection)
terragoat	detect	86 (16)	213	—
iam-vulnerable	detect	201 (83)	269	62
KaiMonkey	detect	43 (11)	100	55

Precision, by name

What Checkov's 129 actually are

On the secure, popular terraform-aws-s3-bucket module, Checkov's 129 failed checks break down by rule like this:

×17 CKV_AWS_144 — cross-region replication (DR/cost preference, not security)

×17 CKV_AWS_300 — abort-incomplete-multipart lifecycle (hygiene)

×17 CKV2_AWS_62 — event notifications enabled (functional choice)

×17 CKV_AWS_21 — versioning on every bucket (opinionated)

×17 CKV_AWS_145 — KMS-specific encryption (SSE-S3 is fine)

×16 CKV_AWS_19 — encryption at rest (stale — S3 encrypted by default since 2023)

×7 CKV_TF_1 — pin module to commit hash (Terraform style, not infra at all)

~20 genuinely security-relevant — block-public-access, logging, KMS rotation/policy

So ~108 of Checkov's 129 are DR/cost preferences, hygiene, stale, or pure style. audytx surfaced 20 findings (3 high) on the same module — actual S3/KMS/SQS security, never the cross-region-replication / multipart / event-notification / commit-hash noise.

4× less noise on secure modules

Across the 4 secure modules: audytx 6 / 20 / 31 / 29 vs Checkov 25 / 129 / 88 / 107 and Trivy 25 / 53 / 73 / 0 — audytx 0–5 high vs dozens.

Recall

Loud where it should be, quiet where it shouldn't

audytx's 86 terragoat findings span 10 AWS services across 10 of 14 files — loud on terragoat (86), quiet on vpc (6). That's the context graph, not luck.

S3 RDS Lambda EC2 EKS ECR KMS IAM VPC Elasticsearch

No spin

Where audytx loses

This is a benchmark, not a highlight reel. The SARIF surfaced three concrete gaps.

No ELB or Neptune checks

terragoat's ELB (no access logging, no TLS listener) and Neptune cluster (unencrypted) produced zero audytx findings — real misconfigs it didn't check yet.

IAM depth was shallow

On iam-vulnerable, only one distinct IAM check fired (AWS_IAM_006, hardcoded keys, ×41). Privilege-escalation detection came later — Checkov was broader here that day.

The OPS family over-fired

151 of the 201 iam-vulnerable findings were operational (AWS_OPS_*) checks — useful signal, but enough volume that audytx isn't immune to its own noise.

Checkov still has more raw coverage. It's breadth-first: more rules across more services. audytx trades breadth for context — fewer findings, each one reasoned against the graph.

Verify it yourself

Reproduce it

The corpus is public. Pull audytx's full findings from Code Scanning SARIF, run Checkov on the same files. The Terraform precision harness with its per-finding adjudication records is also public: audytx-benchmark.

reproduce · code scanning sarif

# the corpus lives as open PRs on the public testbed
gh pr list --repo victorsinha/audytx-testbed --search "bench/ in:head"

# audytx's full findings (not the truncated comment) come from Code Scanning SARIF:
gh api 'repos/victorsinha/audytx-testbed/code-scanning/analyses?ref=refs/heads/bench/terragoat' \
  --jq '.[] | select(.tool.name=="audytx") | .id'
# then: gh api repos/.../code-scanning/analyses/<id> -H "Accept: application/sarif+json"

# Checkov on the same files:
checkov -d corpus -o json --quiet --soft-fail | jq '[.. | objects | select(.check_type=="terraform") | .summary.failed] | add'

Open beta

Start free during the open beta

Install the GitHub App and run the same engine on your own pull requests — every repo, public or private, no feature gates.

Install the GitHub App Read the docs