Public benchmark · 2026-06-03

audytx vs Checkov — 7 real open-source repos

Not hand-picked fixtures — seven real public AWS Terraform repos: four secure-by-default community modules (to measure noise) and three intentionally-vulnerable repos (to measure detection). Both tools run automatically on a PR; full findings are pulled from the Code Scanning SARIF, not the summary comment. Knives out — including where audytx loses.

Methodology

Each repo is vendored (AWS .tf only) onto a long-lived PR in the public audytx-testbed repo. audytx (the GitHub App) scans the PR and uploads SARIF to Code Scanning; a workflow runs stock Checkov (default ruleset, pip-latest, 2026-06) on the same files. Trivy (v0.71, trivy config, default ruleset) was run locally against each corpus dir. Numbers are full finding counts from each tool's structured output. No suppression files. audytx v0.2.4 (~210 rules).

Scoreboard

Precision repos: lower is better (findings are candidate false positives on secure code). Recall repos: detection across the intentional-vuln surface.

Repo	Role	audytx (high)	Checkov	Trivy
Precision — secure-by-default modules (lower = less noise)
terraform-aws-vpc	noise	6 (0)	25	25
terraform-aws-s3-bucket	noise	20 (3)	129	53
terraform-aws-eks	noise	31 (5)	88	73
terraform-aws-secure-baseline	noise	29 (4)	107	0
Recall — intentionally-vulnerable repos (higher = better detection)
terragoat	detect	86 (16)	213	—
iam-vulnerable	detect	201 (83)	269	62
KaiMonkey	detect	43 (11)	100	55

Precision: what Checkov's 129 actually are (s3-bucket)

On the secure, popular terraform-aws-s3-bucket module, we broke down Checkov's 129 failed checks by rule:

×17 CKV_AWS_144 — cross-region replication (DR/cost preference, not security)

×17 CKV_AWS_300 — abort-incomplete-multipart lifecycle (hygiene)

×17 CKV2_AWS_62 — event notifications enabled (functional choice)

×17 CKV_AWS_21 — versioning on every bucket (opinionated)

×17 CKV_AWS_145 — KMS-specific encryption (SSE-S3 is fine)

×16 CKV_AWS_19 — encryption at rest (stale — S3 encrypted by default since 2023)

×7 CKV_TF_1 — pin module to commit hash (Terraform style, not infra at all)

~20 genuinely security-relevant — block-public-access, logging, KMS rotation/policy

So ~108 of Checkov's 129 are DR/cost preferences, hygiene, stale, or pure style. audytx surfaced 20 findings (3 high), focused on actual S3/KMS/SQS security — it never emits the cross-region-replication / multipart / event-notification / commit-hash noise. This is the alert-fatigue gap, shown by name.

6× less noise on good code

Across the 4 secure modules: audytx 6 / 20 / 31 / 29 vs Checkov 25 / 129 / 88 / 107 and Trivy 25 / 53 / 73 / 0 — audytx 0–5 high vs dozens.

Recall: audytx detects broadly (terragoat)

From the SARIF, audytx's 86 terragoat findings span 11 AWS services across 10 of 14 files: S3, RDS, Lambda, EC2, EKS, ECR, KMS, IAM, VPC, Elasticsearch. It is loud where it should be (86 on terragoat) and quiet where it should be (6 on vpc) — the context-awareness working.

Where audytx loses

This is a benchmark, not a sales sheet. The SARIF surfaced three concrete gaps:

1. No ELB or Neptune rules. terragoat's ELB (no access logging, no TLS listener, internet-facing) and Neptune cluster (unencrypted, no audit logs) produced zero audytx findings. Both are real misconfigurations audytx simply does not check yet.

2. IAM depth is shallow — the unbuilt moat. On iam-vulnerable (a repo built around IAM privilege-escalation paths), audytx's 201 findings included only one distinct IAM rule (AWS_IAM_006, hardcoded keys, ×41). It does not yet detect privilege escalation, over-permissive policies, or resource-exposure — exactly the least-privilege analysis on our roadmap. Checkov is broader here today.

3. The OPS family over-fires. 151 of audytx's 201 iam-vulnerable findings were operational (AWS_OPS_*) rules. Useful signal, but enough volume that audytx is not immune to its own noise.

And Checkov has more raw coverage — breadth-first, it has more rules across more services. audytx trades breadth for context: fewer, better-reasoned findings on the AWS patterns it understands deeply.

Reproduce it

# the corpus lives as open PRs on the public testbed
gh pr list --repo victorsinha/audytx-testbed --search "bench/ in:head"

# audytx's full findings (not the truncated comment) come from Code Scanning SARIF:
gh api 'repos/victorsinha/audytx-testbed/code-scanning/analyses?ref=refs/heads/bench/terragoat' \
  --jq '.[] | select(.tool.name=="audytx") | .id'
# then: gh api repos/.../code-scanning/analyses/<id> -H "Accept: application/sarif+json"

# Checkov on the same files:
checkov -d corpus -o json --quiet --soft-fail | jq '[.. | objects | select(.check_type=="terraform") | .summary.failed] | add'

Try it on your repo

One click to install. Free, unlimited.

Install audytx →