The audytx Terraform scanner benchmark

Headline numbers

Three measured results, each carrying its confidence interval and the audit that produced it.

57,004 kept findings across 398 scanned repositories, engine 1.12.2 — from two complementary tracks, described next.

One benchmark story, two tracks

Precision and recall answer different questions, so they come from two complementary harnesses and are reported separately.

Recall — graded against hand-authored ground truth

Four deliberately-vulnerable corpora, each pinned to one immutable commit, with a hand-authored manifest mapping every code-unambiguous vulnerability to the check id(s) that cover it. The committed table below is the point-in-time record (engine 1.12.2, run 2026-07-13).

74 ground-truth items overall. Three metrics, each defined to resist gaming:

• Recall is computed over non-gap items only — the fraction of hand-authored vulnerabilities audytx reports as a kept finding. A finding audytx emits but context-suppresses does not count toward recall.
• Benign-twin false positives is a check-scoped count: each benign twin is a near-identical copy of a vulnerable resource with only the issue fixed, and the metric counts how often the paired check wrongly fires on the fixed resource. An unrelated hygiene check firing on the twin isn't counted; that's why every item declares which check ids own it.
• Suppression correctness is the moat metric: audytx must fire the naive finding and attach the reasoning that reclassifies it as benign. A scanner that stays silent scores the same as one that never had the check; audytx has to fire-then-justify. We haven't found another public IaC scanner recall benchmark that measures this third axis.

The 2 gaps are the published engine limitations SC-015 (public SQS resource policy) and SC-016 (public ECR resource policy) — audytx has no check for those classes today, so they are registered as gaps, disclosed here, and excluded from the recall denominator rather than hidden as silent misses. SC-015 and SC-016 are the only two, out of 74 ground-truth items.

Precision — audited on 398 real-world repositories

A precision number is only worth the audit trail behind it. This track sweeps a pinned corpus of real public AWS Terraform repositories, then hand-audits a fixed-seed stratified sample against primary sources.

Manifest sha256 d36b96086e41ea68d720d4fbe6a6b374406975681209dc5ec41e04cc17e07ab9, retrieved 2026-07-12. The two fatal parses were cloudposse/terraform-aws-alb and aws-ia/terraform-aws-iam-identity-center; the 3,804 suppressions are findings a naive single-resource scanner would have reported as-is.

The audit is a fixed-seed stratified sample: seed 20260712, 100 kept findings and 40 suppressed findings drawn by a sha256-keyed selection over each finding family — no RNG, so the exact 140-item set is reproducible from the seed alone. Grades:

• Kept sample: 100 findings — tp=93, fp=3, unclear=4 → primary precision 96.9% (Wilson 95% 91.2%–98.9%), conservative 93.0%.
• Suppressed sample: 40 findings — correct=38, wrong=1, unclear=1 → suppression correctness 97.4% (Wilson 95% 86.8%–99.5%), conservative 95.0%.

The verdict taxonomy is tp / fp / unclear. An unclear is never forced to a positive: the primary number counts it out of the denominator, and the conservative number counts every unclear against audytx (93.0% and 95.0% are the floor if every ambiguous case is a miss). Every audited fp and every wrong suppression is written up with a root cause in the cross-version ledger precision/results/register-history.md in the public harness, so a claimed precision never rests on a defect we noticed but did not record.

Methodology — why the numbers are reproducible

The four reproducibility pillars both harnesses share.

Wilson interval: Wilson, E. B. (1927), Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association 22:209–212; z=1.96 for the 95% interval.

Corpus credibility

A precision number is only as trustworthy as the corpus behind it. The public audytx-testbed publishes a diversity report — an independent instrument — that quantifies how wide and how hard the corpus reaches.

Control coverage is measured with KICS purely as a neutral instrument — to size the corpus, not as a scored competitor. The AWS universe is pinned to hashicorp/terraform-provider-aws v6.52.0 (1,669 resource types), and the provenance spread is 179 distinct upstream vendors, role-labeled 253 precision, 76 recall, and 2 custom.

The role labels are measured, not asserted: recall-role corpora average 19.66 HIGH+CRITICAL KICS findings per corpus, versus 3.02 for the precision-role corpora — the vulnerable fixtures really do light scanners up, and the clean modules really are the residual false-positive surface.

Full report: audytx-testbed Actions run 29226801605 (2026-07-13).

How the competitor tables are produced

The head-to-head scorecards live on /comparison. Here's how the audytx column in them is kept honest.

The audytx column is pulled from live GitHub Code Scanning SARIF — per bench/* branch of the public audytx-testbed, by scripts/benchmark-scorecard.sh — and a row is accepted only when the SARIF driver.version matches the live engine, so a stale scan can't slip in. Every competing scanner runs at a pinned version with no suppression file, on the same corpus, scored by the same deterministic script: a measurement, not an argument.

Where the claims stop

Reproduce it yourself

You don't have to take these numbers on faith. The precision and CloudFormation harnesses are public with their per-finding adjudication records; the recall suite runs as tests in the engine repository.

# Public harness — manifests, scoring, per-finding adjudication records:
git clone https://github.com/victorsinha/audytx-benchmark && cd audytx-benchmark

# Terraform precision — seed-pinned 80-repo sample (free Client ID from the dashboard):
export AUDYTX_CLIENT_ID=<your-client-id>
make precision
# Precision aggregator self-check (anti-gaming + determinism gates, offline):
python3 precision/aggregate.py --self-test

# CloudFormation 4-tool scorecard (needs checkov, trivy, kics):
make cfn

# Corpus diversity report — download the exact artifact this page cites:
gh run download 29226801605 --repo victorsinha/audytx-testbed --name diversity-report-2026-07-13

Ground-truth write-ups: precision/results/scorecard.md and the per-defect resolution history precision/results/register-history.md, both in the public harness; the recall suite and its write-up (benchmark/recall/README.md) run as tests in the engine repo. Live engine check and axis counts are on /status.

Benchmark FAQ

The three headline metrics, defined in one place.