The audytx benchmark
Three measured results for audytx's own engine, each with its engine version and date: audited precision on real-world Terraform, recall against hand-authored ground truth, and how correctly audytx sets aside, with a written rationale, the findings that context shows are not a problem.
Headline numbers
Three results, each with its confidence interval and the sample behind it.
57,004 kept findings across 398 scanned repositories, engine 1.12.2.
Recall — graded against hand-authored ground truth
Five deliberately-vulnerable third-party corpora, each pinned to one commit, plus an
in-repo seeded set. A hand-authored list maps every code-unambiguous vulnerability to
the check that should catch it. One corpus (cfngoat) is CloudFormation.
The table is the committed record of the run of 2026-08-07 (engine 1.18.276).
| Corpus | Pinned SHA | Ground-truth items | Recall | Benign-twin FPs | Suppression-correct | Gaps |
|---|---|---|---|---|---|---|
| cfngoat (CloudFormation) | 0c09b69cfc3dbc6cb3ef01883415c35c588ced48 | 20 | 90.0% | 0 | n/a | 0 |
| iam-vulnerable | 0f298666f9b7cfa01488b86912afdb211773188a | 31 | 100.0% | 0 | n/a | 0 |
| kaimonkey | 3feb0bf1a5ab81a79f225407a5ed16db3acd59a4 | 13 | 92.3% | 0 | n/a | 0 |
| sadcloud | f538652921a7f3e8e25c0afaecad2d4382f8b21e | 16 | 100.0% | 0 | n/a | 0 |
| seeded | in-repo | 30 | 100.0% | 0 | 100.0% | 0 |
| terragoat | 729f8da62c6a85ce4af5ad3d123de97776d954c4 | 15 | 100.0% | 0 | n/a | 0 |
| Overall | — | 125 | 97.2% | 0 | 100.0% | 0 |
125 ground-truth items overall. How to read the columns:
- Recall is the share of hand-authored vulnerabilities audytx reports as a kept finding. A finding audytx emits but sets aside does not count.
- Benign-twin false positives counts how often the paired check fires on a near-identical copy of a vulnerable resource that has only the issue fixed.
- Suppression-correct is the share of expected set-aside findings that audytx fires and then sets aside with the right reason.
There are no registered gaps. Three items were missed in this run:
- KM-005 — an EC2 root volume left unencrypted by omitting
root_block_deviceentirely. Firing on the omitted block alone would also flag every instance that relies on an account's EBS encryption-by-default setting, which AWS applies per Region, or on an encrypted AMI. audytx does not fire on the omitted block alone. - CG-004 (S3 access logging) and CG-005 (S3 versioning) — checks for both have shipped since this run. The table has not been re-run since, so it still counts them as misses.
Precision — audited on 398 real-world repositories
A sweep of a pinned set of real public AWS Terraform repositories, then a hand audit of a fixed-seed stratified sample of its findings against primary sources. Engine 1.12.2, July 2026.
- Kept sample: 100 findings — 93 true positives, 3 false positives, 4 unclear → precision 96.9% (Wilson 95% 91.2%–98.9%), conservative 93.0%.
- Set-aside sample: 40 findings — 38 correct, 1 wrong, 1 unclear → suppression correctness 97.4% (Wilson 95% 86.8%–99.5%), conservative 95.0%.
An unclear verdict is left out of the primary number and counted against audytx in the conservative one, so the conservative figure is the floor if every ambiguous case is a miss.
Where the claims stop
- Point in time. Precision is engine 1.12.2 (July 2026) and recall is engine 1.18.276 (August 2026). A later engine or corpus gives different numbers.
- Precision is a sample estimate. 96.9% is the point estimate over a 100-item kept sample; the Wilson range is 91.2%–98.9% and the conservative floor is 93.0%.
- A single model graded the audit samples. It applied primary-source citations to each verdict; it was not a panel of independent reviewers.
- The corpora do not cover everything. The repositories skew toward starred, recently active ones, and the vulnerable corpora do not cover every AWS resource type, so recall is scoped to the authored ground truth.
- AWS only. The precision track is Terraform only. CloudFormation has one recall corpus here (cfngoat, 20 items); its own results are on the CloudFormation benchmark.
Benchmark FAQ
The three headline metrics, defined in one place.
What is audytx's measured precision?
96.9% audited precision on the kept-findings sample (Wilson 95% CI 91.2%–98.9%; conservative floor 93.0%), engine 1.12.2, July 2026.
What is recall?
Recall is the share of hand-authored vulnerabilities audytx reports as a kept finding. A finding audytx emits but sets aside does not count. Measured recall is 97.2% on 125 ground-truth items across six corpora (engine 1.18.276, August 2026), with no registered gaps.
What is suppression correctness?
Suppression correctness is the share of findings audytx sets aside whose written reason is true and does remove the risk. Measured suppression correctness is 97.4% (engine 1.12.2, July 2026).
Start free
Install the GitHub App, open a pull request, and read the same engine these numbers measure — one comment, precision you can trace. The Free plan includes 100 scans a month.