audytx vs Checkov, Trivy, KICS: three real CloudFormation corpora, full receipts
CloudFormation support (250+ rules and 20+ context-reasoning axes run largely unchanged against normalized CFN, per the same engine that scans Terraform) gets its own measured numbers, separate from the published Terraform benchmark. Pinned corpus SHAs, retained raw tool outputs, a reproducible offline harness — knives out, including a false positive audytx's own catalog produced.
Methodology
Same staged input, same four tools, run locally — no live-deployment SARIF, no engine-state mixing.
Three corpora, pinned
AWS's own aws-cloudformation-templates (precision control), cfn_nag_examples (recall proxy), and a deterministic 25-pattern SAM slice of serverless-patterns — each fetched at one immutable commit SHA.
audytx, engine-honest
Numbers come from a local host binary running the identical parse_all → evaluate() chain the Worker runs, at the exact commit this page ships with — never a separately-sourced live scan result, which would mix engine states. That build includes the CloudFormation-native rules (AWS_CFN_001–004) and the dynamic-reference context axis introduced in the same release as this page, so the engine measured here is the same engine scanning pull requests in production.
Checkov, Trivy, KICS
Stock rule sets, native JSON output, same staged directory per corpus. cloudformation-guard is excluded: it's a policy-as-code DSL you write your own rules in, and AWS ships its rule sets as a separate, pickable Guard Rules Registry rather than a built-in catalog — so benchmarking it would measure our rule-set pick, not the tool.
Reproducible offline
A public harness (audytx-benchmark) with a preflight self-test layer that turns silent zeros — a mis-pathed queries dir, a wrong framework flag — into loud aborts before the real run starts.
Scoreboard
Precision corpus: lower is better. Recall/SAM corpora: detection across the surface. Run 2026-07-25 — checkov 3.3.0, trivy 0.72.0, kics 2.1.20, audytx workspace commit 9bfd27b.
| Corpus | Role | audytx total (high+crit / suppressed) | Trivy total (high) | KICS total (high+crit) | Checkov total† |
|---|---|---|---|---|---|
| aws-cloudformation-templates | precision | 2642 (350 / 215) | 1129 (643) | 2604 (306) | DNF‡ |
| cfn_nag_examples | recall proxy | 24 (12 / 2) | 13 (7) | 35 (2) | 13 |
| serverless-patterns (25-pattern SAM slice) | SAM | 325 (6 / 54) | 107 (27) | 292 (38) | 152 |
† Checkov's OSS free-tier check objects carry "severity": null in every entry we captured — total only, no high/critical split, matching the published Terraform benchmark's footnote. ‡ Checkov 3.3.0 crashes with a TypeError inside its own CloudFormation check registry partway through the 316-template precision corpus — a bug in checkov's engine, not something a CLI flag routes around (verified: --skip-check for a differently-logged exception didn't help either). It completed normally on the other two corpora. Full methodology, the per-finding adjudication records, and the reproduce steps live in the public audytx-benchmark repository — see "Reproduce it" below for the exact commands.
A suppressed finding, with rationale
Checkov, Trivy, and KICS have no equivalent layer — they can only report a raw raised count, never explain why a naive finding doesn't apply.
resource: aws_lambda_function.Handler
reason: "aws_lambda_function.Handler is not invoked async-push (sync via API GW / Function URL, or polled-async via SQS / Kinesis event source mapping) — Lambda-level DLQ would never receive events. The invoker handles failures."
A single-resource scanner sees a Lambda function with no dead-letter queue and flags it unconditionally. audytx's invocation context axis traces how the function is actually invoked across the template graph before deciding a DLQ is even meaningful for it — from the real aws-cloudformation-templates run above, not a constructed example.
Where audytx got it wrong
A benchmark that only shows wins isn't a benchmark. Adjudicating a sample of the precision corpus's 350 high/critical findings surfaced a real rule-catalog gap.
AWS_RDS_001 fired on Aurora cluster-member instances — found here, resolved since
On DMS/DMSAuroraToS3FullLoadAndOngoingReplication.json, audytx's AWS_RDS_001 ("RDS storage should be encrypted at rest") fired on aws_db_instance.AuroraDB, an Aurora cluster member with no instance-level StorageEncrypted set. AWS's own CloudFormation documentation for AWS::RDS::DBInstance's StorageEncrypted property states, under Amazon Aurora: "Not applicable. The encryption for DB instances is managed by the DB cluster." The sibling AuroraCluster resource in the same template correctly sets StorageEncrypted: true — the cluster IS encrypted; the per-instance rule just didn't know Aurora manages this at the cluster level, and fired anyway.
This was a genuine gap in the rule catalog at the time this benchmark ran (commit 9bfd27b, 2026-07-25) — recorded here rather than papered over, not a benchmark artifact. It's since been resolved (audytx 1.18.x): a context-suppression axis now recognizes an Aurora cluster-member aws_db_instance (CloudFormation's Engine + DBClusterIdentifier) and suppresses its instance-level AWS_RDS_001 finding with a rationale naming the cluster's own encryption finding — the same adjudication pass separately found the identical rule misreading AWS's documented encryption-inheritance for ordinary read replicas (SourceDBInstanceIdentifier set) and Aurora's cluster-managed deletion protection on AWS_RDS_005, both fixed the same way. The full per-finding adjudication that followed found four more catalog false-positive classes and one whole-repo scan-mode defect — each documented per finding in the harness's adjudication record and queued as engine follow-up work the same way.
Checkov's own coverage gap between formats
Re-counted from a fresh pinned clone, not quoted secondhand from an earlier draft.
As of checkov commit 6da0af4, 2026-07-25: Checkov ships 6 graph-based checks for CloudFormation versus 196 for Terraform (across all cloud providers its Terraform runner supports). Graph checks are the class of check that reasons across more than one resource — the same category audytx's context axes and cross-resource skills live in. This isn't a claim that audytx is more complete than Checkov in the abstract; it's evidence that CloudFormation's cross-resource tooling, industry-wide, is younger than Terraform's — the gap this whole track exists to close. See the CloudFormation graph_checks tree and its Terraform counterpart at that pin.
What these numbers do and don't claim
Architecture, not a deficit
For CloudFormation, Checkov, Trivy, and KICS evaluate findings almost entirely against one resource in isolation — Checkov's own cross-resource graph checks number just 6 for this format (counted below). audytx pre-computes cross-resource relationship graphs and suppresses a naive finding when context proves it benign, with a stated rationale. The two approaches measure different things, which is why the suppressed count is reported as its own column rather than folded into (or hidden from) the total.
Every finding adjudicated, in the open
All 350 high/critical findings on the precision corpus have been individually adjudicated — no sampling: 287 hold as true positives, 57 are catalog false positives, and 6 are defensible either way. The full per-finding record ships with the public harness (cfn/adjudication-aws-cloudformation-templates-2026-07.md in audytx-benchmark) and supersedes the earlier nine-finding representative sample.
AWS's own samples aren't a hardened baseline
Unlike the widely-used hardened Terraform modules in the published TF benchmark, aws-cloudformation-templates's own README calls itself "sample templates... not meant to be production-ready 'QuickStarts'" — several samples carry explicit CloudFormation Guard suppression metadata acknowledging the exact pattern audytx flags. A higher finding count here reflects the corpus, not necessarily a precision regression.
SAM handling is "as reported"
No tool-specific SAM normalization was applied for the scoreboard — every tool scanned the identical staged SAM templates with its own default flags. audytx expands AWS::Serverless::* before normalization; the comparators scan the raw SAM text with their stock CloudFormation rule sets.
Checkov didn't finish the precision corpus. A crash in checkov 3.3.0's own CloudFormation check registry — not a flag we could route around — means its precision-corpus number is DNF, not zero. See the scoreboard footnote and the full traceback retained with the harness.
Reproduce it
Every corpus is public, pinned to an immutable SHA. The harness is a public repository, not a one-off script.
# public harness: manifests, fetch, scoring, per-finding adjudication records git clone https://github.com/victorsinha/audytx-benchmark cd audytx-benchmark # full 4-tool scorecard (needs checkov/trivy/kics; free Client ID from /dashboard) export AUDYTX_CLIENT_ID=<your-client-id> make cfn # scorer self-check (offline, no network) python3 cfn/score.py --selftest
Start free during the open beta
Install the GitHub App and run the same engine on your own Terraform and CloudFormation pull requests — no feature gates.