audytx on three real CloudFormation corpora
CloudFormation support runs the same engine that scans Terraform, against normalized
CloudFormation. This page reports audytx's own results on three public CloudFormation
corpora, run on 2026-07-25 at engine 1.17.68 (workspace commit 9bfd27b),
and the hand adjudication of its high and critical findings on one of them, false
positives included.
Method
The engine at one commit, run locally over the same staged input for each corpus.
Three corpora, pinned
AWS's own aws-cloudformation-templates (precision control), cfn_nag_examples (recall proxy), and a deterministic 25-pattern SAM slice of serverless-patterns — each fetched at one commit.
One engine state
Every number comes from the engine at commit 9bfd27b (version 1.17.68), run locally. The measured engine includes the CloudFormation-native rules and the dynamic-reference handling released with this page. Production has moved on since.
Scoreboard
audytx's own counts. Run 2026-07-25, engine 1.17.68. “Suppressed” counts findings audytx fired and then set aside with a written reason.
| Corpus | Role | Templates | audytx total | High + critical | Suppressed |
|---|---|---|---|---|---|
| aws-cloudformation-templates | precision | 316 | 2642 | 350 | 215 |
| cfn_nag_examples | recall proxy | 4 | 24 | 12 | 2 |
| serverless-patterns (25-pattern SAM slice) | SAM | 25 | 325 | 6 | 54 |
A suppressed finding, with rationale
Every finding audytx sets aside is listed with its reason. This one is from the aws-cloudformation-templates run above, not a constructed example.
resource: aws_lambda_function.Handler
reason: "aws_lambda_function.Handler is not invoked async-push (sync via API GW / Function URL, or polled-async via SQS / Kinesis event source mapping) — Lambda-level DLQ would never receive events. The invoker handles failures."
A dead-letter queue only matters for a function that is invoked asynchronously. audytx checks how the function is invoked across the template before deciding the finding applies.
Where audytx got it wrong
The adjudication found 57 catalog false positives among the 350 findings, in six patterns. One is shown here.
AWS_RDS_001 fired on an Aurora cluster-member instance
On DMS/DMSAuroraToS3FullLoadAndOngoingReplication.json, AWS_RDS_001 (“RDS storage should be encrypted at rest”) fired on aws_db_instance.AuroraDB, an Aurora cluster member with no instance-level StorageEncrypted. AWS's CloudFormation documentation for AWS::RDS::DBInstance says under Amazon Aurora: “Not applicable. The encryption for DB instances is managed by the DB cluster.” The cluster resource in the same template sets StorageEncrypted: true.
This was a real gap in the rule catalog when the benchmark ran. That false-positive class was found by this run and fixed in 1.18.
What these numbers do and don't claim
- AWS's own samples are not a hardened baseline.
aws-cloudformation-templatessays in its README: “Keep in mind that these templates are not meant to be production-ready "QuickStarts".” A higher finding count here reflects the corpus, not necessarily a precision regression. - Only one corpus is adjudicated finding by finding. The 350 high and critical findings on
aws-cloudformation-templateswere each reviewed; the other two corpora report counts only. - SAM is scanned as written. No special normalization was applied beyond the engine's own expansion of
AWS::Serverless::*resources. - Point in time. Every figure is engine 1.17.68 on 2026-07-25. A later engine gives different counts.
Start free
Install the GitHub App and run the same engine on your own Terraform and CloudFormation pull requests — no feature gates.