What Frontier Models Get Wrong About AI-Generated Terraform Security
We asked Claude's flagship model to write Terraform for 50 realistic AWS scenarios — no security guidance, one shot each — then scanned every output with audytx and terraform validate. This is what AI-generated Terraform ships by default, before anyone asks for security.
terraform validateAI-generated Terraform looks like production code, not a vulnerability lab. At 48.3 findings per 100 resources it's on par with hand-written production modules (52.2) and ~5× cleaner than a deliberately vulnerable benchmark (269.1) — the model avoids the gaping holes. The consistent gap is operational and cross-resource: it creates secrets but forgets rotation, reaches for security knobs that don't exist and hallucinates the argument name, and 1 in 7 generations doesn't pass terraform validate.
Method
Fifty developer-voiced AWS scenarios, one shot each, zero security guidance — we measure what the model ships by default, not what it produces when asked to harden.
50 prompts · 8 archetypes
Web app, serverless API, data pipeline, EKS, static site, async worker, multi-env, and vague "vibe" one-liners — every prompt security-free.
One-shot generation
Each prompt sent verbatim to claude-opus-4-8: "Write the Terraform. Output only HCL." No retries — a broken output is data. Generators isolated from the audytx repo.
Isolated scan
audytx v0.5.1 through the MCP scan_terraform endpoint, one root module at a time — 50 separate scans, so cross-resource reasoning stays inside each app.
Reference corpora
terragoat (deliberately vulnerable) and 21 clean production modules from the audytx benchmark, scanned with the identical per-directory method.
Key findings
Density comparison
On every axis measured, the AI corpus tracks hand-written production modules — and stays nowhere near terragoat's deliberately-vulnerable numbers.
| Corpus | Resources | Findings / 100 | Security / 100 | HIGH+CRIT / 100 |
|---|---|---|---|---|
| AI corpus (Opus 4.8) | 1,421 | 48.3 | 12.1 | 1.3 |
| Clean production modules (21) | 882 | 52.2 | 9.9 | 3.1 |
| terragoat (deliberately vulnerable) | 55 | 269.1 | 130.9 | 94.5 |
Where the debt lands
Most of it is operational rather than security debt — missing CloudWatch alarms, unset prevent_destroy on stateful resources, and loose provider pins dominate the count.
| Category | Findings | % of total |
|---|---|---|
| Reliability | 179 | 26% |
| Observability | 168 | 24% |
| Cost | 167 | 24% |
| Security | 163 | 24% |
| Data Protection | 9 | 1% |
The 18 high-severity findings
One cross-resource failure pattern accounts for 61% of every HIGH finding: Secrets Manager rotation left unconfigured.
| Finding | Rule | Count |
|---|---|---|
| Secrets Manager rotation not configured | AWS_SM_001 | 11 |
| S3 bucket policy doesn't deny non-TLS | AWS_S3_005 | 3 |
| ALB listener HTTP without HTTPS redirect | AWS_ELB_005 | 1 |
| Redshift parameter group doesn't enforce SSL | AWS_REDSHIFT_002 | 1 |
| S3 versioning disabled | AWS_S3_003 | 1 |
| Lambda 3s timeout in VPC | AWS_XREF_001 | 1 |
Hallucination rate
7 of 50 (14%) fail terraform validate (Terraform 1.9.8). The signature failure: the model reaches for a security setting and invents the argument name.
pipeline-03/redshift.tf:71—Unsupported argument: "require_tls"(no such attribute on the resource)pipeline-04/redshift.tf:92—Unsupported argument: "require_ssl"webapp-01/database.tf:62—Unsupported argument: "storage_encrypted_kms_key_id"(invented fusion of two real RDS arguments)
Three of the seven hallucinations are invented security knobs: the model knows it should enforce TLS or scope the KMS key, but invents the syntax. The intent is there; the provider knowledge is wrong.
The cross-resource failure pattern
The most consistent security gap — 11 of 18 HIGH findings — is a single missing dependency: the model creates an aws_secretsmanager_secret for the database password but never the companion aws_secretsmanager_secret_rotation. A related pattern appears in arXiv:2512.14792 (Nekrasov et al., December 2025), whose error analysis says omitted cross-resource references “indicate failures in modeling resource interdependencies”.
A check that looks at one resource flags the secret for a missing attribute. audytx looks across the plan, finds that no rotation resource exists anywhere, and reports it with a message that names the missing resource, not just the missing attribute.
What the model got right
The naive prompts did not lead the model into obvious security traps — it reached for the right primitives on its own.
No wildcard-admin IAM
Zero ATTACK_PATH_* privilege-escalation chains fired across all 50 apps. Service roles are scoped to the action they need, never *.
S3 locked down
26 of 50 directories set aws_s3_bucket_public_access_block; private ACLs cover most of the rest. No public bucket reached the confirmed findings.
DLQ wiring present
Dead-letter queues appear in nearly every async-worker config. The model understands the pattern — it just forgets to alarm on the DLQs.
Encryption at rest
44 of 50 directories reference encryption; 39 of 50 create a KMS key. The at-rest story is broadly present, not bolted on.
audytx's cross-resource reasoning suppressed 85 findings a single-resource matcher would have fired — 71 of them IAM-role findings on service roles that are scoped and non-escalatable. Built and tuned on human-written Terraform, the suppression generalizes to AI-generated Terraform too.
Methodology caveats
- One model, one date. This is
claude-opus-4-8on 2026-06-14 only. The harness is built for multi-model follow-up; that's out of scope for v1. The flagship is the strongest case — if it ships these gaps, smaller or older models likely ship more. - The prompt set is ours. 50 prompts we authored, not a random sample of real developer requests.
- audytx authored the scanner and this study. Conflict of interest, disclosed up front. The corpus and raw scan results are not published, so the numbers on this page cannot be re-checked from outside.
terraform validateresults are tool-independent. - Findings ≠ vulnerabilities. Most are reliability/cost/observability gaps, not exploitable holes. The category and severity splits are there precisely so the raw total (686) isn't misleading.
- Post-publication note on AWS_VPC_004. The v0.5.1 scan reported 47 "Needs Review" findings for security groups with unknown port exposure. After freezing this study, analysis found these were scanner false positives: the AI-generated code uses the newer
aws_vpc_security_group_ingress_ruleresource pattern, which the v0.5.1 engine couldn't resolve. The AI-generated code wasn't actually exposing debug ports. Without those 47, the totals are 639 findings and 45.0 per 100 resources; the category and severity splits were not recomputed.
Data & citations
Datasets used
- AI corpus — 50 Terraform configurations (1,421 resources) generated one-shot by
claude-opus-4-8on 2026-06-14. The raw outputs and scan results are held privately and are not published. - terragoat — the deliberately-vulnerable Terraform training corpus (terragoat, open source), scanned with the identical per-directory method as the vulnerable reference point.
- Clean production modules — the 21-module hand-written corpus from the audytx benchmark methodology.
Related work
- arXiv:2512.14792 — Nekrasov et al., “IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection” (December 2025). Related in theme; it is not a replication of this study.
Cite this study
Reuse the numbers or tables with attribution:
Frequently asked questions
Is AI-generated Terraform more or less secure than hand-written?
On par with hand-written production Terraform on the metrics we measured, on engine v0.5.1 in June 2026. The AI corpus (50 configurations, 1,421 resources, generated with Claude Opus 4.8) produced 48.3 findings per 100 resources, compared to 52.2 findings per 100 resources across clean, hand-written production Terraform modules — and far below terragoat's deliberately vulnerable 269.1 findings per 100 resources.
What's the most common security gap in AI-generated Terraform?
Secrets Manager rotation. 11 of the 18 HIGH-severity findings — 61% of all HIGH findings — trace to one cross-resource failure pattern: the model creates an aws_secretsmanager_secret for a database password but never creates the companion aws_secretsmanager_secret_rotation resource.
Fix it where the agent writes it
The gaps in this study — the unwired rotation, the escalation-capable role, the invented argument — are cheapest to close in the same session that wrote the code. Wire audytx into your coding agent over MCP and it scans its own Terraform before the pull request exists. Setup is one config entry and one standing instruction per agent:
Claude Code →
One claude mcp add, then a line in CLAUDE.md so it scans before every PR.
Cursor →
A .cursor/mcp.json server entry plus an always-on project rule.
GitHub Copilot →
A .vscode/mcp.json server plus .github/copilot-instructions.md.
Start free
Audit your AI-generated Terraform where it's written — audytx posts findings straight into your pull request with cross-resource context, one-click fixes, and false-positive suppression that states its rationale.
Also available as an MCP server for Claude Code, Cursor, and any MCP-compatible agent.