audytx corpus study

What Frontier Models Get Wrong About AI-Generated Terraform Security

june 2026 · claude opus 4.8 · 50 configs

We asked Claude's flagship model to write Terraform for 50 realistic AWS scenarios — no security guidance, one shot each — then scanned every output with audytx and terraform validate. This is what AI-generated Terraform ships by default, before anyone asks for security.

48.3

findings / 100 resources — on par with production's 52.2

14%

of generations fail terraform validate

61%

of HIGH findings trace to one gap: rotation

false positives suppressed by cross-resource context

50 naive prompts 8 archetypes 1,421 resources 686 findings 1 model · Claude Opus 4.8 3 reference corpora

TL;DR

AI-generated Terraform looks like production code, not a vulnerability lab. At 48.3 findings per 100 resources it's on par with hand-written production modules (52.2) and ~5× cleaner than a deliberately vulnerable benchmark (269.1) — the model avoids the gaping holes. The consistent gap is operational and cross-resource: it creates secrets but forgets rotation, reaches for security knobs that don't exist and hallucinates the argument name, and 1 in 7 generations doesn't pass terraform validate.

Method

Fifty developer-voiced AWS scenarios, one shot each, zero security guidance — we measure what the model ships by default, not what it produces when asked to harden.

50 prompts · 8 archetypes

Web app, serverless API, data pipeline, EKS, static site, async worker, multi-env, and vague "vibe" one-liners — every prompt security-free.

One-shot generation

Each prompt sent verbatim to claude-opus-4-8: "Write the Terraform. Output only HCL." No retries — a broken output is data. Generators isolated from the audytx repo.

Isolated scan

audytx v0.5.1 through the MCP scan_terraform endpoint, one root module at a time — 50 separate scans, so cross-resource reasoning stays inside each app.

Reference corpora

terragoat (deliberately vulnerable) and 21 clean production modules from the audytx benchmark, scanned with the identical per-directory method.

Key findings

Density comparison

On every axis measured, the AI corpus tracks hand-written production modules — and stays nowhere near terragoat's deliberately-vulnerable numbers.

Corpus	Resources	Findings / 100	Security / 100	HIGH+CRIT / 100
AI corpus (Opus 4.8)	1,421	48.3	12.1	1.3
Clean production modules (21)	882	52.2	9.9	3.1
terragoat (deliberately vulnerable)	55	269.1	130.9	94.5

Where the debt lands

Most of it is operational rather than security debt — missing CloudWatch alarms, unset prevent_destroy on stateful resources, and loose provider pins dominate the count.

Category	Findings	% of total
Reliability	179	26%
Observability	168	24%
Cost	167	24%
Security	163	24%
Data Protection	9	1%

The 18 high-severity findings

One cross-resource failure pattern accounts for 61% of every HIGH finding: Secrets Manager rotation left unconfigured.

Finding	Rule	Count
Secrets Manager rotation not configured	AWS_SM_001	11
S3 bucket policy doesn't deny non-TLS	AWS_S3_005	3
ALB listener HTTP without HTTPS redirect	AWS_ELB_005	1
Redshift parameter group doesn't enforce SSL	AWS_REDSHIFT_002	1
S3 versioning disabled	AWS_S3_003	1
Lambda 3s timeout in VPC	AWS_XREF_001	1

Hallucination rate

7 of 50 (14%) fail terraform validate (Terraform 1.9.8). The signature failure: the model reaches for a security setting and invents the argument name.

pipeline-03/redshift.tf:71 — Unsupported argument: "require_tls" (no such attribute on the resource)
pipeline-04/redshift.tf:92 — Unsupported argument: "require_ssl"
webapp-01/database.tf:62 — Unsupported argument: "storage_encrypted_kms_key_id" (invented fusion of two real RDS arguments)

Three of the seven hallucinations are invented security knobs: the model knows it should enforce TLS or scope the KMS key, but invents the syntax. The intent is there; the provider knowledge is wrong.

The cross-resource failure pattern

The most consistent security gap — 11 of 18 HIGH findings — is a single missing dependency: the model creates an aws_secretsmanager_secret for the database password but never the companion aws_secretsmanager_secret_rotation. This matches the failure mode documented in arXiv:2512.14792: LLMs systematically fail to model cross-resource dependencies in infrastructure code.

# What the model generates:
resource "aws_secretsmanager_secret" "db_password" {
  name       = "${var.project}-db-password"
  kms_key_id = aws_kms_key.main.arn  # ✓ encrypted
}

# What it consistently omits:
resource "aws_secretsmanager_secret_rotation" "db_password" {
  secret_id           = aws_secretsmanager_secret.db_password.id
  rotation_lambda_arn = aws_lambda_function.rotate_secret.arn
  rotation_rules {
    automatically_after_days = 30
  }
}
    

A single-resource pattern matcher flags the secret for a missing attribute. audytx's cross-resource engine finds the absent companion resource, confirms rotation is configured nowhere in the plan, and fires — with a rationale that names the missing resource, not just the missing attribute.

What the model got right

The naive prompts did not lead the model into obvious security traps — it reached for the right primitives on its own.

No wildcard-admin IAM

Zero ATTACK_PATH_* privilege-escalation chains fired across all 50 apps. Service roles are scoped to the action they need, never *.

S3 locked down

26 of 50 directories set aws_s3_bucket_public_access_block; private ACLs cover most of the rest. No public bucket reached the confirmed findings.

DLQ wiring present

Dead-letter queues appear in nearly every async-worker config. The model understands the pattern — it just forgets to alarm on the DLQs.

Encryption at rest

44 of 50 directories reference encryption; 39 of 50 create a KMS key. The at-rest story is broadly present, not bolted on.

The false-positive moat

audytx's cross-resource reasoning suppressed 85 findings a single-resource matcher would have fired — 71 of them IAM-role findings on service roles that are scoped and non-escalatable. Built and tuned on human-written Terraform, the suppression generalizes to AI-generated Terraform too.

Methodology caveats

One model, one date. This is claude-opus-4-8 on 2026-06-14 only. The harness is built for multi-model follow-up; that's out of scope for v1. The flagship is the strongest case — if it ships these gaps, smaller or older models likely ship more.
The prompt set is ours. 50 prompts we authored, not a random sample of real developer requests.
audytx authored the scanner and this study. Conflict of interest, disclosed up front. The corpus, manifest, and raw scan results are committed to the testbed repo so anyone can re-scan with another tool. terraform validate results are tool-independent.
Findings ≠ vulnerabilities. Most are reliability/cost/observability gaps, not exploitable holes. The category and severity splits are there precisely so the raw total (686) isn't misleading.
Post-publication note on AWS_VPC_004. The v0.5.1 scan reported 47 "Needs Review" findings for security groups with unknown port exposure. After freezing this study, analysis found these were scanner false positives: the AI-generated code uses the newer aws_vpc_security_group_ingress_rule resource pattern, which the v0.5.1 engine couldn't resolve. Fixed in audytx v0.14.33. The AI-generated code wasn't actually exposing debug ports.

Reproduce this study

The corpus, manifest, validate results, and scan results are committed to the audytx-testbed repo on the bench/ai-claude branch:

corpus/ai-claude/           # raw Terraform outputs (50 dirs)
corpus/ai-claude/manifest.yaml   # model ID, date, prompt hash per dir
results/ai-claude/validate/      # terraform validate per dir
results/ai-claude/audytx.json    # per-dir MCP scan results (v0.5.1)

Re-scan any directory against the live engine:

curl -s https://audytx.com/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0", "id": 1,
    "method": "tools/call",
    "params": {
      "name": "scan_terraform",
      "arguments": { "files": [{"path": "main.tf", "content": "..."}] }
    }
  }'

Data & citations

Datasets used

AI corpus — 50 Terraform configurations (1,421 resources) generated one-shot by claude-opus-4-8 on 2026-06-14. Raw outputs, per-directory prompt manifest, terraform validate results, and scan results are versioned in audytx-testbed on the bench/ai-claude branch.
terragoat — the deliberately-vulnerable Terraform training corpus (bridgecrewio/terragoat), scanned with the identical per-directory method as the vulnerable reference point.
Clean production modules — the 21-module hand-written corpus from the audytx benchmark methodology, which also anchors the false-positive comparison.

Related work

arXiv:2512.14792 — documents the cross-resource dependency failure mode in LLM-generated infrastructure code; this study's Secrets-Manager-rotation result (11 of 18 HIGH findings) independently corroborates it on a frontier model.

Cite this study

Reuse the numbers, tables, or corpus with attribution:

audytx (2026). "What Frontier Models Get Wrong About AI-Generated
Terraform Security." June 2026.
https://audytx.com/research/ai-terraform-2026

@misc{audytx2026aiterraform,
  title  = {What Frontier Models Get Wrong About AI-Generated Terraform Security},
  author = {audytx},
  year   = {2026},
  month  = {jun},
  url    = {https://audytx.com/research/ai-terraform-2026}
}

Frequently asked questions

Is AI-generated Terraform more or less secure than hand-written?

On par with hand-written production Terraform on the metrics we measured. The AI corpus (50 configurations, 1,421 resources, generated with Claude Opus 4.8) produced 48.3 findings per 100 resources, compared to 52.2 findings per 100 resources across clean, hand-written production Terraform modules — and far below terragoat's deliberately vulnerable 269.1 findings per 100 resources.

What's the most common security gap in AI-generated Terraform?

Secrets Manager rotation. 11 of the 18 HIGH-severity findings — 61% of all HIGH findings — trace to one cross-resource failure pattern: the model creates an aws_secretsmanager_secret for a database password but never creates the companion aws_secretsmanager_secret_rotation resource.

Fix it where the agent writes it

The gaps in this study — the unwired rotation, the escalation-capable role, the invented argument — are cheapest to close in the same session that wrote the code. Wire audytx into your coding agent over MCP and it scans its own Terraform before the pull request exists. Setup is one config entry and one standing instruction per agent:

Claude Code →

One claude mcp add, then a line in CLAUDE.md so it scans before every PR.

Cursor →

A .cursor/mcp.json server entry plus an always-on project rule.

GitHub Copilot →

A .vscode/mcp.json server plus .github/copilot-instructions.md.

Open beta

Start free during the open beta

Audit your AI-generated Terraform where it's written — audytx posts findings straight into your pull request with cross-resource context, one-click fixes, and false-positive suppression that states its rationale.

Install the GitHub App Read the docs

Also available as an MCP server for Claude Code, Cursor, and any MCP-compatible agent. See the benchmark for the false-positive comparison.