AI-Generated Terraform in 2026: Benchmarks, Failure Modes, Guardrails

Most of the Terraform I review now has a model somewhere in its history: an autocomplete suggestion, a chat answer pasted in, or a whole module an agent drafted while I was doing something else. I let it in, because writing aws_s3_bucket_public_access_block by hand for the four hundredth time teaches nobody anything. But I also read the research on AI-generated Terraform, and it is sobering. This post covers what the benchmarks measured when LLMs write infrastructure as code (Terraform mostly, plus Kubernetes YAML and Helm), the specific ways the output goes wrong, and the guardrails that catch each failure before it becomes a change to a real account. Short version: syntax is now mostly fine, intent and security are not, and the fix is the same plan, policy and review discipline we should have had anyway.

How accurate is AI-generated Terraform?

Less accurate than its Python, by a wide margin. The reference point is IaC-Eval, a NeurIPS 2024 benchmark of 458 human-written AWS Terraform scenarios, each scored by whether the generated program actually matches the user's intent, not just whether it parses. GPT-4, the best model tested, managed 19.36% pass@1. The same model scored 86.6% on EvalPlus, a Python benchmark. Few-shot prompting, chain-of-thought and multi-turn repair helped only a little.

That was 2024 and GPT-4, so the obvious objection is that newer models fixed it. The newer papers say partly. DPIaC-Eval, accepted at FSE 2026, gave six models 153 real-world tasks across 58 AWS services (CloudFormation rather than Terraform) and checked whether the result actually deployed. First-attempt deployment success was 20.8% to 30.2%. Templates that deployed covered the user's stated requirements 25.2% of the time, and only 8.4% passed a standard security policy check. An iterative loop that fed deployment errors back to the model got 54.6% to 91.6% of templates deployable within ten attempts, which is a useful result in itself: the model is much better at fixing errors you show it than at avoiding them.

A December 2025 error-taxonomy study by Nekrasov and colleagues extended IaC-Eval with cloud emulation and found 27.1% overall success at baseline. Injecting provider documentation through Graph RAG raised technical validity to 75.3% and overall success to 62.6%. But intent alignment plateaued, and the authors named the result a "Correctness-Congruence Gap": better coders, still poor architects of what you actually asked for.

Security is its own axis. A security-first benchmark published in August 2026 (Vargas, Mansilha and Kreutz, accepted at SBSeg 2026) ran seven models, including Claude Opus 4, GPT-5.4 and Gemini 2.5 Pro, across 17 AWS Terraform scenarios and scanned the output with Checkov and Trivy. Validity and security turned out to be largely independent. WizardCoder-33B produced Terraform that passed terraform validate 77.8% of the time and passed zero Checkov checks. Claude Opus 4 with a detailed security prompt reached 23.1% Checkov compliance and a 92.5% Trivy pass rate. Same scenarios, two scanners, very different numbers, which is a point about scanners as much as models.

Study	What it measured	Headline result	Source
IaC-Eval (NeurIPS 2024)	458 AWS Terraform tasks, intent-checked	GPT-4 19.36% pass@1 vs 86.6% on Python	NeurIPS
DPIaC-Eval (FSE 2026)	153 CloudFormation tasks, deployed for real	20.8% to 30.2% deploy first try; 8.4% security compliant	arXiv 2506.05623
Nekrasov et al. (Dec 2025)	IaC-Eval plus emulation and Graph RAG	27.1% baseline, 62.6% with docs injected	arXiv 2512.14792
Vargas et al. (SBSeg 2026)	17 Terraform tasks, Checkov and Trivy	Opus 4: 23.1% Checkov, 92.5% Trivy	arXiv 2608.02672
TerraFormer (ICSE 2026)	Fine-tuning with verifier feedback	+15.94% correctness on IaC-Eval over base	arXiv 2601.08734

Why do LLMs get infrastructure code wrong?

Three reasons keep coming up in the papers. First, there is less of it. Sonar's summary of the TerraFormer paper says that across 17 models Terraform did worse than YAML and JSON formats, which the authors attribute to HCL being rarer in training data. Second, providers move. The AWS provider is at v6.69.1 as I write, and a model trained on blog posts from the v3 era will confidently produce v3 shapes. Third, infrastructure correctness is not local. A Python function either returns the right value or it does not; a security group is "correct" only relative to the network, the identity model and the compliance regime around it, none of which is in the prompt.

What failure modes should you look for?

Sonar's post groups the recurring problems into four buckets, and they match what I see in review:

• Permissive defaults. Action = "*" in IAM policies, 0.0.0.0/0 ingress, public ACLs. The model optimises for "it works", and wide open always works.
• Silent omissions. No encryption block, no access logging, no deletion protection. Nothing errors, because these are optional arguments.
• Hardcoded values. Literal CIDRs, account IDs, and occasionally a credential in a default.
• Stale patterns. Provider syntax from two major versions ago.

I would add hallucinated arguments, and it is worth knowing exactly which of these the cheap tools catch. I ran a typical model-style S3 bucket through terraform validate with Terraform 1.16.6 and AWS provider 6.69.1. An invented encryption = "AES256" argument fails immediately with An argument named "encryption" is not expected here. But the v3-era inline versioning { enabled = true } block and acl = "private" pass, with only a deprecation warning and exit code 0. Validation catches invention, not obsolescence, and catches nothing about security.

Kubernetes YAML has the same split. A manifest I wrote the way models often do, with resources.limit instead of limits and a policy/v1beta1 PodDisruptionBudget, sailed through my eyes the first time. kubeconform -strict against Kubernetes 1.34 schemas flagged both: "additional properties 'limit' not allowed", and no schema at all for the PDB, because that API version stopped being served in v1.25. For Helm, render first (helm template) and validate the output, since the chart source tells you little.

The supply-chain variant matters too. Spracklen and colleagues (USENIX Security 2025) found code models hallucinating package names in at least 5.2% of commercial-model samples and 21.7% of open-source ones, with 205,474 unique invented names. That study covered Python and JavaScript, not the Terraform Registry, so I would not quote a module rate. But a model suggesting source = "someone/vpc-complete/aws" that nobody vetted deserves the same suspicion as a suggested npm package.

Do MCP servers fix hallucinated Terraform?

They fix the retrieval part. HashiCorp's Terraform MCP server (v1.3.0, August 2026) gives an assistant live access to the Registry: provider docs, latest versions, module details and Sentinel policies. That is exactly the documentation injection that moved the Nekrasov results from 27.1% to 62.6%, so I use it, and the default registry toolset is read-only public data.

The trouble is the other toolset. With a TFE_TOKEN, the terraform toolset can list and create runs, edit variables, create workspaces and grant team access. According to the tool reference, applying runs, deleting workspaces and force-unlocking sit behind ENABLE_TF_OPERATIONS=true, as do the auto_approve and is_destroy options on create_run, but create_workspace and update_workspace are described as destructive and are not gated. The README is blunt about it: "Do not use the MCP server with untrusted MCP clients or LLMs." My rule is --toolsets=registry on developer machines, and any HCP token an agent holds is read-only. I covered the wider risk in MCP security in 2026.

Which guardrails actually catch AI-written IaC?

Layer them by cost, so cheap checks run on every keystroke and expensive ones run once per pull request. The benchmarks tell you why each layer exists: validate catches invention, linters catch provider-specific nonsense (TFLint's AWS ruleset knows t3.mega is not an instance type), scanners catch the permissive defaults and omissions, and a policy on the plan catches the thing nothing else sees, which is what will happen to existing resources.

This is the gate I put in front of any agent-authored infrastructure change. Every tool is pinned; the versions are the current releases as of today.

#!/usr/bin/env bash
# gate.sh: run on every PR that touches *.tf. Fails fast, cheapest first.
# Pinned: Terraform 1.16.6, TFLint 0.64.0, Checkov 3.3.29,
#         Trivy 0.76.0, Conftest 0.71.1
set -euo pipefail

terraform fmt -check -recursive          # style drift, unreadable diffs
terraform init -backend=false -input=false
terraform validate                       # invented arguments, bad references

tflint --init && tflint --recursive      # provider rules: bad instance types etc.

checkov -d . --framework terraform --quiet --compact
trivy config --severity HIGH,CRITICAL --exit-code 1 .

# The plan is the real review artefact. Credentials here are read-only.
terraform init -input=false
terraform plan -out=tfplan -input=false -lock=false
terraform show -json tfplan > tfplan.json
conftest test tfplan.json --policy policy/   # deletes need a human

The policy that matters most is tiny. It denies any plan that deletes or replaces a resource, which forces a human to look at exactly the changes that cannot be undone:

# policy/destroy.rego  (OPA v1 syntax, Conftest 0.71.1)
package main

import rego.v1

deny contains msg if {
	some rc in input.resource_changes
	"delete" in rc.change.actions   # also matches replace (delete + create)
	msg := sprintf("%s would be deleted (actions: %v); needs a human-approved label",
		[rc.address, rc.change.actions])
}

I tested it against a real plan JSON from Terraform 1.16.6 that removes a resource. Conftest reported FAIL ... terraform_data.db would be deleted (actions: ["delete"]) and exited 1. In CI, the escape hatch is a label only a CODEOWNER can apply, not a flag the agent can pass. My policy as code for agents post goes further into where these checks should live.

Two cautions about the scanner layer. First, scanners disagree, as the 23.1% versus 92.5% result shows, so run two if you can afford the noise. Second, the scanners are themselves supply chain. In March 2026 attackers force-pushed malicious commits to 76 of 77 version tags of aquasecurity/trivy-action, stealing CI secrets before running a normal scan. Pin actions to commit SHAs, not tags.

For the hands-on version, with mocks, terraform test, TFLint, Checkov and a full CI workflow, see my sibling guide: how to test Terraform.

Should an AI agent ever run terraform apply?

Not against anything you would miss. The cautionary case from February 2026, which I wrote about in infrastructure as code in 2026, was an agent running terraform destroy on a production stack after a state mix-up. Gruntwork's analysis lists six conditions that had to hold at once: local state, no state integrity, direct CLI execution, broad credentials, no review checkpoint and no deletion protection. None of those is an AI problem. Its recommendation is to keep AI "as an assistant, not an operator", with plans in CI, apply after merge and separate read and write credentials.

That maps cleanly onto the research. Models are good at repairing errors you feed them, so let them iterate freely against validate, linters, scanners, mocked tests and a read-only plan. Keep the apply step, and the credentials that make it possible, on the far side of a human-reviewed merge. If the agent runs in an isolated sandbox with no cloud credentials at all, a bad suggestion stays a bad suggestion. (I build PandaStack, a microVM sandbox for agents, so weigh that last sentence accordingly.)

FAQ

Can ChatGPT or Claude write production-ready Terraform?

They can write Terraform that parses and often deploys, but the research says it rarely matches intent and security requirements on the first try. IaC-Eval measured 19.36% pass@1 for GPT-4, and DPIaC-Eval found 8.4% of deployable templates passing a security policy check. Treat the output as a draft that must pass validation, scanning and plan review.

Does terraform validate catch AI hallucinations?

It catches invented arguments and broken references, but not deprecated syntax or insecure configuration. In my test, a made-up encryption argument failed validation while a deprecated inline versioning block passed with only a warning. You need TFLint, a scanner such as Checkov or Trivy, and plan review on top.

Is it safe to connect an AI assistant to the Terraform MCP server?

The registry toolset is low risk and improves accuracy, because it gives the model current provider documentation. The HCP Terraform toolset can create and modify workspaces and runs when given a token, and some of those tools are not behind the ENABLE_TF_OPERATIONS gate. Use --toolsets=registry locally and read-only tokens for anything else.

How do I validate AI-generated Kubernetes YAML and Helm charts?

Run kubeconform -strict with the Kubernetes version you deploy to, which catches misspelt fields and removed API versions. For Helm, render with helm template and validate the rendered output, then apply admission policy (Kyverno, Gatekeeper or ValidatingAdmissionPolicy) in the cluster as the last line.

This quarter I would do three things: put the gate above in front of every repository where an agent can open a pull request, enable the Terraform MCP server in registry-only mode so the models stop guessing at arguments, and move every apply credential out of developer machines and into a merge-triggered pipeline. None of it is new. The models just made skipping it more expensive.

Related: Infrastructure as Code in 2026: Two Forks at 1.16 and 1.13, a $6.4B Owner, 912 Public State Files, and One Agent That Ran terraform destroy, Policy as Code for AI Agents in 2026: Cedar at the Gateway, Rego in the Hook, and the 11.2% the Model Still Lets Through and MCP Security in 2026: The Protocol Got Hardened. The Ecosystem Didn't.