Terraform Best Practices: Files

A fantastic feature of Terraform is that the code is simply a set of files in a directory with the *.tf extension (or the *.tf.json extension). It doesn’t include subdirectories and we would call this our “working directory” when we’re using our code editor or we’re in the shell. We can check the directory with the pwd command on UNIX-type shells.

Another practice frequented by tutorials leverages this capability and creates separate files for everything.

For example, a narrative might include:

• I need to create a VPC and some related resources, so I will make a networking.tf
• I need to create an EC2 instance and related resources, so I will make a compute.tf

The number of files can grow quickly and the flexibility is nice because resources can be organized with this pattern. However, the standard practice that has been adopted by the Terraform community is to have just three (3) Terraform files:

• main.tf which contains all of the principal code (a Go-ism that worked its way into Terraform… many exist)
• variables.tf which contains all of the variable declarations
• outputs.tf which contains all of the output declarations

Why just the three files? What about when my code continues to grow to a thousand lines, or more? Why not just put everything in the main.tf, then?
There isn’t much discussion out in the wild that speaks to why this is the practice, but we can apply our thoughts to it.

The main.tf file is a pretty straight forward concept. It is our main file (hence the name). This is a standard practice in Go, as well, which is what Terraform is written in. Here we will include all of our resource, data source, and module blocks, as well as our primary terraform block and our provider blocks. We can also use as many locals blocks. What about when the main.tf it several hundred or more lines? That is a point of feedback. If the file is getting too large with too many resources, that is a cue to reduce what you’re doing in the file, either by shifting to a module that deploys certain resources in a more simplified way, or limiting the scope of what your code is intended to do. Adopt the “UNIX philosophy” of doing one thing well, for modules (this doesn’t necessarily mean one resource for a module, which should be the exception). Decompose resources in chunks where they can be re-used and have separate lifecycles. This aids in collaboration, as well.

Separating out the variables.tf seems widely adopted, but why? If you’re in your code editor, it can be really handy to see your variable declarations. Sometimes you’re working with what can be a complex schema if you have nested maps, optional properties with defaults, and more. If you keep the variable declarations in the same file, you then may need to scroll to the variable declarations to check the schema before returning to a resource to set the property to the appropriate value in the variable. Having it as a separate file allows to have the main.tf and the variables.tf open side-by-side in the code editor and the variables file can have the exact variable declaration presented while your main.tf is open to the location where you’re working with that variable in a for_each, or otherwise.
The outputs.tf is similar to the variables.tf. When dealing with multiples of a given resource as the result of a loop, it can be necessary to compare the resource and the outputs if you’re interested in a specific property from the data structure.

But with so much flexibility, why should we limit our code to the main.tf file? Toggling between many files can become rather tedious. Where was that resource I want to reference? There was a locals defined for manipulating some data, which file was it in? Keeping the primary code in file reduces these issues. It is about workflow and efficiency, simply put.

Many times, there is some statement from a CIO or other leader wanting to have all of the infrastructure defined in a single codebase. That is an absolute extreme that is likely intuitive to realize that it is the wrong approach, but there is a lot of gray area where it may seem like a matter of style and opinion. There are many principles that can guide this decision that we’ll review at another time.

There are reasons to deviate from this practice, however.  These relate to temporary files in Terraform.  I am not talking about files that go in a temp directory, but files that don’t need to be long-lived.  Some of these files should be checked into version control, and others may never live in version control.

Config-driven state management blocks are an example that should be tracked and committed to the code repository, but they can and should be removed (they still be in the commit history with Git Blame and all).  These include import, moved, and removed blocks.  I will make a file intuitively named, import.tf, moved.tf, and removed.tf, respectively.  This ensures that the operation is tracked and follows GitOps principles so that it is captured.  Using the CLI equivalents doesn’t accomplish this.  If we were to compare this Kubernetes, these operations would be treated more imperatively, rather than declarative.

If you’re using a CI platform to execute Terraform, it can become tedious to configure your backend, and also quite limiting in terms of code re-use.  The terraform block only accepts literal values which creates a challenge.  This can be overcome, to a degree, by relying on arguments being passed via the CLI for the details of the backend authentication.  However, cluttering up a terraform init statement in a pipeline is tedious.  Instead, as a result of this feature in Terraform that allows for any number of files with a *.tf extension to be used, a shell script with heredoc interpolation can be used to generate a Terraform file on the fly that is never checked into version control, but it can be captured and stored as an artifact for auditing preservation and troubleshooting.

Here is an sample used with GitHub Actions that pulls in variables using OIDC to reach Azure Storage:

This will create a file in the working directory called backend.tf containing the details about the Azure Storage account.  It also provides for plenty of flexibility because you can have the code do exactly what you want and you’re not limiting your ability to use this code for a different environment which would have a different state file.  Since it isn’t in version control, you’re not going leak secrets into your repo, either (although this example uses OIDC, so there are no secrets to begin with).  Also, if you have a question regarding this, you can have multiple terraform blocks in your configuration, which is generally the case whenever you have blocks of any type, with the main exception being when a provider limits it for a particular resource in a nested block.

Personally, I find the heredoc interpolation cleaner and more readable than using partial configuration options on the CLI.  If you use a backend file, it would still be preferable to generate it on the fly, with heredoc, so that any potential secrets aren’t checked in.  Also, it can be cluttered to have a series of CLI arguments that define this:

However, if that is your preference, it accomplishes the same result.

That’s wraps up this discussion, for now.  As with the last discussion, you may not be entirely sold, but the best practices build on each other and support more flexibility down the line, so when we get to other best practices, they can be dependent on something like this, to some degree.