Structure of Terraform S3 State Backend Bucket

This guide explains the structure of a Terraform S3 state backend bucket, including the use of workspaces, key prefixes, and buckets. It details how the backend.tf.json file is used to configure the S3 backend for storing Terraform state, and how S3 native state locking provides consistency checking. The document provides examples and best practices for managing and accessing the Terraform state backend.

Understand the anatomy of a Terraform S3 state backend bucket and how workspaces, key prefixes and buckets are used.

S3 Native State Locking

As of Terraform 1.10 and OpenTofu 1.8, the S3 backend supports native state locking using S3's conditional writes feature. This eliminates the need for a separate DynamoDB table for state locking.

To enable S3 native state locking, set use_lockfile = true in your backend configuration. This uses S3 conditional writes to create a .tflock file alongside your state file, providing the same consistency guarantees as DynamoDB locking without the additional infrastructure.

tip

S3 native state locking is the recommended approach for new deployments. It simplifies your infrastructure by removing the DynamoDB dependency while maintaining the same locking guarantees.

The backend.tf.json File

This file is programmatically generated by Atmos using all the capabilities of Stacks to deep merge. Every component defines a backend.tf.json, which is what distinguishes it as a root module (as opposed to a terraform child module). The backend tells terraform where to access the last known deployed state of infrastructure for the given component. Since the backend is stored in S3, it’s easily accessed by in a distributed manner by anyone running terraform.

info

An identical backend.tf.json file is used by all environments (stacks). Environments are selected using the terraform workspace command, which happens automatically when using atmos together with the --stackargument.

For reference, this is the anatomy of the backend configuration: (note this is just a JSON representation of HCL)

{
  "terraform": {
    "backend": {
      "s3": {
        "acl": "bucket-owner-full-control",
        "bucket": "acme-ue2-root-tfstate",
        "encrypt": true,
        "key": "terraform.tfstate",
        "profile": "acme-gbl-root-terraform",
        "region": "us-east-2",
        "use_lockfile": true,
        "workspace_key_prefix": "vpc"
      }
    }
  }
}

note

• Either profile or role_arn can be used for authentication
• The use_lockfile setting enables S3 native state locking (requires Terraform 1.10+ or OpenTofu 1.8+)

S3 Backend

The S3 bucket is created in the cold start using the tfstate-backend component provisioned in the root account.

The state format is s3://{bucket_name}/{component}/{stack}/terraform.tfstate

• The bucket name format is {namespace}-{optional tenant}-{environment}-{stage}-tfstate

• We deploy this bucket in the root account so here are some example bucket names

acme-ue2-root-tfstate (without tenant) acme-mgmt-ue2-root-tfstate (with tenant: mgmt)

• The component name provided is used as the terraform state’s workspace_key_prefix in each component’s backend.tf.json. Therefore, this will be the first s3 key after the bucket name.

• The stack is where the component is provisioned and the name of the workspace created

• Finally, the terraform.tfstate is the key provided in each component’s backend.tf.json

The terraform commands run by atmos for the backend s3://acme-ue2-root-tfstate/vpc/ue2-prod/terraform.tfstate

atmos terraform deploy vpc --stack ue2-prod
| atmos will create the input variables from the YAML and run the following commands
| -- terraform init
| -- terraform workspace ue2-prod
| -- terraform plan
| -- terraform apply

To better visualize what’s going on, we recommend running the commands below to explore your own state bucket. Make sure to use the correct profile for your organization (acme-gbl-root-admin is just a placeholder).

Find the bucket. It should contain tfstate in its name. In the example below, we can see the vpc component is deployed to use2-auto, use2-corp, use2-dev, use2-qa, use2-sbx01, use2-staging. As you can see, the workspace is constructed as the {environment}-{stage}. This setting is defined in the atmos.yaml config with the stacks.name_pattern setting (see Atmos for all settings).

$ aws --profile acme-gbl-root-admin \
  s3 ls --recursive
...
2021-11-01 19:53:48     120926 vpc/use2-auto/terraform.tfstate        # workspace key prefix: vpc, workspace name is `use2-auto`
2021-11-01 19:49:12     123604 vpc/use2-corp/terraform.tfstate
2021-11-01 19:50:18     123486 vpc/use2-dev/terraform.tfstate
2021-11-01 19:48:39     123354 vpc/use2-qa/terraform.tfstate
2021-11-01 19:49:46     123735 vpc/use2-sbx01/terraform.tfstate
2021-11-01 19:50:50     124014 vpc/use2-staging/terraform.tfstate

See where all the VPC components contain state

aws --profile acme-gbl-root-admin \
  s3 ls s3://{bucket_name}/vpc/

note

If a component is mistakenly deployed somewhere and destroyed, a leftover terraform.tfstate file will be present on your local filesystem with a small file size so while this is a good way to search for backends, it's not the best way to determine where a component is deployed. Also, the S3 bucket has versioning enabled, ensuring we can always (manually) revert to a previous state if need be.

DynamoDB Locking (Optional)

info

DynamoDB locking is optional and primarily retained for backwards compatibility with existing deployments. For new deployments, we recommend using S3 native state locking with use_lockfile = true instead.

If you have an existing deployment using DynamoDB for state locking, or prefer to use DynamoDB, you can configure it by adding the dynamodb_table field to your backend configuration:

{
  "terraform": {
    "backend": {
      "s3": {
        "bucket": "acme-ue2-root-tfstate",
        "dynamodb_table": "acme-ue2-root-tfstate-lock",
        "encrypt": true,
        "key": "terraform.tfstate",
        "region": "us-east-2",
        "workspace_key_prefix": "vpc"
      }
    }
  }
}

To find and inspect a DynamoDB lock table:

# Find the table (should contain `tfstate-lock` in its name)
aws --profile acme-gbl-root-admin \
  dynamodb list-tables

# Get a LockID
aws --profile acme-gbl-root-admin \
  dynamodb get-item \
    --table-name {table_name} \
    --key '{"LockID": {"S": "{bucket_name}/{component}/{stack}/terraform.tfstate-md5"}}'

References

• https://www.terraform.io/docs/language/settings/backends/s3.html backend configuration documentation.