How to Use Atmos with Existing Terraform

Problem

Not everyone beginning to implement Atmos is starting with a clean slate, and many users would like to put Atmos to use with their existing Terraform infrastructure. However, the SweetOps approach is highly opinionated and as a result, can be intimidating for new users to begin to apply the practices to existing infrastructure. How can we leverage Atmos to incrementally build off our existing Terraform infrastructure and begin to DRY variable configurations?

Solution

Applying Atmos to existing infrastructure can be an easy process if broken into a few parts. The following sections will outline each key concept with Atmos and the steps to apply each to your existing Terraform code.

tip

TL;DR: Follow the 5 steps below for a detailed explanation of each step.

1. Copy all terraform components to components/terraform/. See Components

2. Create each stack under stacks/. See Stacks

3. Define each component within stacks/catalog/ and import into stacks. See How to Use Imports and Catalogs in Stacks

4. Copy existing Terraform variable definitions into the component defaults, catalog definitions, or stack configuration.

5. Import existing state using Atmos. See Atmos

Part 1: Organization

The first and foremost key to success with Atmos is organization. Organization of Terraform components, variables, and environment configuration naturally lead to DRY code, easier understanding, and less room for errors. With Atmos, we organize all components (Components) and variables configuration, referred to as stacks (Stacks), into respective folders. An example layout would look like the following:

infrastructure/
├── ...
├── components/
│   ├── terraform/
│   │   ├── ec2-instance/
│   │   └── vpc/
└── stacks/

Steps

1. Decide on a unique name to give your components. Make sure if follows our Components.

2. Create the folder in components/terraform with the given name of the component

3. Move all existing Terraform code to the path components/terraform.

Part 2: Stacks

Stacks are what Atmos uses to define configuration and variables for each component in any environment. Our standard convention is that each stack operates in a given region and environment; for example, ue1-nonprod.yaml defines the configuration for nonprod operating in the us-east-1 region. Anything included in these stacks should be unique to that given region and environment or be imported by a global configuration file. Read more about Stacks

infrastructure/
├── ...
├── components/
└── stacks/
    ├── use1-nonprod.yaml
    ├── usw1-nonprod.yaml
    ├── use1-prod.yaml
    └── usw1-prod.yaml

Steps

1. Create a yaml file for each environment and region

2. Use the following as an example to start each stack. Only include unique configuration for the given environment and region.

vars:
  stage: nonprod
terraform:
  vars: {}
components:
  terraform: {}

Part 3: Catalog

The catalog defines every component available to use. Each component is given a configuration file within the catalog where common config is set.

For example, a basic configuration for a component called ec2-instance would look like the following:

components:
  terraform:
    ec2-instance:
      backend:
        s3:
          workspace_key_prefix: ec2-instance
      vars:
        enabled: true

When we want to use a component with a stack, we can import that component from the catalog. Then that given stack can define unique variables for that component. Read more about the catalog and how to use imports under How to Use Imports and Catalogs in Stacks

Steps

1. Create the catalog folder under stacks/

2. Create a yaml configuration file for each component.

3. Import the component into any stack file you want the component deployed, using the path to the catalog entry. For example:

import:
  - catalog/ec2-instance

Part 4: DRY Components

Another key concept with Atmos is to avoid all variable repetition; any non-unique variable should only be specified a single time. With Atmos, they are usually three places to define a variable:

Component variables

There are variables defined with the Terraform component in components/terraform and are nonspecific to your organization. Typically default values are included with defaults.tfvars. For example:

# components/terraform/ec2-instance/default.auto.tfvars
enabled = false

Catalog variables

Variables defined within the catalog can be specific to your organization but should be nonspecific to any region or environment. For example, a catalog configuration might look like the following:

# stacks/catalog/ec2-instance.yaml
components:
  terraform:
    ec2-instance:
      backend:
        s3:
          workspace_key_prefix: ec2-instance
      vars:
        enabled: true
        instance_type: t3.xlarge
        name: example

Stack variables

Stack variables include any variables defined specifically for an environment, region, or a combination of the two. Define these in the most optimal way to avoid any repetition between stacks, and use imports as necessary to pull shared configuration. In this example below, the us-east-1 sandbox stack pulls the shared configuration for us-east-1, imports the ec2-instance component, and defines unique variables for the vpc component.

# stacks/use1-sbx01.yaml
import:
  - use1-globals
  - catalog/ec2-instance
vars:
  stage: sbx01
terraform:
  vars: {}
components:
  terraform:
    vpc:
      vars:
        cidr_block: 10.88.0.0/18

Steps

1. Copy any unique variables from the Terraform component into the highest stack level possible

2. Remove any non-general variables from the Terraform component

Part 5: Importing State

Atmos supports all built-in Terraform Subcommands, including import. Once the components and stacks are properly created, running atmos terraform import <component-name> <resource-name> <resource-id> -s <stack-name> will pull the existing state for any resource. Each resource in the existing Terraform state will need to be imported into the new Atmos-created Terraform state. See the Terraform documentation on import for additional details.

Summary

At this point, all Terraform components should be defined under components/terraform and all variables configurations listed in stacks/catalog. Any deployment into a given environment and region is listed within the given stack with an import. For example:

infrastructure/
├── ...
├── components/
│   ├── terraform/
│   │   └── ec2-instance/
└── stacks/
    ├── catalog
    │   └── ec2-instance.yaml
    ├── use1-nonprod.yaml
    ├── usw1-nonprod.yaml
    ├── use1-prod.yaml
    └── usw1-prod.yaml

Next Steps

Now that your environment is properly configured, follow the Atmos documentation for running commands.