The Problem

While all companies are unique, their infrastructure doesn't need to be. Well-built infrastructure consists of reusable building blocks that implement all the standard components like servers, clusters, load balancers, etc. Rather than building everything from scratch “the hard way”, there's an easier way. Using our “reference architecture” and its service catalog of all the essential pieces of infrastructure, everything a business needs can be composed together as an architecture using “Stack” configurations. Best of all, it's all native terraform.

AI generated voice

Our Solution

Cloud Posse defines components. Components are opinionated, self-contained building blocks of Infrastructure-as-Code (IAC) that solve one specific problem or use-case. Components are similar to a Terraform root module and define a set of resources for any deployment.

• Terraform components live under the components/terraform directory.
• Cloud Posse maintains a collection of public components with terraform-aws-components
• The best components are generic enough to be reused in any organization, but there's nothing wrong with writing specialized components for your company.
• Detailed documentation for using components with Atmos can be found under atmos.tools Core Concepts

Pro tip!

We recommend that you always check first with Cloud Posse to see if we have an existing component before writing your own. Sometimes we have work that has not yet been upstreamed to our public repository.

Prerequisites

In order to be able to create a new component, this document assumes the developer has the following requirements:

• Authentication to AWS, typically with Leapp
• The infrastructure repository cloned locally
• Geodesic up and running
• A basic understanding of Atmos
• An intermediate understanding of Terraform

1 Create the component in Terraform

• Make a new directory in components/terraform with the name of the component

• Add the files that should typically be in all components:

  .
  ├── README.md
  ├── component.yaml # if vendoring from cloudposse
  ├── context.tf
  ├── main.tf
  ├── outputs.tf
  ├── providers.tf
  ├── remote-state.tf
  ├── variables.tf
  └── versions.tf

  All the files above should look familiar to Terraform developers, except for a few of the following.

  context.tf

  Cloud Posse uses context.tf to consistently set metadata across all resources. The context.tf is always identical. Copy it exactly from here.

  curl -sL https://raw.githubusercontent.com/cloudposse/terraform-null-label/master/exports/context.tf -o context.tf

  providers.tf

  • For providers.tf, if we are just using AWS providers only, copy it from our common files (commonly referred to as mixins) folder.
  • If we are using Kubernetes, then you may need an additional providers file for Helm and Kubernetes providers. Also copy this file from the mixins folder.

  remote-state.tf

  By convention, we use this file when we want to pull Terraform Outputs from other components. See the remote-state Module.

  component.yaml

  The component manifest is used for vendoring the latest version from Cloud Posse. More on this later.

2 Add Terraform Modules or Resources

• In your main.tf, or other file names of your choosing, add configurations of Terraform modules and/or resources directly.
• When you use a Cloud Posse module, you should pass the context metadata into the module, like this. All Cloud Posse modules have the variable context, which you pass module.this.context
• You could also use other external modules that are not provided by Cloud Posse.
• Use module.this.tags when you want to pass a list of tags to a resources or module not provided by Cloud Posse. Tags are already included with var.context for any Cloud Posse module.
  tip

  Cloud Posse has a lot of open source modules, so check here first to avoid repeating existing effort.

• Handle the variable module.this.enabled, so that resources are not created when var.enabled is set to false. Cloud Posse modules will do this automatically when passed var.context. When adding a resource or using a non-Cloud Posse module, then configure enabled with a count, for example count = module.this.enabled ? 1 : 0
• Use remote-state to read Terraform Output from other components. For example the eks/alb-controller component

3 Configure Stacks

Directory Structure

• Put all Stack configurations in the stacks directory

  stacks/
  ├── catalog/
  ├── mixins/
  ├── orgs/
  └── workflows/

• Default configurations for a component live in the catalog directory, and configurations for deployed accounts live in the orgs directory.

• All files in the catalog and orgs directories are stack configuration files. Read on for more information.

Define defaults for a component in the catalog directory

• The Stack Catalog is used to define a component's default configuration for a specific organization. Define variables that would not be shared in an Open Source setting here.
• By convention, name the configuration the same as your component. For example, if your component is components/terraform/foobar then the file would be named stacks/catalog/foobar.yaml

  components:
    terraform:
      foobar:
• Above, foobar is the name of the component.
• Pass variables into Terraform like this:

  components:
    terraform:
      foobar:
        vars:
          sample_variable_present_in_variables_tf_of_component: "hello-world"

Component Types

• Atmos supports component types with the metadata.type parameter

• There are two types of components:

  real

  is a "concrete" component instance

  abstract

  a component configuration, which cannot be instantiated directly. The concept is borrowed from abstract base classes of Object Oriented Programming.

• By default, all components are real

• Define an abstract component by setting metadata.type to abstract. See the following example

  components:
    terraform:
      foobar/defaults: # We can name this anything
        metadata:
          type: abstract # This is what makes the component `abstract`
          component: foobar # This needs to match exactly an existing component name
        vars:
          tags:
            team: devops

  For more details, see atmos.tools

• With an abstract component default, we can inherit default settings for any number of derived components. For example:

  components:
    terraform:
      foobar: # Since this component name matches exactly, we do not need to add `metadata.component`
        metadata:
          type: real # This is the default value and is only added for visibility
          inherits:
            - foobar/defaults # The name of the `abstract` component
        vars:
          sample_variable_present_in_variables_tf_of_component: "hello-world"

  Now foobar will uses the same configuration as foobar/defaults but may describe additional variables.

4 Add Component Imports

• In a stack configuration file, we can import other stack configuration files with import
• When a file is imported, the YAML from that file is deep merged on top of earlier imports. This is the same idea as merging two dictionaries together.
• Stack configurations can import each other as needed, and there can be multiple layers or different hierarchies of configurations

5 Deploy Components with a Stack

• In the directory corresponding to the environment you want to deploy in, for example stacks/orgs/acme/plat/sandbox/us-east-1/, add a new file (or adding to an existing file) your component by importing it from the catalog.

  import:
    # These two imports add default variables
    - orgs/acme/plat/sandbox/_defaults
    - mixins/region/us-east-1
    # This imports a real component, which will deploy even if we do not
    # inherit from it or override any values.
    - catalog/foobar

• In the above example, we have imported the foobar catalog configuration into the plat-use1-sandbox environment into a new YAML file of any name. For example foobar.yaml

  import:
    - orgs/acme/plat/sandbox/_defaults
    - mixins/region/us-east-1
    - catalog/foobar
  
  components:
    terraform:
      foobar:
        vars:
          sample_variable_present_in_variables_tf_of_component: "env-specific-config"

6 Deploy

Now that the component is (1) defined in Terraform, (2) created in Atmos, and (3) imported in the target Stack, now deploy the component with Atmos.

atmos terraform apply foobar -s plat-use1-sandbox

Vendoring

Atmos supports component vendoring. We use vendor to pull a specific version of the component from the upstream library.

When vendoring a component,

1. Create a branch of your repository
2. Add a component.yaml file to the components directory

   apiVersion: atmos/v1
   kind: ComponentVendorConfig
   spec:
     source:
       uri: github.com/cloudposse/terraform-aws-components.git//modules/foobar?ref={{ .Version }}
       version: 1.160.0
       included_paths:
         - "**/**"
       excluded_paths: []
3. Fill out the component.yaml with the latest version from the upstream library
4. Run the vendor commands: atmos vendor pull --component foobar
5. Create a Pull Request to check for changes against any existing component. Keep in mind vendoring will overwrite any custom changes to existing files upstream.

Next Steps

At this point, your component is complete in code, but there is still more to do!

1. Run precommit Terraform docs and linting against the new component
2. Add your new component to your GitOps automation tooling, such as Spacelift
3. Configure CODEOWNERS for the new component, if necessary
4. Documentation!

References

• Cloud Posse's Library of Terraform Components
• Cloud Posse's Library of Terraform Modules
• Atmos Core Concepts