Component: `spacelift`

This component is responsible for provisioning Spacelift stacks.

Spacelift is a specialized, Terraform-compatible continuous integration and deployment (CI/CD) platform for infrastructure-as-code. It's designed and implemented by long-time DevOps practitioners based on previous experience with large-scale installations - dozens of teams, hundreds of engineers and tens of thousands of cloud resources.

Usage

Stack Level: Regional

This component provisions an administrative Spacelift stack and assigns it to a worker pool. Although the stack can manage stacks in any region, it should be provisioned in the same region as the worker pool.

components:
  terraform:
    spacelift/defaults:
      metadata:
        type: abstract
        component: spacelift
      settings:
        spacelift:
          workspace_enabled: true
          administrative: true
          autodeploy: true
          before_init:
            - spacelift-configure
            - spacelift-write-vars
            - spacelift-tf-workspace
          before_plan:
            - spacelift-configure
          before_apply:
            - spacelift-configure
          component_root: components/terraform/spacelift
          description: Spacelift Administrative stack
          stack_destructor_enabled: false
          # TODO: replace with the name of the worker pool
          worker_pool_name: WORKER_POOL_NAME
          repository: infra
          branch: main
          labels:
            - folder:admin
          # Do not add normal set of child policies to admin stacks
          policies_enabled: []
          policies_by_id_enabled: []
      vars:
        enabled: true
        spacelift_api_endpoint: https://TODO.app.spacelift.io
        administrative_stack_drift_detection_enabled: true
        administrative_stack_drift_detection_reconcile: true
        administrative_stack_drift_detection_schedule: ["0 4 * * *"]
        administrative_trigger_policy_enabled: false
        autodeploy: false
        aws_role_enabled: false
        drift_detection_enabled: true
        drift_detection_reconcile: true
        drift_detection_schedule: ["0 4 * * *"]
        external_execution: true
        git_repository: infra # TODO: replace with your repository name
        git_branch: main

        # List of available default Rego policies to create in Spacelift.
        # These policies are defined in the catalog https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/tree/master/catalog/policies
        # These policies will not be attached to Spacelift stacks by default (but will be created in Spacelift, and could be attached to a stack manually).
        # For specify policies to attach to each Spacelift stack, use `var.policies_enabled`.
        policies_available:
          - "git_push.proposed-run"
          - "git_push.tracked-run"
          - "plan.default"
          - "trigger.dependencies"
          - "trigger.retries"

        # List of default Rego policies to attach to all Spacelift stacks.
        # These policies are defined in the catalog https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/tree/master/catalog/policies
        policies_enabled:
          - "git_push.proposed-run"
          - "git_push.tracked-run"
          - "plan.default"
          - "trigger.dependencies"

        # List of custom policy names to attach to all Spacelift stacks
        # These policies must exist in `components/terraform/spacelift/rego-policies`
        policies_by_name_enabled: []

        runner_image: 000000000000.dkr.ecr.us-west-2.amazonaws.com/infra #TODO: replace with your ECR repository
        spacelift_component_path: components/terraform
        stack_config_path_template: stacks/%s.yaml
        stack_destructor_enabled: false
        worker_pool_name_id_map:
          <core-region-auto>-spacelift-worker-pool: SOMEWORKERPOOLID #TODO: replace with your worker pool ID
        infracost_enabled: false # TODO: decide on infracost
        terraform_version: "1.3.6"
        terraform_version_map:
          "1": "1.3.6"

        # These could be moved to $PROJECT_ROOT/.spacelift/config.yml
        before_init:
          - spacelift-configure
          - spacelift-write-vars
          - spacelift-tf-workspace
        before_plan:
          - spacelift-configure
        before_apply:
          - spacelift-configure

    # Manages policies, admin stacks, and core OU accounts
    spacelift:
      metadata:
        component: spacelift
        inherits:
          - spacelift/defaults
      settings:
        spacelift:
          policies_by_id_enabled:
            # This component also creates this policy so this is omitted prior to the first apply
            # then added so it's consistent with all admin stacks.
            - trigger-administrative-policy
      vars:
        enabled: true
        # Use context_filters to split up admin stack management
        # context_filters:
        #   stages:
        #     - artifacts
        #     - audit
        #     - auto
        #     - corp
        #     - dns
        #     - identity
        #     - marketplace
        #     - network
        #     - public
        #     - security
        # These are the policies created from https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/tree/master/catalog/policies
        # Make sure to remove the .rego suffix
        policies_available:
          - git_push.proposed-run
          - git_push.tracked-run
          - plan.default
          - trigger.dependencies
          - trigger.retries
          # This is to auto deploy launch template image id changes
          - plan.warn-on-resource-changes-except-image-id
          # This is the global admin policy
          - trigger.administrative
        # These are the policies added to each spacelift stack created by this admin stack
        policies_enabled:
          - git_push.proposed-run
          - git_push.tracked-run
          - plan.default
          - trigger.dependencies
        # Keep these empty
        policies_by_id_enabled: []

Prerequisites

GitHub Integration

The GitHub owner will need to sign up for a free trial of Spacelift
Once an account is created take note of the URL - usually its https://<GITHUBORG>.app.spacelift.io/

Create a Login Policy

Click on Policies then Add Policy
Use the following policy and replace GITHUBORG with the GitHub Organization slug and DEV with the GitHub id for the Dev setting up the Spacelift module.

package spacelift

# See https://docs.spacelift.io/concepts/policy/login-policy for implementation details.
# Note: Login policies don't affect GitHub organization or SSO admins.
# Note 2: Enabling SSO requires that all users have an IdP (G Suite) account, so we'll just use
#          GitHub authentication in the meantime while working with external collaborators.
# Map session input data to human friendly variables to use in policy evaluation

username    := input.session.login
member_of   := input.session.teams # Input is friendly name, e.g. "SRE" not "sre" or "@GITHUBORG/sre"
GITHUBORG   := input.session.member # Is this user a member of the CUSTOMER GitHub org?

# Define GitHub usernames of non org external collaborators with admin vs. user access
admin_collaborators := { "DEV" }
user_collaborators  := { "GITHUBORG" } # Using GITHUBORG as a placeholder to avoid empty set

# Grant admin access to GITHUBORG org members in the CloudPosse group
admin {
  GITHUBORG
  member_of[_] == "CloudPosse"
}

# Grant admin access to non-GITHUBORG org accounts in the admin_collaborators set
admin {
  # not GITHUBORG
  admin_collaborators[username]
}

# Grant user access to GITHUBORG org members in the Developers group
# allow {
#   GITHUBORG
#   member_of[_] == "Developers"
# }

# Grant user access to non-GITHUBORG org accounts in the user_collaborators set
allow {
  not GITHUBORG
  user_collaborators[username]
}

# Deny access to any non-GITHUBORG org accounts who aren't defined in external collaborators sets
deny {
  not GITHUBORG
  not user_collaborators[username]
  not admin_collaborators[username]
}

Spacelift Layout

Runtime configuration is a piece of setup that is applied to individual runs instead of being global to the stack. It's defined in .spacelift/config.yml YAML file at the root of your repository. It is required for Spacelift to work with atmos.

Create Spacelift helper scripts

/rootfs/usr/local/bin/spacelift-tf-workspace manages selecting or creating a Terraform workspace; similar to how atmos manages workspaces during a Terraform run.

/rootfs/usr/local/bin/spacelift-write-vars writes the component config using atmos to the spacelift.auto.tfvars.json file.

NOTE: make sure they are all executable:

chmod +x rootfs/usr/local/bin/spacelift*

Bootstrapping

After creating & linking Spacelift to this repo (see the docs), follow these steps...

Deploy the `spacelift-worker-pool` Component

See spacelift-worker-pool README for the configuration and deployment needs.

Update the `spacelift` catalog

git_repository = Name of infrastructure repository
git_branch = Name of main/master branch
worker_pool_name_id_map = Map of arbitrary names to IDs Spacelift worker pools, taken from the worker_pool_id output of the spacelift-worker-pool component.
Set components.terraform.spacelift.settings.spacelift.worker_pool_name to the name of the worker pool you want to use for the spacelift component, the name being the key you set in the worker_pool_name_id_map map.

Deploy the admin stacks

Set these ENV vars:

export SPACELIFT_API_KEY_ENDPOINT=https://<GITHUB_ORG>.app.spacelift.io
export SPACELIFT_API_KEY_ID=...
export SPACELIFT_API_KEY_SECRET=...

The name of the spacelift stack resource will be different depending on the name of the component and the root atmos stack. This would be the command if the root atmos stack is core-gbl-auto and the spacelift component is spacelift.

atmos terraform apply spacelift --stack core-gbl-auto -target 'module.spacelift.module.stacks["core-gbl-auto-spacelift"]'

Note that this is the only manually operation you need to perform in geodesic using atmos to create the initial admin stack. All other infrastructure stacks wil be created in Spacelift by this admin stack.

Pull Request Workflow

Create a new branch & make changes
Create a new pull request (targeting the main branch)
View the modified resources directly in the pull request
View the successful Spacelift checks in the pull request
Merge the pull request and check the Spacelift job

spacectl

See docs https://github.com/spaceone-dev/spacectl

Install

⨠ apt install -y spacectl -qq

Setup a profile

⨠ spacectl profile login gbl-identity
Enter Spacelift endpoint (eg. https://unicorn.app.spacelift.io/): https://<GITHUB_ORG>.app.spacelift.io
Select credentials type: 1 for API key, 2 for GitHub access token: 1
Enter API key ID: 01FKN...
Enter API key secret:

Listing stacks

spacectl stack list

Grab all the stack ids (use the JSON output to avoid bad chars)

spacectl stack list --output json | jq -r '.[].id' > stacks.txt

If the latest commit for each stack is desired, run something like this.

NOTE: remove the echo to remove the dry-run functionality

cat stacks.txt | while read stack; do echo $stack && echo spacectl stack set-current-commit --sha 25dd359749cfe30c76cce19f58e0a33555256afd --id $stack; done

Requirements

Name	Version
terraform	>= 1.3
aws	>= 4.0
spacelift	>= 0.1.31

Providers

Name	Version
aws	>= 4.0

Modules

Name	Source	Version
iam_roles	../account-map/modules/iam-roles	n/a
spacelift	cloudposse/cloud-infrastructure-automation/spacelift	0.55.0
this	cloudposse/label/null	0.25.0

Resources

Name	Type
aws_ssm_parameter.spacelift_key_id	data source
aws_ssm_parameter.spacelift_key_secret	data source

Inputs

Name	Description	Type	Default	Required
additional_tag_map	Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`. This is for some rare cases where resources want additional configuration of tags and therefore take a list of maps with tag key, value, and additional configuration.	`map(string)`	`{}`	no
administrative_push_policy_enabled	Flag to enable/disable the global administrative push policy	`bool`	`true`	no
administrative_stack_drift_detection_enabled	Flag to enable/disable administrative stack drift detection	`bool`	`true`	no
administrative_stack_drift_detection_reconcile	Flag to enable/disable administrative stack drift automatic reconciliation. If drift is detected and `reconcile` is turned on, Spacelift will create a tracked run to correct the drift	`bool`	`true`	no
administrative_stack_drift_detection_schedule	List of cron expressions to schedule drift detection for the administrative stack	`list(string)`	`[ "0 4 * * *" ]`	no
administrative_trigger_policy_enabled	Flag to enable/disable the global administrative trigger policy	`bool`	`true`	no
attachment_space_id	Specify the space ID for attachments (e.g. policies, contexts, etc.)	`string`	`"legacy"`	no
attributes	ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`, in the order they appear in the list. New attributes are appended to the end of the list. The elements of the list are joined by the `delimiter` and treated as a single ID element.	`list(string)`	`[]`	no
autodeploy	Default autodeploy value for all stacks created by this project	`bool`	n/a	yes
aws_role_arn	ARN of the AWS IAM role to assume and put its temporary credentials in the runtime environment	`string`	`null`	no
aws_role_enabled	Flag to enable/disable Spacelift to use AWS STS to assume the supplied IAM role and put its temporary credentials in the runtime environment	`bool`	`false`	no
aws_role_external_id	Custom external ID (works only for private workers). See https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html for more details	`string`	`null`	no
aws_role_generate_credentials_in_worker	Flag to enable/disable generating AWS credentials in the private worker after assuming the supplied IAM role	`bool`	`false`	no
before_init	List of before-init scripts	`list(string)`	`[]`	no
context	Single object for setting entire context at once. See description of individual variables for details. Leave string and numeric variables as `null` to use default value. Individual variable settings (non-null) override settings in context object, except for attributes, tags, and additional_tag_map, which are merged.	`any`	`{ "additional_tag_map": {}, "attributes": [], "delimiter": null, "descriptor_formats": {}, "enabled": true, "environment": null, "id_length_limit": null, "label_key_case": null, "label_order": [], "label_value_case": null, "labels_as_tags": [ "unset" ], "name": null, "namespace": null, "regex_replace_chars": null, "stage": null, "tags": {}, "tenant": null }`	no
context_filters	Context filters to create stacks for specific context information. Valid lists are `namespaces`, `environments`, `tenants`, `stages`.	`map(list(string))`	`{}`	no
delimiter	Delimiter to be used between ID elements. Defaults to `-` (hyphen). Set to `""` to use no delimiter at all.	`string`	`null`	no
descriptor_formats	Describe additional descriptors to be output in the `descriptors` output map. Map of maps. Keys are names of descriptors. Values are maps of the form `{<br/> format = string<br/> labels = list(string)<br/>}` (Type is `any` so the map values can later be enhanced to provide additional options.) `format` is a Terraform format string to be passed to the `format()` function. `labels` is a list of labels, in order, to pass to `format()` function. Label values will be normalized before being passed to `format()` so they will be identical to how they appear in `id`. Default is `{}` (`descriptors` output will be empty).	`any`	`{}`	no
drift_detection_enabled	Flag to enable/disable drift detection on the infrastructure stacks	`bool`	`true`	no
drift_detection_reconcile	Flag to enable/disable infrastructure stacks drift automatic reconciliation. If drift is detected and `reconcile` is turned on, Spacelift will create a tracked run to correct the drift	`bool`	`true`	no
drift_detection_schedule	List of cron expressions to schedule drift detection for the infrastructure stacks	`list(string)`	`[ "0 4 * * *" ]`	no
enabled	Set to false to prevent the module from creating any resources	`bool`	`null`	no
environment	ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'	`string`	`null`	no
external_execution	Set this to true if you're calling this module from outside of a Spacelift stack (e.g. the `complete` example)	`bool`	`false`	no
git_branch	The Git branch name	`string`	`"main"`	no
git_commit_sha	The commit SHA for which to trigger a run. Requires `var.spacelift_run_enabled` to be set to `true`	`string`	`null`	no
git_repository	The Git repository name	`string`	n/a	yes
id_length_limit	Limit `id` to this many characters (minimum 6). Set to `0` for unlimited length. Set to `null` for keep the existing setting, which defaults to `0`. Does not affect `id_full`.	`number`	`null`	no
import_profile_name	AWS Profile name to use when importing a resource	`string`	`null`	no
import_role_arn	IAM Role ARN to use when importing a resource	`string`	`null`	no
infracost_enabled	Flag to enable/disable infracost. If this is enabled, it will add infracost label to each stack. See spacelift infracost docs for more details.	`bool`	`false`	no
label_key_case	Controls the letter case of the `tags` keys (label names) for tags generated by this module. Does not affect keys of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper`. Default value: `title`.	`string`	`null`	no
label_order	The order in which the labels (ID elements) appear in the `id`. Defaults to ["namespace", "environment", "stage", "name", "attributes"]. You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.	`list(string)`	`null`	no
label_value_case	Controls the letter case of ID elements (labels) as included in `id`, set as tag values, and output by this module individually. Does not affect values of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper` and `none` (no transformation). Set this to `title` and set `delimiter` to `""` to yield Pascal Case IDs. Default value: `lower`.	`string`	`null`	no
labels_as_tags	Set of labels (ID elements) to include as tags in the `tags` output. Default is to include all labels. Tags with empty values will not be included in the `tags` output. Set to `[]` to suppress all generated tags. Notes: The value of the `name` tag, if included, will be the `id`, not the `name`. Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be changed in later chained modules. Attempts to change it will be silently ignored.	`set(string)`	`[ "default" ]`	no
name	ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'. This is the only ID element not also included as a `tag`. The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input.	`string`	`null`	no
namespace	ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique	`string`	`null`	no
policies_available	List of available default policies to create in Spacelift (these policies will not be attached to Spacelift stacks by default, use `var.policies_enabled`)	`list(string)`	`[ "git_push.proposed-run", "git_push.tracked-run", "plan.default", "trigger.dependencies", "trigger.retries" ]`	no
policies_by_id_enabled	List of existing policy IDs to attach to all Spacelift stacks. These policies must already exist in Spacelift	`list(string)`	`[]`	no
policies_by_name_enabled	List of existing policy names to attach to all Spacelift stacks. These policies must exist at `modules/spacelift/rego-policies` OR `var.policies_by_name_path`.	`list(string)`	`[]`	no
policies_by_name_path	Path to the catalog of external Rego policies. The Rego files must exist in the caller's code at the path. The module will create Spacelift policies from the external Rego definitions	`string`	`""`	no
policies_enabled	DEPRECATED: Use `policies_by_id_enabled` instead. List of default policies created by this stack to attach to all Spacelift stacks	`list(string)`	`[]`	no
regex_replace_chars	Terraform regular expression (regex) string. Characters matching the regex will be removed from the ID elements. If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits.	`string`	`null`	no
region	AWS Region	`string`	n/a	yes
runner_image	Full address & tag of the Spacelift runner image (e.g. on ECR)	`string`	n/a	yes
spacelift_api_endpoint	The Spacelift API endpoint URL (e.g. https://example.app.spacelift.io)	`string`	n/a	yes
spacelift_component_path	The Spacelift Component Path	`string`	`"components/terraform"`	no
spacelift_run_enabled	Enable/disable creation of the `spacelift_run` resource	`bool`	`false`	no
spacelift_stack_dependency_enabled	If enabled, the `spacelift_stack_dependency` Spacelift resource will be used to create dependencies between stacks instead of using the `depends-on` labels. The `depends-on` labels will be removed from the stacks and the trigger policies for dependencies will be detached	`bool`	`false`	no
stack_config_path_template	Stack config path template	`string`	`"stacks/%s.yaml"`	no
stack_destructor_enabled	Flag to enable/disable the stack destructor to destroy the resources of a stack before deleting the stack itself	`bool`	`false`	no
stacks_space_id	Override the space ID for all stacks (unless the stack config has `dedicated_space` set to true). Otherwise, it will default to the admin stack's space.	`string`	`null`	no
stage	ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'	`string`	`null`	no
tag_filters	A map of tags that will filter stack creation by the matching `tags` set in a component `vars` configuration.	`map(string)`	`{}`	no
tags	Additional tags (e.g. `{'BusinessUnit': 'XYZ'}`). Neither the tag keys nor the tag values will be modified by this module.	`map(string)`	`{}`	no
tenant	ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for	`string`	`null`	no
terraform_version	Default Terraform version for all stacks created by this project	`string`	n/a	yes
terraform_version_map	A map to determine which Terraform patch version to use for each minor version	`map(string)`	`{}`	no
worker_pool_name_id_map	Map of worker pool names to worker pool IDs	`map(any)`	`{}`	no

Outputs

Name	Description
stacks	Spacelift stacks

References

cloudposse/terraform-aws-components - Cloud Posse's upstream component

Component: spacelift

Usage​

Prerequisites​

GitHub Integration​

Spacelift Layout​

Create Spacelift helper scripts​

Bootstrapping​

Deploy the spacelift-worker-pool Component​

Update the spacelift catalog​

Deploy the admin stacks​

Pull Request Workflow​

spacectl​

Install​

Listing stacks​

Requirements​

Providers​

Modules​

Resources​

Inputs​

Outputs​

References​