How to rollback Terraform state for a component

Problem

If Terraform state for a component gets corrupted or lost for some reason, it’s possible to rollback the terraform state to a previous version since we use S3 bucket versioning on all state objects.

Solution

Components represent terraform “root” modules and their Terraform state is stored in an s3 state bucket. Rolling back to that state requires a series of commands to be executed under the appropriate IAM role.

Assume the root-admin IAM role to be able to access s3 buckets in the root account. From geodesic, run the following command (replace <namespace> with your company namespace):

assume-role <namespace>-gbl-root-admin bash -l

Run the command aws s3 ls to list all the buckets in the root account. You should see a state bucket with a name similar to <namespace>-<environment>-root-tfstate , where <namespace> is your company namespace and <environment> is the region abbreviation (e.g. uw2 or ue2).
Run the following command to list the versions of the state file for a component:

aws s3api list-object-versions --bucket <BucketName> --prefix <BucketPrefix> --max-items 10

where <BucketName> is the name of the bucket in the format <namespace>-<environment>-root-tfstate <BucketPrefix> is the path to the state file for the component and is usually in the format <component>/<environment>-<stage>/terraform.tfstate

You should get a list of state file versions for the component similar to this:

{
    "Versions": [
        {
            "ETag": "\"af821255ecd29b03f5ef538ffc9ded4b\"",
            "Size": 180384,
            "StorageClass": "STANDARD",
            "Key": "<component>/<environment>-<stage>/terraform.tfstate",
            "VersionId": "j1hisFJ91fUxQM2N1FyMmNtsMMBJE5NP",
            "IsLatest": true,
            "LastModified": "2021-04-30T18:23:24+00:00"
        },
        {
            "ETag": "\"9053b298ae1aea5e79080d3eba067c4d\"",
            "Size": 172389,
            "StorageClass": "STANDARD",
            "Key": "<component>/<environment>-<stage>/terraform.tfstate",
            "VersionId": "ff9uG.N8Iwoe97Bfo1eOD7Ki8p7N1snX",
            "IsLatest": false,
            "LastModified": "2021-04-22T15:26:19+00:00",
        }
    ],
}

Choose the "VersionId" you want to restore to.

Restore the state file to the selected previous version. You do this by using the copy-object command. You must copy the previous version of the object into the same bucket, using the same object key. For more information, see copy-object in the AWS CLI Command Reference.

Run the following command to copy the previous version of the state file into the same bucket:

aws s3api copy-object --copy-source "<BucketName>/<BucketPrefix>?versionId=<VersionId>" --key <BucketPrefix> --bucket <BucketName> --server-side-encryption aws:kms

where <VersionId> is the ID of the previous version that you want to restore to (you can get it from the list returned from the aws s3api list-object-versions command)

In a separate geodesic run atmos terraform plan for the stack you were recovering. You will be prompted with an error:

update the Digest value stored in the DynamoDB table to the following value: <DIGEST>

Copy that digest value.

From the <namespace>-gbl-root-admin shell, update DynamoDB digest to the one you were prompted with

 aws dynamodb update-item --table-name <namespace>-<environment>-<stage>-tfstate-lock --key '{"LockID": {"S": "<BucketName>/<ComponentName>/<environment>-<stage>/terraform.tfstate-md5"}}' --attribute-updates '{"Digest": {"Value": {"S": "<DIGEST>"},"Action": "PUT"}}' --return-values UPDATED_NEW | jq '.Attributes.RuleSetVersion.S'

https://docs.aws.amazon.com/AmazonS3/latest/userguide/RestoringPreviousVersions.html

https://docs.aws.amazon.com/AmazonS3/latest/userguide/RetrievingObjectVersions.html

https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

Problem​

Solution​

Problem

Solution