Skip to main content
Sponsor @cloudposse

Use SSM over ASM for Infrastructure

Latest!

The content in this ADR is up-to-date! For questions, please reach out to Cloud Posse

We primarily provision static credentials randomly generated by Terraform using the database provider and then write them to SSM, encrypted with KMS.

Amazon Secrets Manager (ASM) was launched well after Amazon Systems Manager (formerly AWS Simple Systems Manager, hence SSM) Parameter Store. Both fulfill similar use-cases, except for Parameter Store was create more generally as a Key/Value store for parameters that can also be encrypted with KMS/CMK keys. ASM on the other hand was purpose-built to manage secrets and has the concept of automatic rotation using Lambdas.

Security best practices call for secrets to be rotated (changed) on a regular basis so that if a secret is improperly obtained by an adversary, the length of time the adversary can use that secret is limited. However, if everyone sharing the secret has to agree on what the secret is all the time, changing the secret presents a synchronization problem: all parties need to be updated with the new secret at exactly the same time. Parameter store only solves part of this problem: by having a single source of truth accessed by all parties, changing the secret once in parameter store makes the new secret available at the same time to all, but there is no mechanism to inform the parties of the change or to synchronize their adoption of the new secret.

ASM was built to solve this synchronization problem. It allows you to store and retrieve several versions of a secret, with one being designated the “current” one and one being designated the “previous” one. To take advantage of this, servers (whatever is requiring the secret to be presented for authentication) must allow clients (whatever is presenting secrets as authentication) to present either the current or previous secret. When this is done, the synchronization problem is solved by executing the following steps in order

  1. Confirm all clients and servers are using the “current” secret

  2. “Rotate” the secret by changing the label of the current secret from “current” to “previous” and creating a new “current” secret. AWS provides a Lambda function that implements this.

  3. Update all servers to accept either secret. This allows old clients who only know about the now “previous” secret to continue to access the servers, while allowing new clients to pick up and use the new “current” secret. Servers should be designed to be able to be updated in this way while running, without causing a service interruption.

  4. Have all clients pick up and start using the new “current” secret when it is convenient for them.

By maintaining 2 active secrets, clients can be designed to refresh their secrets when convenient, without significant time pressure. There is no need for special notification or synchronization features to be built into the clients. However, server-side support is critical: If the servers do not support simultaneous use of 2 different secrets for the same purpose, there is no practical benefit to making the “previous” secret available. One key reason to use ASM is that some Amazon services, such as RDS, have built-in integration with ASM to provide this simultaneous support.

After ASM was built, SSM was enhanced to provide similar capabilities, although without as streamlined an API or a Lambda to do secrets rotation. SSM can store up to 100 versions of a parameter, and the versions can be given symbolic tags, such as “current” and “previous”. So if you are building your own servers, you can implement the same kind of secrets rotation strategy with SSM as with ASM.

ASM​

Pros​

Cons​

  • Only consumer applications that support dynamic credentials can take advantage of this functionality

  • Most systems do not support the complex pattern of key rotation; namely, concentric keys need some period of overlapping validity. Therefore, in practice it’s used mostly just for RDMS systems.

  • Still need to deploy the Lambda functions, which means all the CD/CD machinery to support lambdas (workflows, builds, integration tests, artifacts, deployments, etc)

  • No way to aggregate all secrets with a prefix

  • No built-in audit trail metadata (but it writes CloudTrail like any other AWS API).

Other​

SSM​

Pros​

  • Very simple to operate with (true to it’s original name)

  • Can easily aggregate all key/values with a given prefix

  • Encrypted with KMS

  • Multiple Kubernetes Operators exist to synchronize SSM Parameters to Kubernetes Secrets

  • Built-in audit trail for every parameter in addition to CloudTrail

Cons​


Other​

  • Historical rate limits for SSM were very low, now they are up to 3,000.