Skip to main content
Sponsor @cloudposse

Setup Argo CD

This setup guide will walk you through the process of setting up Argo CD in your environment.

Requirements​

In order to deploy Argo CD, EKS must be fully deployed and functional. In particular, the user deploying the cluster must have a working VPN connection to the targeted account. See the EKS documentation for details.

All deployment steps below assume that the environment has been successfully set up with the following steps.

Authentication​

  1. Sign into AWS via Leapp
  2. Connect to the VPN
  3. Open Geodesic

Setup Steps​

1 Vendor Argo CD components​

First vendor all related components for the Argo CD layer:

vendor:
These are the commands included in the vendor workflow in the examples/snippets/stacks/workflows/argocd.yaml file:
    No commands found

Too many commands? Consider using the Atmos workflow! 🚀

2 Create Argo CD GitHub Repositories​

3 Prepare Authentication​

Argo CD can be integrated with GitHub using either GitHub Apps (recommended) or Personal Access Tokens (PATs). GitHub Apps provide more granular permissions, better security, and improved audit capabilities.

Argo CD requires several different types of GitHub authentication for various components and workflows. While these could be combined, we follow the principle of least privilege by creating separate authentication credentials for each specific purpose. The following authentication methods are required:

  1. Terraform argocd-repo Access​

    The first PAT is used by Terraform with the argocd-repo component. This is used to create and manage the Argo CD desired state repositories and (optionally) add deployment ssh keys for those repos.

    Terraform will pull that PAT from SSM typically using the argocd/github path in the core-auto account.

    Alternatively, you can enforce local access to apply the component. This would require an engineer to locally authenticate with GitHub and apply this component locally. Since this component is rarely updated, this can be a reasonable trade-off.

    At this time, we do not have GitHub App support for this step and prefer using local developer access.

  2. Terraform eks/argocd Access​

    The next two PATs or GitHub App installations are also used by Terraform now with the eks/argocd component. Each of these is used to register the webhook in GitHub for the Argo CD Application created with this given component. One is used for non-prod stages and the other is used for prod stages.

    Terraform will pull that token from SSM typically in plat-dev, plat-staging, and plat-prod accounts.

  3. Argo CD GitHub Notification Access​

    The next PAT or GitHub App token is used by the Argo CD notifications system to update the GitHub commit status on deployments. This is stored in SSM and pulled by the eks/argocd component. That component will pass the token to the Argo CD instance in the given EKS cluster. That Argo CD instance uses that token only when synchronous mode is enabled.

  4. GitHub Release Workflow Access​

    The final two PATs or GitHub App installations are used in the release engineering workflows: one for non-prod and one for prod. This token should allow write permissions in the given Argo CD desired state repositories.

1 Set up GitHub Apps​

Follow the instructions in Argo CD Integrations: How to set up Authorization for Argo CD with GitHub Apps to create and configure a GitHub App for Argo CD.

2 Configure Argo CD Desire State Repositories to Use GitHub Apps​

Update your Argo CD desired state repository configuration to use the GitHub App instead of PAT:

components:
  terraform:
    argocd-repo:
      vars:
        # 1. Use local access to apply this component rather than a PAT
        # https://registry.terraform.io/providers/integrations/github/latest/docs#github-cli
        use_local_github_credentials: true

        # 2. If synchronous mode is enabled, set the notifications to send to "github" and not to the "webhook"
        github_notifications:
          - "notifications.argoproj.io/subscribe.on-deploy-started.github: \"\""
          - "notifications.argoproj.io/subscribe.on-deploy-succeeded.github: \"\""
          - "notifications.argoproj.io/subscribe.on-deploy-failed.github: \"\""

        # 3. Optional, disable the SSH deploy keys to use a GitHub App 
        #    for the Argo CD instance to authenticate with the desired state repository
        deploy_keys_enabled: false

3 Configure Argo CD to Use GitHub Apps​

Update your Argo CD configuration to use the GitHub App instead of PATs:

components:
  terraform:
    eks/argocd:
      vars:
        # GitHub App (Argo CD Instance)
        # This GitHub App is used for the Argo CD instance to manage webhooks and read from the desired state repository.
        # ie https://github.com/acme/argocd-deploy-non-prod
        github_app_enabled: true
        github_app_id: "1234567"
        github_app_installation_id: "44444444"
        # The SSM parameter must exist in the account and region where Argo CD is deployed.
        ssm_github_app_private_key: "/argocd/argo_cd_instance/app_private_key"
        # Optional, disable the SSH deploy keys to use this GitHub App 
        # for the Argo CD instance to authenticate with the desired state repository
        github_deploy_keys_enabled: false

        # GitHub App (Argo CD Notifications)
        # This GitHub App is used for the Argo CD instance to send commit status updates back to each ap repository.
        # This is only required if synchronous mode is enabled.
        # ie https://github.com/acme/example-app-on-eks
        github_notifications_app_enabled: true
        github_notifications_app_id: "8901235"
        github_notifications_app_installation_id: "55555555"
        # The SSM parameter must exist in the account and region where Argo CD is deployed.
        ssm_github_notifications_app_private_key: "/argocd/argo_cd_notifications/app_private_key"

4 Deploy Argo CD repo configuration​

Deploy the Argo CD configuration for the two GitHub repos with the following workflow:

deploy/argocd-repos:
These are the commands included in the deploy/argocd-repos workflow in the examples/snippets/stacks/workflows/argocd.yaml file:
    No commands found

Too many commands? Consider using the Atmos workflow! 🚀

Once this finishes, review the two repos in your GitHub Organization. These should both be fully configured at this point.

5 Create AWS Identity Center Applications​

In order to authenticate with Argo CD, we recommend using an AWS IAM Identity Center SAML Application. These apps can use existing Identity Center groups that we've already setup as part of the Identity layer.

Please see Argo CD Integrations: How to create an AWS Identity Center Application and follow all steps.

6 Deploy Argo CD to each EKS Cluster​

Once the GitHub repositories are in place and the SAML applications have been created and configuration uploaded to SSM, we're ready to deploy Argo CD to each cluster.

Deploy eks/argocd to each cluster with the following workflow:

deploy/argocd:
These are the commands included in the deploy/argocd workflow in the examples/snippets/stacks/workflows/argocd.yaml file:
    No commands found

Too many commands? Consider using the Atmos workflow! 🚀

7 Validation​

Once all deployment steps are completed, Argo CD should be accessible at the following URLs. Please note that you must be able to authenticate with AWS Identity Center to access any given app.

Next Steps​

Assuming login goes well, here's a checklist of GitHub repos needed to connect Argo CD:

  • acme/infra-acme repo (Should already exist!)
    • acme/infra-acme/.github/environments private workflows. This directory stores private environment configurations. Primarily, that is the cloudposse/github-action-yaml-config-query action used to get role, namespace, and cluster mapping for each environment.
  • (2) Argo CD deploy nonprod and prod (Should already be created by argocd-repo component in earlier step)
    • argocd-deploy-non-prod
    • argocd-deploy-prod
  • acme/example-app repo should be private repo generated from the app-on-eks-with-argocd template
Sensitive Log Output

Note that all of these workflow runs run from within your private app repo, so any sensitive log output will not be public.

1 Environment Configuration​

Update the cloudposse/github-action-interface-environment action to point to your infrastructure repository.

  1. Set implementation_repository to acme/infra-acme
  2. Verify implementation_path, implementation_file, and implementation_ref match your local configuration.

Example app reference

2 Verify GitHub OIDC Access Roles​

Capitalization Matters!

The IDP permissions in IAM will be sensitive to capitalization, and yet the docker image must -not- have uppercase letters!

Make sure that your repo is allowed to assume roles for all relevant clusters and ECR repos:

  1. Update the github_actions_allowed_repos variable in ecr, eks/cluster, or any other relevant components with GitHub OIDC access.
  2. If your GitHub Organization has mixed capitalization cases, make sure these entries are case-sensitive

3 GitHub Environment Secrets​

Add each of the following secrets to the acme/example-app repo:

  1. github-private-actions-pat: ${{ secrets.PUBLIC_REPO_ACCESS_TOKEN }}
  2. registry: ${{ secrets.ECR_REGISTRY }}
  3. secret-outputs-passphrase: ${{ secrets.GHA_SECRET_OUTPUT_PASSPHRASE }}
  4. ecr-region: ${{ secrets.ECR_REGION }}
  5. ecr-iam-role: ${{ secrets.ECR_IAM_ROLE }}

4 Specify Ingress Group​

  1. Update the deploy/releases/app.yaml
  2. Make sure the ingress is not set to default. It should likely be alb-controller-ingress-group. you can read more about this from our docs on the alb controller component
  3. Set the domain accordingly. Each environment will need the service domain + environment.stage.tenant (ie. use2.staging.plat.acme-svc.com )
  4. If your organization has mixed case, you'll need to edit the organization parameter to be lowercased in the GitHub workflows: feature-branch.yml, main-branch.yaml, and release.yaml

FAQ​

Migrating from Argo CD version <1.305.0 to 1.305.0​

In order to migrate to 1.305.0 from a lesser version, delete the previous SSM parameters in each account for /argocd/notifications/notifiers/service_webhook_github-commit-status/github-token and redeploy github-webhook-pat, eks/argocd, and argocd-repos.

Or use the following workflow:

migrate_less_1_305:
These are the commands included in the migrate_less_1_305 workflow in the examples/snippets/stacks/workflows/argocd.yaml file:
    No commands found

Too many commands? Consider using the Atmos workflow! 🚀

GitHub Apps vs Personal Access Tokens​

We recommend using GitHub Apps for Argo CD integration with GitHub. GitHub Apps offer several advantages over Personal Access Tokens:

  1. Granular Permissions: GitHub Apps can be granted access to specific repositories rather than requiring organization-wide access.

  2. Better Security: GitHub Apps use JWT authentication and short-lived tokens, reducing the risk of token exposure.

  3. Improved Audit Capabilities: Actions performed by GitHub Apps are clearly identified in audit logs.

  4. Rate Limiting: GitHub Apps have their own rate limits, separate from user-based limits.

  5. Webhook Support: GitHub Apps can receive webhooks for events in repositories they have access to.

  6. Multiple Installations: The same GitHub App can be installed on different repositories with different permissions.

For more information on setting up Argo CD with GitHub Apps, see Argo CD Integrations: How to set up Authorization for Argo CD with GitHub Apps.