Monitoring FAQ

How do I add a new monitor?

The easiest way to get started with an IaC monitor is to create it by hand in Datadog! While this may seem counterintuitive, seeing live graphs of your data and being able to view available metrics makes it much easier to figure out what is really important to look at.

Once you have a monitor configured in Datadog, you can export it to JSON, convert it to YAML, and then add it to your catalog of monitors.

Don't forget to replace hardcoded variables with Terraform variables or Datadog variables, as both interpolations will work.

What is Datadog Interpolation vs Terraform Interpolation?

When looking at a Datadog monitor, anything with ${foo} is Terraform interpolation; this will be substituted before being sent to Datadog. {{bar}} is Datadog interpolation. This will be used by datadog when the event comes in. This is useful for things like tags where you want to tag the event with the name of the cluster, but you don't know the name of the cluster when you create the monitor.

I'm not receiving the metrics I need, what do I do?

First off, we need to figure out where should the metrics be coming from.

If you are trying to receive metrics or logs from an EKS deployment or service, check if the datadog-agent is deployed and its logs look healthy.

If you are trying to receive metrics or logs from an ECS service, check the datadog agent sidecar container is deployed to your ecs-service.

If you are trying to receive metrics from an AWS Service, first check if the Datadog AWS Integration is deployed via datadog-integration and that the tile is working. Then check if the Datadog AWS Integration is enabled for the service you are trying to monitor. You can often find the metric is under the integration tiles' metrics tab. If it is enabled and working and you are not receiving metrics, check the integration role has the right permissions.

If you are trying to receive logs from an AWS Service, check the datadog-lambda-forwarder is deployed and working