Module: `emr-cluster`

Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS.

Usage

For a complete example, see examples/complete

For automated tests of the complete example using bats and Terratest (which tests and deploys the example on AWS), see test.

provider "aws" {
  region = "us-east-2"
}

module "vpc" {
  source  = "cloudposse/vpc/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  ipv4_primary_cidr_block = "172.19.0.0/16"

  context = module.this.context
}

module "subnets" {
  source = "cloudposse/dynamic-subnets/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  availability_zones   = var.availability_zones
  vpc_id               = module.vpc.vpc_id
  igw_id               = [module.vpc.igw_id]
  ipv4_cidr_block      = [module.vpc.vpc_cidr_block]
  nat_gateway_enabled  = false
  nat_instance_enabled = false

  context = module.this.context
}

module "s3_log_storage" {
  source = "cloudposse/s3-log-storage/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  region        = var.region
  namespace     = var.namespace
  stage         = var.stage
  name          = var.name
  attributes    = ["logs"]
  force_destroy = true
}

module "aws_key_pair" {
  source = "cloudposse/key-pair/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  namespace           = var.namespace
  stage               = var.stage
  name                = var.name
  attributes          = ["ssh", "key"]
  ssh_public_key_path = var.ssh_public_key_path
  generate_ssh_key    = var.generate_ssh_key
}

module "emr_cluster" {
  source = "cloudposse/emr-cluster/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  namespace                                      = var.namespace
  stage                                          = var.stage
  name                                           = var.name
  master_allowed_security_groups                 = [module.vpc.vpc_default_security_group_id]
  slave_allowed_security_groups                  = [module.vpc.vpc_default_security_group_id]
  region                                         = var.region
  vpc_id                                         = module.vpc.vpc_id
  subnet_id                                      = module.subnets.private_subnet_ids[0]
  route_table_id                                 = module.subnets.private_route_table_ids[0]
  subnet_type                                    = "private"
  ebs_root_volume_size                           = var.ebs_root_volume_size
  visible_to_all_users                           = var.visible_to_all_users
  release_label                                  = var.release_label
  applications                                   = var.applications
  configurations_json                            = var.configurations_json
  core_instance_group_instance_type              = var.core_instance_group_instance_type
  core_instance_group_instance_count             = var.core_instance_group_instance_count
  core_instance_group_ebs_size                   = var.core_instance_group_ebs_size
  core_instance_group_ebs_type                   = var.core_instance_group_ebs_type
  core_instance_group_ebs_volumes_per_instance   = var.core_instance_group_ebs_volumes_per_instance
  master_instance_group_instance_type            = var.master_instance_group_instance_type
  master_instance_group_instance_count           = var.master_instance_group_instance_count
  master_instance_group_ebs_size                 = var.master_instance_group_ebs_size
  master_instance_group_ebs_type                 = var.master_instance_group_ebs_type
  master_instance_group_ebs_volumes_per_instance = var.master_instance_group_ebs_volumes_per_instance
  create_task_instance_group                     = var.create_task_instance_group
  log_uri                                        = format("s3n://%s/", module.s3_log_storage.bucket_id)
  key_name                                       = module.aws_key_pair.key_name
}

Variables

Required Variables

applications (list(string)) required: A list of applications for the cluster. Valid values are: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper (as of EMR 5.25.0). Case insensitive
core_instance_group_ebs_size (number) required: Core instances volume size, in gibibytes (GiB)
core_instance_group_instance_type (string) required: EC2 instance type for all instances in the Core instance group
master_instance_group_ebs_size (number) required: Master instances volume size, in gibibytes (GiB)
master_instance_group_instance_type (string) required: EC2 instance type for all instances in the Master instance group
region (string) required: AWS region
subnet_id (string) required: VPC subnet ID where you want the job flow to launch. Cannot specify the cc1.4xlarge instance type for nodes of a job flow launched in a Amazon VPC
vpc_id (string) required: VPC ID to create the cluster in (e.g. vpc-a22222ee)

Optional Variables

additional_info (string) optional

A JSON string for selecting additional features such as adding proxy information. Note: Currently there is no API to retrieve the value of this argument after EMR cluster creation from provider, therefore Terraform cannot detect drift from the actual EMR cluster if its value is changed outside Terraform

Default value: null

additional_master_security_group (string) optional

The id of the existing additional security group that will be used for EMR master node. If empty, a new security group will be created

Default value: ""

additional_slave_security_group (string) optional

The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created

Default value: ""

auto_termination_idle_timeout (string) optional

Auto termination policy idle timeout in seconds (60 - 604800 supported)

Default value: null

bootstrap_action optional

List of bootstrap actions that will be run before Hadoop is started on the cluster nodes

Type:

list(object({
path = string
name = string
args = list(string)
}))

Default value: [ ]

configurations_json (string) optional

A JSON string for supplying list of configurations for the EMR cluster. See https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html for more details

Default value: ""

core_instance_group_autoscaling_policy (string) optional

String containing the EMR Auto Scaling Policy JSON for the Core instance group

Default value: null

core_instance_group_bid_price (string) optional

Bid price for each EC2 instance in the Core instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances

Default value: null

core_instance_group_ebs_iops (number) optional

The number of I/O operations per second (IOPS) that the Core volume supports

Default value: null

core_instance_group_ebs_type (string) optional

Core instances volume type. Valid options are gp2, io1, standard and st1

Default value: "gp2"

core_instance_group_ebs_volumes_per_instance (number) optional

The number of EBS volumes with this configuration to attach to each EC2 instance in the Core instance group

Default value: 1

core_instance_group_instance_count (number) optional

Target number of instances for the Core instance group. Must be at least 1

Default value: 1

create_task_instance_group (bool) optional

Whether to create an instance group for Task nodes. For more info: https://www.terraform.io/docs/providers/aws/r/emr_instance_group.html, https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html

Default value: false

create_vpc_endpoint_s3 (bool) optional

Set to false to prevent the module from creating VPC S3 Endpoint

Default value: true

custom_ami_id (string) optional

A custom Amazon Linux AMI for the cluster (instead of an EMR-owned AMI). Available in Amazon EMR version 5.7.0 and later

Default value: null

ebs_root_volume_size (number) optional

Size in GiB of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later

Default value: 10

ec2_autoscaling_role_enabled (bool) optional

If set to false, will use existing_ec2_autoscaling_role_arn for an existing EC2 autoscaling IAM role that was created outside of this module

Default value: true

ec2_autoscaling_role_permissions_boundary (string) optional

The Permissions Boundary ARN to apply to the EC2 Autoscaling Role.

Default value: ""

ec2_role_enabled (bool) optional

If set to false, will use existing_ec2_instance_profile_arn for an existing EC2 IAM role that was created outside of this module

Default value: true

ec2_role_permissions_boundary (string) optional

The Permissions Boundary ARN to apply to the EC2 Role.

Default value: ""

emr_role_permissions_boundary (string) optional

The Permissions Boundary ARN to apply to the EMR Role.

Default value: ""

enable_ssm_access (bool) optional

If set to true, attach the existing AmazonSSMManagedInstanceCore IAM policy to the EMR EC2 instance profile role

Default value: false

existing_ec2_autoscaling_role_arn (string) optional

ARN of an existing EC2 autoscaling role to attach to the cluster

Default value: ""

existing_ec2_instance_profile_arn (string) optional

ARN of an existing EC2 instance profile

Default value: ""

existing_service_role_arn (string) optional

ARN of an existing EMR service role to attach to the cluster

Default value: ""

keep_job_flow_alive_when_no_steps (bool) optional

Switch on/off run cluster with no steps or when all steps are complete

Default value: true

kerberos_ad_domain_join_password (string) optional

The Active Directory password for ad_domain_join_user. Terraform cannot perform drift detection of this configuration.

Default value: null

kerberos_ad_domain_join_user (string) optional

Required only when establishing a cross-realm trust with an Active Directory domain. A user with sufficient privileges to join resources to the domain. Terraform cannot perform drift detection of this configuration.

Default value: null

kerberos_cross_realm_trust_principal_password (string) optional

Required only when establishing a cross-realm trust with a KDC in a different realm. The cross-realm principal password, which must be identical across realms. Terraform cannot perform drift detection of this configuration.

Default value: null

kerberos_enabled (bool) optional

Set to true if EMR cluster will use kerberos_attributes

Default value: false

kerberos_kdc_admin_password (string) optional

The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster. Terraform cannot perform drift detection of this configuration.

Default value: null

kerberos_realm (string) optional

The name of the Kerberos realm to which all nodes in a cluster belong. For example, EC2.INTERNAL

Default value: "EC2.INTERNAL"

key_name (string) optional

Amazon EC2 key pair that can be used to ssh to the master node as the user called hadoop

Default value: null

log_uri (string) optional

The path to the Amazon S3 location where logs for this cluster are stored

Default value: null

managed_master_security_group (string) optional

The id of the existing managed security group that will be used for EMR master node. If empty, a new security group will be created

Default value: ""

managed_slave_security_group (string) optional

The id of the existing managed security group that will be used for EMR core & task nodes. If empty, a new security group will be created

Default value: ""

master_allowed_cidr_blocks (list(string)) optional

List of CIDR blocks to be allowed to access the master instances

Default value: [ ]

master_allowed_security_groups (list(string)) optional

List of security group ids to be allowed to connect to the master instances

Default value: [ ]

master_dns_name (string) optional

Name of the cluster CNAME record to create in the parent DNS zone specified by zone_id. If left empty, the name will be auto-asigned using the format emr-master-var.name

Default value: null

master_instance_group_bid_price (string) optional

Bid price for each EC2 instance in the Master instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances

Default value: null

master_instance_group_ebs_iops (number) optional

The number of I/O operations per second (IOPS) that the Master volume supports

Default value: null

master_instance_group_ebs_type (string) optional

Master instances volume type. Valid options are gp2, io1, standard and st1

Default value: "gp2"

master_instance_group_ebs_volumes_per_instance (number) optional

The number of EBS volumes with this configuration to attach to each EC2 instance in the Master instance group

Default value: 1

master_instance_group_instance_count (number) optional

Target number of instances for the Master instance group. Must be at least 1

Default value: 1

release_label (string) optional

The release label for the Amazon EMR release. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html

Default value: "emr-5.25.0"

route_table_id (string) optional

Route table ID for the VPC S3 Endpoint when launching the EMR cluster in a private subnet. Required when subnet_type is private

Default value: ""

scale_down_behavior (string) optional

The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized

Default value: null

security_configuration (string) optional

The security configuration name to attach to the EMR cluster. Only valid for EMR clusters with release_label 4.8.0 or greater. See https://www.terraform.io/docs/providers/aws/r/emr_security_configuration.html for more info

Default value: null

service_access_security_group (string) optional

The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created

Default value: ""

service_role_enabled (bool) optional

If set to false, will use existing_service_role_arn for an existing IAM role that was created outside of this module

Default value: true

slave_allowed_cidr_blocks (list(string)) optional

List of CIDR blocks to be allowed to access the slave instances

Default value: [ ]

slave_allowed_security_groups (list(string)) optional

List of security group ids to be allowed to connect to the slave instances

Default value: [ ]

step_concurrency_level (number) optional

The number of steps that can be executed concurrently. You can specify a maximum of 256 steps. Only valid for EMR clusters with release_label 5.28.0 or greater.

Default value: null

steps optional

List of steps to run when creating the cluster.

Type:

list(object({
name              = string
action_on_failure = string
hadoop_jar_step = object({
  args       = list(string)
  jar        = string
  main_class = string
  properties = map(string)
})
}))

Default value: [ ]

subnet_type (string) optional

Type of VPC subnet ID where you want the job flow to launch. Supported values are private or public

Default value: "private"

task_instance_group_autoscaling_policy (string) optional

String containing the EMR Auto Scaling Policy JSON for the Task instance group

Default value: null

task_instance_group_bid_price (string) optional

Bid price for each EC2 instance in the Task instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances

Default value: null

task_instance_group_ebs_iops (number) optional

The number of I/O operations per second (IOPS) that the Task volume supports

Default value: null

task_instance_group_ebs_optimized (bool) optional

Indicates whether an Amazon EBS volume in the Task instance group is EBS-optimized. Changing this forces a new resource to be created

Default value: false

task_instance_group_ebs_size (number) optional

Task instances volume size, in gibibytes (GiB)

Default value: 10

task_instance_group_ebs_type (string) optional

Task instances volume type. Valid options are gp2, io1, standard and st1

Default value: "gp2"

task_instance_group_ebs_volumes_per_instance (number) optional

The number of EBS volumes with this configuration to attach to each EC2 instance in the Task instance group

Default value: 1

task_instance_group_instance_count (number) optional

Target number of instances for the Task instance group. Must be at least 1

Default value: 1

task_instance_group_instance_type (string) optional

EC2 instance type for all instances in the Task instance group

Default value: null

termination_protection (bool) optional

Switch on/off termination protection (default is false, except when using multiple master nodes). Before attempting to destroy the resource when termination protection is enabled, this configuration must be applied with its value set to false

Default value: false

use_existing_additional_master_security_group (bool) optional

If set to true, will use variable additional_master_security_group using an existing security group that was created outside of this module

Default value: false

use_existing_additional_slave_security_group (bool) optional

If set to true, will use variable additional_slave_security_group using an existing security group that was created outside of this module

Default value: false

use_existing_managed_master_security_group (bool) optional

If set to true, will use variable managed_master_security_group using an existing security group that was created outside of this module

Default value: false

use_existing_managed_slave_security_group (bool) optional

If set to true, will use variable managed_slave_security_group using an existing security group that was created outside of this module

Default value: false

use_existing_service_access_security_group (bool) optional

If set to true, will use variable service_access_security_group using an existing security group that was created outside of this module

Default value: false

visible_to_all_users (bool) optional

Whether the job flow is visible to all IAM users of the AWS account associated with the job flow

Default value: true

zone_id (string) optional

Route53 parent zone ID. If provided (not empty), the module will create sub-domain DNS records for the masters and slaves

Default value: null

Context Variables

The following variables are defined in the context.tf file of this module and part of the terraform-null-label pattern.

additional_tag_map (map(string)) optional

Additional key-value pairs to add to each map in tags_as_list_of_maps. Not added to tags or id.
This is for some rare cases where resources want additional configuration of tags
and therefore take a list of maps with tag key, value, and additional configuration.

Required: No

Default value: { }

attributes (list(string)) optional

ID element. Additional attributes (e.g. workers or cluster) to add to id,
in the order they appear in the list. New attributes are appended to the
end of the list. The elements of the list are joined by the delimiter
and treated as a single ID element.

Required: No

Default value: [ ]

context (any) optional

Single object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as null to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.

Required: No

Default value:

{
  "additional_tag_map": {},
  "attributes": [],
  "delimiter": null,
  "descriptor_formats": {},
  "enabled": true,
  "environment": null,
  "id_length_limit": null,
  "label_key_case": null,
  "label_order": [],
  "label_value_case": null,
  "labels_as_tags": [
    "unset"
  ],
  "name": null,
  "namespace": null,
  "regex_replace_chars": null,
  "stage": null,
  "tags": {},
  "tenant": null
}

delimiter (string) optional

Delimiter to be used between ID elements.
Defaults to - (hyphen). Set to "" to use no delimiter at all.

Required: No

Default value: null

descriptor_formats (any) optional

Describe additional descriptors to be output in the descriptors output map.
Map of maps. Keys are names of descriptors. Values are maps of the form
\{<br/> format = string<br/> labels = list(string)<br/> \}
(Type is any so the map values can later be enhanced to provide additional options.)
format is a Terraform format string to be passed to the format() function.
labels is a list of labels, in order, to pass to format() function.
Label values will be normalized before being passed to format() so they will be
identical to how they appear in id.
Default is {} (descriptors output will be empty).

Required: No

Default value: { }

enabled (bool) optional

Set to false to prevent the module from creating any resources
Required: No

Default value: null

environment (string) optional

ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'
Required: No

Default value: null

id_length_limit (number) optional

Limit id to this many characters (minimum 6).
Set to 0 for unlimited length.
Set to null for keep the existing setting, which defaults to 0.
Does not affect id_full.

Required: No

Default value: null

label_key_case (string) optional

Controls the letter case of the tags keys (label names) for tags generated by this module.
Does not affect keys of tags passed in via the tags input.
Possible values: lower, title, upper.
Default value: title.

Required: No

Default value: null

label_order (list(string)) optional

The order in which the labels (ID elements) appear in the id.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.

Required: No

Default value: null

label_value_case (string) optional

Controls the letter case of ID elements (labels) as included in id,
set as tag values, and output by this module individually.
Does not affect values of tags passed in via the tags input.
Possible values: lower, title, upper and none (no transformation).
Set this to title and set delimiter to "" to yield Pascal Case IDs.
Default value: lower.

Required: No

Default value: null

labels_as_tags (set(string)) optional

Set of labels (ID elements) to include as tags in the tags output.
Default is to include all labels.
Tags with empty values will not be included in the tags output.
Set to [] to suppress all generated tags.
Notes:
The value of the name tag, if included, will be the id, not the name.
Unlike other null-label inputs, the initial setting of labels_as_tags cannot be
changed in later chained modules. Attempts to change it will be silently ignored.

Required: No

Default value:

[
  "default"
]

name (string) optional

ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.
This is the only ID element not also included as a tag.
The "name" tag is set to the full id string. There is no tag with the value of the name input.

Required: No

Default value: null

namespace (string) optional

ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique
Required: No

Default value: null

regex_replace_chars (string) optional

Terraform regular expression (regex) string.
Characters matching the regex will be removed from the ID elements.
If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits.

Required: No

Default value: null

stage (string) optional

ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'
Required: No

Default value: null

tags (map(string)) optional

Additional tags (e.g. {'BusinessUnit': 'XYZ'}).
Neither the tag keys nor the tag values will be modified by this module.

Required: No

Default value: { }

tenant (string) optional

ID element (Rarely used, not included by default). A customer identifier, indicating who this instance of a resource is for
Required: No

Default value: null

Outputs

cluster_id: EMR cluster ID
cluster_name: EMR cluster name
ec2_role: Role name of EMR EC2 instances so users can attach more policies
master_host: Name of the cluster CNAME record for the master nodes in the parent DNS zone
master_public_dns: Master public DNS
master_security_group_id: Master security group ID
slave_security_group_id: Slave security group ID

Dependencies

Requirements

terraform, version: >= 0.14.0
aws, version: >= 3.5.0

Providers

aws, version: >= 3.5.0

Modules

Name	Version	Source	Description
`dns_master`	0.12.2	`cloudposse/route53-cluster-hostname/aws`	n/a
`label_core`	0.25.0	`cloudposse/label/null`	n/a
`label_ec2`	0.25.0	`cloudposse/label/null`	n/a
`label_ec2_autoscaling`	0.25.0	`cloudposse/label/null`	n/a
`label_emr`	0.25.0	`cloudposse/label/null`	n/a
`label_master`	0.25.0	`cloudposse/label/null`	n/a
`label_master_managed`	0.25.0	`cloudposse/label/null`	n/a
`label_service_managed`	0.25.0	`cloudposse/label/null`	n/a
`label_slave`	0.25.0	`cloudposse/label/null`	n/a
`label_slave_managed`	0.25.0	`cloudposse/label/null`	n/a
`label_task`	0.25.0	`cloudposse/label/null`	n/a
`this`	0.25.0	`cloudposse/label/null`	n/a

Resources

The following resources are used by this module:

aws_emr_cluster.default (resource)
aws_emr_instance_group.task (resource)
aws_iam_instance_profile.ec2 (resource)
aws_iam_role.ec2 (resource)
aws_iam_role.ec2_autoscaling (resource)
aws_iam_role.emr (resource)
aws_iam_role_policy_attachment.ec2 (resource)
aws_iam_role_policy_attachment.ec2_autoscaling (resource)
aws_iam_role_policy_attachment.emr (resource)
aws_iam_role_policy_attachment.emr_ssm_access (resource)
aws_security_group.managed_master (resource)
aws_security_group.managed_service_access (resource)
aws_security_group.managed_slave (resource)
aws_security_group.master (resource)
aws_security_group.slave (resource)
aws_security_group_rule.managed_master_egress (resource)
aws_security_group_rule.managed_master_service_access_ingress (resource)
aws_security_group_rule.managed_service_access_egress (resource)
aws_security_group_rule.managed_slave_egress (resource)
aws_security_group_rule.master_egress (resource)
aws_security_group_rule.master_ingress_cidr_blocks (resource)
aws_security_group_rule.master_ingress_security_groups (resource)
aws_security_group_rule.slave_egress (resource)
aws_security_group_rule.slave_ingress_cidr_blocks (resource)
aws_security_group_rule.slave_ingress_security_groups (resource)
aws_vpc_endpoint.vpc_endpoint_s3 (resource)

Data Sources

The following data sources are used by this module:

aws_iam_policy_document.assume_role_ec2 (data source)
aws_iam_policy_document.assume_role_emr (data source)
aws_partition.current (data source)

Module: emr-cluster

Usage​

Variables​

Required Variables​

Optional Variables​

Context Variables​

Outputs​

Dependencies​

Requirements​

Providers​

Modules​

Resources​

Data Sources​

Module: `emr-cluster`

Usage

Variables

Required Variables

Optional Variables

Context Variables

Outputs

Dependencies

Requirements

Providers

Modules

Resources

Data Sources