Module: `emr-cluster`

Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS.

Usage

For a complete example, see examples/complete

For automated tests of the complete example using bats and Terratest (which tests and deploys the example on AWS), see test.

provider "aws" {
  region = "us-east-2"
}

module "vpc" {
  source  = "cloudposse/vpc/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  ipv4_primary_cidr_block = "172.19.0.0/16"

  context = module.this.context
}

module "subnets" {
  source = "cloudposse/dynamic-subnets/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"

  availability_zones   = var.availability_zones
  vpc_id               = module.vpc.vpc_id
  igw_id               = [module.vpc.igw_id]
  ipv4_cidr_block      = [module.vpc.vpc_cidr_block]
  nat_gateway_enabled  = false
  nat_instance_enabled = false

  context = module.this.context
}

module "s3_log_storage" {
  source = "cloudposse/s3-log-storage/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  region        = var.region
  namespace     = var.namespace
  stage         = var.stage
  name          = var.name
  attributes    = ["logs"]
  force_destroy = true
}

module "aws_key_pair" {
  source = "cloudposse/key-pair/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  namespace           = var.namespace
  stage               = var.stage
  name                = var.name
  attributes          = ["ssh", "key"]
  ssh_public_key_path = var.ssh_public_key_path
  generate_ssh_key    = var.generate_ssh_key
}

module "emr_cluster" {
  source = "cloudposse/emr-cluster/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  
  namespace                                      = var.namespace
  stage                                          = var.stage
  name                                           = var.name
  master_allowed_security_groups                 = [module.vpc.vpc_default_security_group_id]
  slave_allowed_security_groups                  = [module.vpc.vpc_default_security_group_id]
  region                                         = var.region
  vpc_id                                         = module.vpc.vpc_id
  subnet_id                                      = module.subnets.private_subnet_ids[0]
  route_table_id                                 = module.subnets.private_route_table_ids[0]
  subnet_type                                    = "private"
  ebs_root_volume_size                           = var.ebs_root_volume_size
  visible_to_all_users                           = var.visible_to_all_users
  release_label                                  = var.release_label
  applications                                   = var.applications
  configurations_json                            = var.configurations_json
  core_instance_group_instance_type              = var.core_instance_group_instance_type
  core_instance_group_instance_count             = var.core_instance_group_instance_count
  core_instance_group_ebs_size                   = var.core_instance_group_ebs_size
  core_instance_group_ebs_type                   = var.core_instance_group_ebs_type
  core_instance_group_ebs_volumes_per_instance   = var.core_instance_group_ebs_volumes_per_instance
  master_instance_group_instance_type            = var.master_instance_group_instance_type
  master_instance_group_instance_count           = var.master_instance_group_instance_count
  master_instance_group_ebs_size                 = var.master_instance_group_ebs_size
  master_instance_group_ebs_type                 = var.master_instance_group_ebs_type
  master_instance_group_ebs_volumes_per_instance = var.master_instance_group_ebs_volumes_per_instance
  create_task_instance_group                     = var.create_task_instance_group
  log_uri                                        = format("s3n://%s/", module.s3_log_storage.bucket_id)
  key_name                                       = module.aws_key_pair.key_name
}

Requirements

Name	Version
terraform	>= 0.14.0
aws	>= 3.5.0

Providers

Name	Version
aws	>= 3.5.0

Modules

Name	Source	Version
dns_master	cloudposse/route53-cluster-hostname/aws	0.12.2
label_core	cloudposse/label/null	0.25.0
label_ec2	cloudposse/label/null	0.25.0
label_ec2_autoscaling	cloudposse/label/null	0.25.0
label_emr	cloudposse/label/null	0.25.0
label_master	cloudposse/label/null	0.25.0
label_master_managed	cloudposse/label/null	0.25.0
label_service_managed	cloudposse/label/null	0.25.0
label_slave	cloudposse/label/null	0.25.0
label_slave_managed	cloudposse/label/null	0.25.0
label_task	cloudposse/label/null	0.25.0
this	cloudposse/label/null	0.25.0

Resources

Name	Type
aws_emr_cluster.default	resource
aws_emr_instance_group.task	resource
aws_iam_instance_profile.ec2	resource
aws_iam_role.ec2	resource
aws_iam_role.ec2_autoscaling	resource
aws_iam_role.emr	resource
aws_iam_role_policy_attachment.ec2	resource
aws_iam_role_policy_attachment.ec2_autoscaling	resource
aws_iam_role_policy_attachment.emr	resource
aws_iam_role_policy_attachment.emr_ssm_access	resource
aws_security_group.managed_master	resource
aws_security_group.managed_service_access	resource
aws_security_group.managed_slave	resource
aws_security_group.master	resource
aws_security_group.slave	resource
aws_security_group_rule.managed_master_egress	resource
aws_security_group_rule.managed_master_service_access_ingress	resource
aws_security_group_rule.managed_service_access_egress	resource
aws_security_group_rule.managed_slave_egress	resource
aws_security_group_rule.master_egress	resource
aws_security_group_rule.master_ingress_cidr_blocks	resource
aws_security_group_rule.master_ingress_security_groups	resource
aws_security_group_rule.slave_egress	resource
aws_security_group_rule.slave_ingress_cidr_blocks	resource
aws_security_group_rule.slave_ingress_security_groups	resource
aws_vpc_endpoint.vpc_endpoint_s3	resource
aws_iam_policy_document.assume_role_ec2	data source
aws_iam_policy_document.assume_role_emr	data source
aws_partition.current	data source

Inputs

Name	Description	Type	Default	Required
additional_info	A JSON string for selecting additional features such as adding proxy information. Note: Currently there is no API to retrieve the value of this argument after EMR cluster creation from provider, therefore Terraform cannot detect drift from the actual EMR cluster if its value is changed outside Terraform	`string`	`null`	no
additional_master_security_group	The id of the existing additional security group that will be used for EMR master node. If empty, a new security group will be created	`string`	`""`	no
additional_slave_security_group	The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created	`string`	`""`	no
additional_tag_map	Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`. This is for some rare cases where resources want additional configuration of tags and therefore take a list of maps with tag key, value, and additional configuration.	`map(string)`	`{}`	no
applications	A list of applications for the cluster. Valid values are: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper (as of EMR 5.25.0). Case insensitive	`list(string)`	n/a	yes
attributes	ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`, in the order they appear in the list. New attributes are appended to the end of the list. The elements of the list are joined by the `delimiter` and treated as a single ID element.	`list(string)`	`[]`	no
auto_termination_idle_timeout	Auto termination policy idle timeout in seconds (60 - 604800 supported)	`string`	`null`	no
bootstrap_action	List of bootstrap actions that will be run before Hadoop is started on the cluster nodes	`list(object({ path = string name = string args = list(string) }))`	`[]`	no
configurations_json	A JSON string for supplying list of configurations for the EMR cluster. See https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html for more details	`string`	`""`	no
context	Single object for setting entire context at once. See description of individual variables for details. Leave string and numeric variables as `null` to use default value. Individual variable settings (non-null) override settings in context object, except for attributes, tags, and additional_tag_map, which are merged.	`any`	`{ "additional_tag_map": {}, "attributes": [], "delimiter": null, "descriptor_formats": {}, "enabled": true, "environment": null, "id_length_limit": null, "label_key_case": null, "label_order": [], "label_value_case": null, "labels_as_tags": [ "unset" ], "name": null, "namespace": null, "regex_replace_chars": null, "stage": null, "tags": {}, "tenant": null }`	no
core_instance_group_autoscaling_policy	String containing the EMR Auto Scaling Policy JSON for the Core instance group	`string`	`null`	no
core_instance_group_bid_price	Bid price for each EC2 instance in the Core instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances	`string`	`null`	no
core_instance_group_ebs_iops	The number of I/O operations per second (IOPS) that the Core volume supports	`number`	`null`	no
core_instance_group_ebs_size	Core instances volume size, in gibibytes (GiB)	`number`	n/a	yes
core_instance_group_ebs_type	Core instances volume type. Valid options are `gp2`, `io1`, `standard` and `st1`	`string`	`"gp2"`	no
core_instance_group_ebs_volumes_per_instance	The number of EBS volumes with this configuration to attach to each EC2 instance in the Core instance group	`number`	`1`	no
core_instance_group_instance_count	Target number of instances for the Core instance group. Must be at least 1	`number`	`1`	no
core_instance_group_instance_type	EC2 instance type for all instances in the Core instance group	`string`	n/a	yes
create_task_instance_group	Whether to create an instance group for Task nodes. For more info: https://www.terraform.io/docs/providers/aws/r/emr_instance_group.html, https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-master-core-task-nodes.html	`bool`	`false`	no
create_vpc_endpoint_s3	Set to false to prevent the module from creating VPC S3 Endpoint	`bool`	`true`	no
custom_ami_id	A custom Amazon Linux AMI for the cluster (instead of an EMR-owned AMI). Available in Amazon EMR version 5.7.0 and later	`string`	`null`	no
delimiter	Delimiter to be used between ID elements. Defaults to `-` (hyphen). Set to `""` to use no delimiter at all.	`string`	`null`	no
descriptor_formats	Describe additional descriptors to be output in the `descriptors` output map. Map of maps. Keys are names of descriptors. Values are maps of the form `{<br/> format = string<br/> labels = list(string)<br/>}` (Type is `any` so the map values can later be enhanced to provide additional options.) `format` is a Terraform format string to be passed to the `format()` function. `labels` is a list of labels, in order, to pass to `format()` function. Label values will be normalized before being passed to `format()` so they will be identical to how they appear in `id`. Default is `{}` (`descriptors` output will be empty).	`any`	`{}`	no
ebs_root_volume_size	Size in GiB of the EBS root device volume of the Linux AMI that is used for each EC2 instance. Available in Amazon EMR version 4.x and later	`number`	`10`	no
ec2_autoscaling_role_enabled	If set to `false`, will use `existing_ec2_autoscaling_role_arn` for an existing EC2 autoscaling IAM role that was created outside of this module	`bool`	`true`	no
ec2_autoscaling_role_permissions_boundary	The Permissions Boundary ARN to apply to the EC2 Autoscaling Role.	`string`	`""`	no
ec2_role_enabled	If set to `false`, will use `existing_ec2_instance_profile_arn` for an existing EC2 IAM role that was created outside of this module	`bool`	`true`	no
ec2_role_permissions_boundary	The Permissions Boundary ARN to apply to the EC2 Role.	`string`	`""`	no
emr_role_permissions_boundary	The Permissions Boundary ARN to apply to the EMR Role.	`string`	`""`	no
enable_ssm_access	If set to `true`, attach the existing `AmazonSSMManagedInstanceCore` IAM policy to the EMR EC2 instance profile role	`bool`	`false`	no
enabled	Set to false to prevent the module from creating any resources	`bool`	`null`	no
environment	ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'	`string`	`null`	no
existing_ec2_autoscaling_role_arn	ARN of an existing EC2 autoscaling role to attach to the cluster	`string`	`""`	no
existing_ec2_instance_profile_arn	ARN of an existing EC2 instance profile	`string`	`""`	no
existing_service_role_arn	ARN of an existing EMR service role to attach to the cluster	`string`	`""`	no
id_length_limit	Limit `id` to this many characters (minimum 6). Set to `0` for unlimited length. Set to `null` for keep the existing setting, which defaults to `0`. Does not affect `id_full`.	`number`	`null`	no
keep_job_flow_alive_when_no_steps	Switch on/off run cluster with no steps or when all steps are complete	`bool`	`true`	no
kerberos_ad_domain_join_password	The Active Directory password for ad_domain_join_user. Terraform cannot perform drift detection of this configuration.	`string`	`null`	no
kerberos_ad_domain_join_user	Required only when establishing a cross-realm trust with an Active Directory domain. A user with sufficient privileges to join resources to the domain. Terraform cannot perform drift detection of this configuration.	`string`	`null`	no
kerberos_cross_realm_trust_principal_password	Required only when establishing a cross-realm trust with a KDC in a different realm. The cross-realm principal password, which must be identical across realms. Terraform cannot perform drift detection of this configuration.	`string`	`null`	no
kerberos_enabled	Set to true if EMR cluster will use kerberos_attributes	`bool`	`false`	no
kerberos_kdc_admin_password	The password used within the cluster for the kadmin service on the cluster-dedicated KDC, which maintains Kerberos principals, password policies, and keytabs for the cluster. Terraform cannot perform drift detection of this configuration.	`string`	`null`	no
kerberos_realm	The name of the Kerberos realm to which all nodes in a cluster belong. For example, EC2.INTERNAL	`string`	`"EC2.INTERNAL"`	no
key_name	Amazon EC2 key pair that can be used to ssh to the master node as the user called `hadoop`	`string`	`null`	no
label_key_case	Controls the letter case of the `tags` keys (label names) for tags generated by this module. Does not affect keys of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper`. Default value: `title`.	`string`	`null`	no
label_order	The order in which the labels (ID elements) appear in the `id`. Defaults to ["namespace", "environment", "stage", "name", "attributes"]. You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.	`list(string)`	`null`	no
label_value_case	Controls the letter case of ID elements (labels) as included in `id`, set as tag values, and output by this module individually. Does not affect values of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper` and `none` (no transformation). Set this to `title` and set `delimiter` to `""` to yield Pascal Case IDs. Default value: `lower`.	`string`	`null`	no
labels_as_tags	Set of labels (ID elements) to include as tags in the `tags` output. Default is to include all labels. Tags with empty values will not be included in the `tags` output. Set to `[]` to suppress all generated tags. Notes: The value of the `name` tag, if included, will be the `id`, not the `name`. Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be changed in later chained modules. Attempts to change it will be silently ignored.	`set(string)`	`[ "default" ]`	no
log_uri	The path to the Amazon S3 location where logs for this cluster are stored	`string`	`null`	no
managed_master_security_group	The id of the existing managed security group that will be used for EMR master node. If empty, a new security group will be created	`string`	`""`	no
managed_slave_security_group	The id of the existing managed security group that will be used for EMR core & task nodes. If empty, a new security group will be created	`string`	`""`	no
master_allowed_cidr_blocks	List of CIDR blocks to be allowed to access the master instances	`list(string)`	`[]`	no
master_allowed_security_groups	List of security group ids to be allowed to connect to the master instances	`list(string)`	`[]`	no
master_dns_name	Name of the cluster CNAME record to create in the parent DNS zone specified by `zone_id`. If left empty, the name will be auto-asigned using the format `emr-master-var.name`	`string`	`null`	no
master_instance_group_bid_price	Bid price for each EC2 instance in the Master instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances	`string`	`null`	no
master_instance_group_ebs_iops	The number of I/O operations per second (IOPS) that the Master volume supports	`number`	`null`	no
master_instance_group_ebs_size	Master instances volume size, in gibibytes (GiB)	`number`	n/a	yes
master_instance_group_ebs_type	Master instances volume type. Valid options are `gp2`, `io1`, `standard` and `st1`	`string`	`"gp2"`	no
master_instance_group_ebs_volumes_per_instance	The number of EBS volumes with this configuration to attach to each EC2 instance in the Master instance group	`number`	`1`	no
master_instance_group_instance_count	Target number of instances for the Master instance group. Must be at least 1	`number`	`1`	no
master_instance_group_instance_type	EC2 instance type for all instances in the Master instance group	`string`	n/a	yes
name	ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'. This is the only ID element not also included as a `tag`. The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input.	`string`	`null`	no
namespace	ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique	`string`	`null`	no
regex_replace_chars	Terraform regular expression (regex) string. Characters matching the regex will be removed from the ID elements. If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits.	`string`	`null`	no
region	AWS region	`string`	n/a	yes
release_label	The release label for the Amazon EMR release. https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html	`string`	`"emr-5.25.0"`	no
route_table_id	Route table ID for the VPC S3 Endpoint when launching the EMR cluster in a private subnet. Required when `subnet_type` is `private`	`string`	`""`	no
scale_down_behavior	The way that individual Amazon EC2 instances terminate when an automatic scale-in activity occurs or an instance group is resized	`string`	`null`	no
security_configuration	The security configuration name to attach to the EMR cluster. Only valid for EMR clusters with `release_label` 4.8.0 or greater. See https://www.terraform.io/docs/providers/aws/r/emr_security_configuration.html for more info	`string`	`null`	no
service_access_security_group	The id of the existing additional security group that will be used for EMR core & task nodes. If empty, a new security group will be created	`string`	`""`	no
service_role_enabled	If set to `false`, will use `existing_service_role_arn` for an existing IAM role that was created outside of this module	`bool`	`true`	no
slave_allowed_cidr_blocks	List of CIDR blocks to be allowed to access the slave instances	`list(string)`	`[]`	no
slave_allowed_security_groups	List of security group ids to be allowed to connect to the slave instances	`list(string)`	`[]`	no
stage	ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'	`string`	`null`	no
step_concurrency_level	The number of steps that can be executed concurrently. You can specify a maximum of 256 steps. Only valid for EMR clusters with release_label 5.28.0 or greater.	`number`	`null`	no
steps	List of steps to run when creating the cluster.	`list(object({ name = string action_on_failure = string hadoop_jar_step = object({ args = list(string) jar = string main_class = string properties = map(string) }) }))`	`[]`	no
subnet_id	VPC subnet ID where you want the job flow to launch. Cannot specify the `cc1.4xlarge` instance type for nodes of a job flow launched in a Amazon VPC	`string`	n/a	yes
subnet_type	Type of VPC subnet ID where you want the job flow to launch. Supported values are `private` or `public`	`string`	`"private"`	no
tags	Additional tags (e.g. `{'BusinessUnit': 'XYZ'}`). Neither the tag keys nor the tag values will be modified by this module.	`map(string)`	`{}`	no
task_instance_group_autoscaling_policy	String containing the EMR Auto Scaling Policy JSON for the Task instance group	`string`	`null`	no
task_instance_group_bid_price	Bid price for each EC2 instance in the Task instance group, expressed in USD. By setting this attribute, the instance group is being declared as a Spot Instance, and will implicitly create a Spot request. Leave this blank to use On-Demand Instances	`string`	`null`	no
task_instance_group_ebs_iops	The number of I/O operations per second (IOPS) that the Task volume supports	`number`	`null`	no
task_instance_group_ebs_optimized	Indicates whether an Amazon EBS volume in the Task instance group is EBS-optimized. Changing this forces a new resource to be created	`bool`	`false`	no
task_instance_group_ebs_size	Task instances volume size, in gibibytes (GiB)	`number`	`10`	no
task_instance_group_ebs_type	Task instances volume type. Valid options are `gp2`, `io1`, `standard` and `st1`	`string`	`"gp2"`	no
task_instance_group_ebs_volumes_per_instance	The number of EBS volumes with this configuration to attach to each EC2 instance in the Task instance group	`number`	`1`	no
task_instance_group_instance_count	Target number of instances for the Task instance group. Must be at least 1	`number`	`1`	no
task_instance_group_instance_type	EC2 instance type for all instances in the Task instance group	`string`	`null`	no
tenant	ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for	`string`	`null`	no
termination_protection	Switch on/off termination protection (default is false, except when using multiple master nodes). Before attempting to destroy the resource when termination protection is enabled, this configuration must be applied with its value set to false	`bool`	`false`	no
use_existing_additional_master_security_group	If set to `true`, will use variable `additional_master_security_group` using an existing security group that was created outside of this module	`bool`	`false`	no
use_existing_additional_slave_security_group	If set to `true`, will use variable `additional_slave_security_group` using an existing security group that was created outside of this module	`bool`	`false`	no
use_existing_managed_master_security_group	If set to `true`, will use variable `managed_master_security_group` using an existing security group that was created outside of this module	`bool`	`false`	no
use_existing_managed_slave_security_group	If set to `true`, will use variable `managed_slave_security_group` using an existing security group that was created outside of this module	`bool`	`false`	no
use_existing_service_access_security_group	If set to `true`, will use variable `service_access_security_group` using an existing security group that was created outside of this module	`bool`	`false`	no
visible_to_all_users	Whether the job flow is visible to all IAM users of the AWS account associated with the job flow	`bool`	`true`	no
vpc_id	VPC ID to create the cluster in (e.g. `vpc-a22222ee`)	`string`	n/a	yes
zone_id	Route53 parent zone ID. If provided (not empty), the module will create sub-domain DNS records for the masters and slaves	`string`	`null`	no

Outputs

Name	Description
cluster_id	EMR cluster ID
cluster_name	EMR cluster name
ec2_role	Role name of EMR EC2 instances so users can attach more policies
master_host	Name of the cluster CNAME record for the master nodes in the parent DNS zone
master_public_dns	Master public DNS
master_security_group_id	Master security group ID
slave_security_group_id	Slave security group ID

Module: emr-cluster

Usage​

Requirements​

Providers​

Modules​

Resources​

Inputs​

Outputs​