Module: efs-backup

Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline.

The workflow is simple:

• Periodically launch resource (EC2 instance) based on schedule
• Execute the shell command defined in the activity on the instance
• Sync data from Production EFS to S3 Bucket by using aws-cli
• The execution log of the activity is stored in S3
• Publish the success or failure of the activity to an SNS topic
• Automatically rotate the backups using S3 lifecycle rule

Usage

Include this module in your existing terraform code:

module "efs_backup" {
  source = "git::https://github.com/cloudposse/terraform-aws-efs-backup.git?ref=master"

  name                               = "${var.name}"
  stage                              = "${var.stage}"
  namespace                          = "${var.namespace}"
  vpc_id                             = "${var.vpc_id}"
  efs_mount_target_id                = "${var.efs_mount_target_id}"
  use_ip_address                     = "false"
  noncurrent_version_expiration_days = "${var.noncurrent_version_expiration_days}"
  ssh_key_pair                       = "${var.ssh_key_pair}"
  datapipeline_config                = "${var.datapipeline_config}"
  modify_security_group              = "true"
}

output "efs_backup_security_group" {
  value = "${module.efs_backup.security_group_id}"
}

Integration with EFS

To enable connectivity between the DataPipeline instances and the EFS, use one of the following methods to configure Security Groups:

1. Explicitly add the DataPipeline SG (the output of this module security_group_id) to the list of the ingress rules of the EFS SG. For example:

module "elastic_beanstalk_environment" {
  source     = "git::https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment.git?ref=master"
  namespace  = "${var.namespace}"
  name       = "${var.name}"
  stage      = "${var.stage}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("eb-env")))}"]
  tags       = "${var.tags}"

  # ..............................
}

module "efs" {
  source     = "git::https://github.com/cloudposse/terraform-aws-efs.git?ref=tmaster"
  namespace  = "${var.namespace}"
  name       = "${var.name}"
  stage      = "${var.stage}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("efs")))}"]
  tags       = "${var.tags}"

  # Allow EB/EC2 instances and DataPipeline instances to connect to the EFS
  security_groups = ["${module.elastic_beanstalk_environment.security_group_id}", "${module.efs_backup.security_group_id}"]
}

module "efs_backup" {
  source     = "git::https://github.com/cloudposse/terraform-aws-efs-backup.git?ref=master"
  name       = "${var.name}"
  stage      = "${var.stage}"
  namespace  = "${var.namespace}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("efs-backup")))}"]
  tags       = "${var.tags}"
  
  # Important to set it to `false` since we added the `DataPipeline` SG (output of the `efs_backup` module) to the `security_groups` of the `efs` module
  # See NOTE below for more information
  modify_security_group = "false"

  # ..............................
}

2. Set modify_security_group attribute to true so the module will modify the EFS SG to allow the DataPipeline to connect to the EFS

NOTE: Do not mix these two methods together. Terraform does not support using a Security Group with in-line rules in conjunction with any Security Group Rule resources. https://www.terraform.io/docs/providers/aws/r/security_group_rule.html

NOTE on Security Groups and Security Group Rules: Terraform currently provides both a standalone Security Group Rule resource (a single ingress or egress rule), and a Security Group resource with ingress and egress rules defined in-line. At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

Variables

Required Variables

efs_mount_target_id (string) required

EFS Mount Target ID (e.g. fsmt-279bfc62)

ssh_key_pair (string) required

SSH key that will be deployed on DataPipeline's instance

Optional Variables

datapipeline_config (map(string)) optional

DataPipeline configuration options

Default value:

{
  "email": "",
  "instance_type": "t2.micro",
  "period": "24 hours",
  "timeout": "60 Minutes"
}

datapipeline_security_group (string) optional

Optionally specify a security group to use for the datapipeline instances

Default value: ""

modify_security_group (string) optional

Should the module modify the EFS security group

Default value: "false"

noncurrent_version_expiration_days (string) optional

S3 object versions expiration period (days)

Default value: "35"

region (string) optional

(Optional) AWS Region. If not specified, will be derived from 'aws_region' data source

Default value: ""

subnet_id (string) optional

Optionally specify the subnet to use

Default value: ""

use_ip_address (string) optional

If set to true, will use IP address instead of DNS name to connect to the EFS

Default value: "false"

vpc_id (string) optional

VPC ID

Default value: ""

Context Variables

The following variables are defined in the context.tf file of this module and part of the terraform-null-label pattern.

name (any) required

The Name of the application or solution (e.g. bastion or portal)
Required: Yes

Default value: ``

namespace (any) required

Namespace (e.g. cp or cloudposse)
Required: Yes

Default value: ``

stage (any) required

Stage (e.g. prod, dev, staging)
Required: Yes

Default value: ``

attributes (list(string)) optional

Additional attributes (e.g. efs-backup)
Required: No

Default value: [ ]

delimiter (string) optional

Delimiter to be used between name, namespace, stage, etc.
Required: No

Default value: "-"

tags (map(string)) optional

Additional tags (e.g. map('BusinessUnit,XYZ)
Required: No

Default value: { }

Outputs

backups_bucket_name

Backups bucket name

datapipeline_ids

Datapipeline ids

logs_bucket_name

Logs bucket name

security_group_id

Security group id

sns_topic_arn

Backup notification SNS topic ARN

Dependencies

Providers

• aws

Modules

Name	Version	Source	Description
backups_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
datapipeline_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
logs_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
resource_role_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
role_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a
sns_label	tags/0.3.1	git::https://github.com/cloudposse/terraform-null-label.git	n/a

Resources

The following resources are used by this module:

• aws_cloudformation_stack.datapipeline (resource)
• aws_cloudformation_stack.sns (resource)
• aws_iam_instance_profile.resource_role (resource)
• aws_iam_role.resource_role (resource)
• aws_iam_role.role (resource)
• aws_iam_role_policy_attachment.resource_role (resource)
• aws_iam_role_policy_attachment.role (resource)
• aws_s3_bucket.backups (resource)
• aws_s3_bucket.logs (resource)
• aws_security_group.datapipeline (resource)
• aws_security_group_rule.datapipeline_efs_ingress (resource)

Data Sources

The following data sources are used by this module:

• aws_ami.amazon_linux (data source)
• aws_efs_mount_target.default (data source)
• aws_iam_policy_document.resource_role (data source)
• aws_iam_policy_document.role (data source)
• aws_region.default (data source)
• aws_subnet_ids.default (data source)
• aws_vpc.default (data source)