Module: `efs-backup`

Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline.

The workflow is simple:

Periodically launch resource (EC2 instance) based on schedule
Execute the shell command defined in the activity on the instance
Sync data from Production EFS to S3 Bucket by using aws-cli
The execution log of the activity is stored in S3
Publish the success or failure of the activity to an SNS topic
Automatically rotate the backups using S3 lifecycle rule

Usage

Include this module in your existing terraform code:

module "efs_backup" {
  source = "git::https://github.com/cloudposse/terraform-aws-efs-backup.git?ref=master"

  name                               = "${var.name}"
  stage                              = "${var.stage}"
  namespace                          = "${var.namespace}"
  vpc_id                             = "${var.vpc_id}"
  efs_mount_target_id                = "${var.efs_mount_target_id}"
  use_ip_address                     = "false"
  noncurrent_version_expiration_days = "${var.noncurrent_version_expiration_days}"
  ssh_key_pair                       = "${var.ssh_key_pair}"
  datapipeline_config                = "${var.datapipeline_config}"
  modify_security_group              = "true"
}

output "efs_backup_security_group" {
  value = "${module.efs_backup.security_group_id}"
}

Integration with `EFS`

To enable connectivity between the DataPipeline instances and the EFS, use one of the following methods to configure Security Groups:

Explicitly add the DataPipeline SG (the output of this module security_group_id) to the list of the ingress rules of the EFS SG. For example:

module "elastic_beanstalk_environment" {
  source     = "git::https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment.git?ref=master"
  namespace  = "${var.namespace}"
  name       = "${var.name}"
  stage      = "${var.stage}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("eb-env")))}"]
  tags       = "${var.tags}"

  # ..............................
}

module "efs" {
  source     = "git::https://github.com/cloudposse/terraform-aws-efs.git?ref=tmaster"
  namespace  = "${var.namespace}"
  name       = "${var.name}"
  stage      = "${var.stage}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("efs")))}"]
  tags       = "${var.tags}"

  # Allow EB/EC2 instances and DataPipeline instances to connect to the EFS
  security_groups = ["${module.elastic_beanstalk_environment.security_group_id}", "${module.efs_backup.security_group_id}"]
}

module "efs_backup" {
  source     = "git::https://github.com/cloudposse/terraform-aws-efs-backup.git?ref=master"
  name       = "${var.name}"
  stage      = "${var.stage}"
  namespace  = "${var.namespace}"
  delimiter  = "${var.delimiter}"
  attributes = ["${compact(concat(var.attributes, list("efs-backup")))}"]
  tags       = "${var.tags}"
  
  # Important to set it to `false` since we added the `DataPipeline` SG (output of the `efs_backup` module) to the `security_groups` of the `efs` module
  # See NOTE below for more information
  modify_security_group = "false"

  # ..............................
}

Set modify_security_group attribute to true so the module will modify the EFS SG to allow the DataPipeline to connect to the EFS

NOTE: Do not mix these two methods together. Terraform does not support using a Security Group with in-line rules in conjunction with any Security Group Rule resources. https://www.terraform.io/docs/providers/aws/r/security_group_rule.html

NOTE on Security Groups and Security Group Rules: Terraform currently provides both a standalone Security Group Rule resource (a single ingress or egress rule), and a Security Group resource with ingress and egress rules defined in-line. At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

Requirements

No requirements.

Providers

Name	Version
aws	n/a

Modules

Name	Source	Version
backups_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
datapipeline_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
logs_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
resource_role_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
role_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1
sns_label	git::https://github.com/cloudposse/terraform-null-label.git	tags/0.3.1

Resources

Name	Type
aws_cloudformation_stack.datapipeline	resource
aws_cloudformation_stack.sns	resource
aws_iam_instance_profile.resource_role	resource
aws_iam_role.resource_role	resource
aws_iam_role.role	resource
aws_iam_role_policy_attachment.resource_role	resource
aws_iam_role_policy_attachment.role	resource
aws_s3_bucket.backups	resource
aws_s3_bucket.logs	resource
aws_security_group.datapipeline	resource
aws_security_group_rule.datapipeline_efs_ingress	resource
aws_ami.amazon_linux	data source
aws_efs_mount_target.default	data source
aws_iam_policy_document.resource_role	data source
aws_iam_policy_document.role	data source
aws_region.default	data source
aws_subnet_ids.default	data source
aws_vpc.default	data source

Inputs

Name	Description	Type	Default	Required
attributes	Additional attributes (e.g. `efs-backup`)	`list(string)`	`[]`	no
datapipeline_config	DataPipeline configuration options	`map(string)`	`{ "email": "", "instance_type": "t2.micro", "period": "24 hours", "timeout": "60 Minutes" }`	no
datapipeline_security_group	Optionally specify a security group to use for the datapipeline instances	`string`	`""`	no
delimiter	Delimiter to be used between `name`, `namespace`, `stage`, etc.	`string`	`"-"`	no
efs_mount_target_id	EFS Mount Target ID (e.g. `fsmt-279bfc62`)	`string`	n/a	yes
modify_security_group	Should the module modify the `EFS` security group	`string`	`"false"`	no
name	The Name of the application or solution (e.g. `bastion` or `portal`)	`any`	n/a	yes
namespace	Namespace (e.g. `cp` or `cloudposse`)	`any`	n/a	yes
noncurrent_version_expiration_days	S3 object versions expiration period (days)	`string`	`"35"`	no
region	(Optional) AWS Region. If not specified, will be derived from 'aws_region' data source	`string`	`""`	no
ssh_key_pair	`SSH` key that will be deployed on DataPipeline's instance	`string`	n/a	yes
stage	Stage (e.g. `prod`, `dev`, `staging`)	`any`	n/a	yes
subnet_id	Optionally specify the subnet to use	`string`	`""`	no
tags	Additional tags (e.g. `map('BusinessUnit`,`XYZ`)	`map(string)`	`{}`	no
use_ip_address	If set to `true`, will use IP address instead of DNS name to connect to the `EFS`	`string`	`"false"`	no
vpc_id	VPC ID	`string`	`""`	no

Outputs

Name	Description
backups_bucket_name	Backups bucket name
datapipeline_ids	Datapipeline ids
logs_bucket_name	Logs bucket name
security_group_id	Security group id
sns_topic_arn	Backup notification SNS topic ARN

Module: efs-backup

Usage​

Integration with EFS​

Requirements​

Providers​

Modules​

Resources​

Inputs​

Outputs​