Component: `glue/crawler`

This component provisions Glue crawlers.

Usage

Stack Level: Regional

components:
  terraform:
    # The crawler crawls the data in an S3 bucket and puts the results into a table in the Glue Catalog.
    # The crawler will read the first 2 MB of data from the file, and recognize the schema.
    # After that, the crawler will sync the table.
    glue/crawler/example:
      metadata:
        component: glue/crawler
      vars:
        enabled: true
        name: example
        crawler_description: "Glue crawler example"
        glue_iam_component_name: "glue/iam"
        glue_catalog_database_component_name: "glue/catalog-database/example"
        glue_catalog_table_component_name: "glue/catalog-table/example"
        schedule: "cron(0 1 * * ? *)"
        schema_change_policy:
          delete_behavior: LOG
          update_behavior: null

Requirements

Name	Version
terraform	>= 1.3.0
aws	>= 4.0
utils	>= 1.15.0

Providers

No providers.

Modules

Name	Source	Version
glue_catalog_database	cloudposse/stack-config/yaml//modules/remote-state	1.5.0
glue_catalog_table	cloudposse/stack-config/yaml//modules/remote-state	1.5.0
glue_crawler	cloudposse/glue/aws//modules/glue-crawler	0.4.0
glue_iam_role	cloudposse/stack-config/yaml//modules/remote-state	1.5.0
iam_roles	../../account-map/modules/iam-roles	n/a
this	cloudposse/label/null	0.25.0

Resources

No resources.

Inputs

Name	Description	Type	Default	Required
additional_tag_map	Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`. This is for some rare cases where resources want additional configuration of tags and therefore take a list of maps with tag key, value, and additional configuration.	`map(string)`	`{}`	no
attributes	ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`, in the order they appear in the list. New attributes are appended to the end of the list. The elements of the list are joined by the `delimiter` and treated as a single ID element.	`list(string)`	`[]`	no
catalog_target	List of nested Glue catalog target arguments	`list(object({ database_name = string tables = list(string) }))`	`null`	no
classifiers	List of custom classifiers. By default, all AWS classifiers are included in a crawl, but these custom classifiers always override the default classifiers for a given classification	`list(string)`	`null`	no
configuration	JSON string of configuration information	`string`	`null`	no
context	Single object for setting entire context at once. See description of individual variables for details. Leave string and numeric variables as `null` to use default value. Individual variable settings (non-null) override settings in context object, except for attributes, tags, and additional_tag_map, which are merged.	`any`	`{ "additional_tag_map": {}, "attributes": [], "delimiter": null, "descriptor_formats": {}, "enabled": true, "environment": null, "id_length_limit": null, "label_key_case": null, "label_order": [], "label_value_case": null, "labels_as_tags": [ "unset" ], "name": null, "namespace": null, "regex_replace_chars": null, "stage": null, "tags": {}, "tenant": null }`	no
crawler_description	Glue crawler description	`string`	`null`	no
crawler_name	Glue crawler name. If not provided, the name will be generated from the context	`string`	`null`	no
delimiter	Delimiter to be used between ID elements. Defaults to `-` (hyphen). Set to `""` to use no delimiter at all.	`string`	`null`	no
delta_target	List of nested Delta target arguments	`list(object({ connection_name = string delta_tables = list(string) write_manifest = bool }))`	`null`	no
descriptor_formats	Describe additional descriptors to be output in the `descriptors` output map. Map of maps. Keys are names of descriptors. Values are maps of the form `{<br/> format = string<br/> labels = list(string)<br/>}` (Type is `any` so the map values can later be enhanced to provide additional options.) `format` is a Terraform format string to be passed to the `format()` function. `labels` is a list of labels, in order, to pass to `format()` function. Label values will be normalized before being passed to `format()` so they will be identical to how they appear in `id`. Default is `{}` (`descriptors` output will be empty).	`any`	`{}`	no
dynamodb_target	List of nested DynamoDB target arguments	`list(any)`	`null`	no
enabled	Set to false to prevent the module from creating any resources	`bool`	`null`	no
environment	ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'	`string`	`null`	no
glue_catalog_database_component_name	Glue catalog database component name where metadata resides. Used to get the Glue catalog database from the remote state	`string`	n/a	yes
glue_catalog_table_component_name	Glue catalog table component name where metadata resides. Used to get the Glue catalog table from the remote state	`string`	`null`	no
glue_iam_component_name	Glue IAM component name. Used to get the Glue IAM role from the remote state	`string`	`"glue/iam"`	no
id_length_limit	Limit `id` to this many characters (minimum 6). Set to `0` for unlimited length. Set to `null` for keep the existing setting, which defaults to `0`. Does not affect `id_full`.	`number`	`null`	no
jdbc_target	List of nested JBDC target arguments	`list(any)`	`null`	no
label_key_case	Controls the letter case of the `tags` keys (label names) for tags generated by this module. Does not affect keys of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper`. Default value: `title`.	`string`	`null`	no
label_order	The order in which the labels (ID elements) appear in the `id`. Defaults to ["namespace", "environment", "stage", "name", "attributes"]. You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.	`list(string)`	`null`	no
label_value_case	Controls the letter case of ID elements (labels) as included in `id`, set as tag values, and output by this module individually. Does not affect values of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper` and `none` (no transformation). Set this to `title` and set `delimiter` to `""` to yield Pascal Case IDs. Default value: `lower`.	`string`	`null`	no
labels_as_tags	Set of labels (ID elements) to include as tags in the `tags` output. Default is to include all labels. Tags with empty values will not be included in the `tags` output. Set to `[]` to suppress all generated tags. Notes: The value of the `name` tag, if included, will be the `id`, not the `name`. Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be changed in later chained modules. Attempts to change it will be silently ignored.	`set(string)`	`[ "default" ]`	no
lineage_configuration	Specifies data lineage configuration settings for the crawler	`object({ crawler_lineage_settings = string })`	`null`	no
mongodb_target	List of nested MongoDB target arguments	`list(any)`	`null`	no
name	ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'. This is the only ID element not also included as a `tag`. The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input.	`string`	`null`	no
namespace	ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique	`string`	`null`	no
recrawl_policy	A policy that specifies whether to crawl the entire dataset again, or to crawl only folders that were added since the last crawler run	`object({ recrawl_behavior = string })`	`null`	no
regex_replace_chars	Terraform regular expression (regex) string. Characters matching the regex will be removed from the ID elements. If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits.	`string`	`null`	no
region	AWS Region	`string`	n/a	yes
s3_target	List of nested Amazon S3 target arguments	`list(any)`	`null`	no
schedule	A cron expression for the schedule	`string`	`null`	no
schema_change_policy	Policy for the crawler's update and deletion behavior	`map(string)`	`null`	no
security_configuration	The name of Security Configuration to be used by the crawler	`string`	`null`	no
stage	ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'	`string`	`null`	no
table_prefix	The table prefix used for catalog tables that are created	`string`	`null`	no
tags	Additional tags (e.g. `{'BusinessUnit': 'XYZ'}`). Neither the tag keys nor the tag values will be modified by this module.	`map(string)`	`{}`	no
tenant	ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for	`string`	`null`	no

Outputs

Name	Description
crawler_arn	Crawler ARN
crawler_id	Crawler ID
crawler_name	Crawler name

References

cloudposse/terraform-aws-components - Cloud Posse's upstream component

Component: glue/crawler

Usage​

Requirements​

Providers​

Modules​

Resources​

Inputs​

Outputs​

References​

Component: `glue/crawler`

Usage

Requirements

Providers

Modules

Resources

Inputs

Outputs

References