Skip to main content
Sponsor @cloudposse

Decide on Amazon OpenSearch Service (Elasticsearch) Requirements

Problem​

Requirements for the Amazon OpenSearch Service (formerly known as Elasticsearch) clusters deployed to each active compute environment need to be outlined before an Elasticsearch component is configured and deployed

Context​

At a minimum, we need the following for each operating stage (prod, staging, dev, etc)

  • Instance family for each node

  • Number of nodes

  • EBS volume size

  • Whether or not Kibana is required

See https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html for Amazon’s recommendations.

Considered Options​

Create a standardized Elasticsearch (Amazon OpenSearch Service) cluster based on one of these options. We’ll also need to know how these requirements will vary by stage.

caution

This is a reversible decision, however, resizing large OpenSearch clusters can literally take several days.

Option 1: Use Current Infrastructure Requirements​

info

If already opening OpenSearch, we recommend sharing a screenshot of your current setup from the AWS web console for each cluster in every environment.

Option 2: Use Minimal Elasticsearch (Amazon OpenSearch) Cluster Requirements​

Because the Amazon OpenSearch Service recommends having at least 3 nodes in each Elasticsearch cluster in order to avoid a split-brain scenario, each cluster should contain 3 nodes (if it were to be minimally sized).

This, in addition to the requirements outlined in v1 Infrastructure Requirements, concludes that each Elasticsearch cluster will have the following requirements:

RequirementRecommendation
EBS volume size:::caution
The volume size is limited by the size of the instance.
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/limits.html


:::
Number of nodes3
Instance family for each nodeDepends on use-case
KibanaWhether or not Kibana is required: not required.


:::caution
If Kibana is required, we’ll need to discuss how to securely access Kibana. We recommend SAML authentication.

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/saml.html


:::

Consequences​

Provision Amazon OpenSearch Service based on these requirements using the elasticsearch component with terraform.

  • This allows for a standardized Elasticsearch cluster that satisfies the requirements required by the application stack in each active compute environment.

  • This standard size can be easily adjusted as needed, so this is an easily reversible decision.

References​