Elasticsearch Modules - elasticsearch - elasticsearch tutorial - elastic search



 modules

Elasticsearch Modules

  • This section contains modules responsible for various aspects of the functionality in Elasticsearch. Each module has settings which may be:

    static

    • These settings must be set at the node level, either in the elasticsearch.yml file, or as an environment variable or on the command line when starting a node. They must be set on every relevant node in the cluster.

    dynamic

    • These settings can be dynamically updated on a live cluster with the cluster-update-settings API.
    • We will discuss the different modules of Elasticsearch in the following sections of this chapter.

Cluster-Level Routing and Shard Allocation

  • Cluster level settings decide the allocation of shards to different nodes and reallocation of shards to rebalance cluster. These are the following settings to control shard allocation −

Cluster-Level Shard Allocation

Setting Possible value Description
cluster.routing.allocation.enable
all This default value allows shard
allocation for all kinds of shards.
primaries This allows shard allocation only for primary shards.
new_primaries This allows shard allocation only for primary shards for new indices.
none This does not allow any shard allocations.
cluster.routing.allocation .node_concurrent_recoveries Numeric value (by default 2) This restricts the number of concurrent shard recovery.
cluster.routing.allocation .node_initial_primaries_recoveries Numeric value (by default 4) This restricts the number of parallel initial primary recoveries.
cluster.routing.allocation .same_shard.host Boolean value (by default false) This restricts the allocation of more than one replica of the same shard in the same physical node.
indices.recovery.concurrent _streams Numeric value (by default 3) This controls the number of open network streams per node at the time of shard recovery from peer shards.
indices.recovery.concurrent _small_file_streams Numeric value (by default 2) This controls the number of open streams per node for small files having size less than 5mb at the time of shard recovery.
cluster.routing.rebalance.enable
all This default value allows balancing for all kinds of shards.
primaries This allows shard balancing only for primary shards.
replicas This allows shard balancing only for replica shards.
none This does not allow any kind of shard balancing.
cluster.routing.allocation .allow_rebalance
always This default value always allows rebalancing.
indices_primaries _active This allows rebalancing when all primary shards in cluster are allocated.
Indices_all_active This allows rebalancing when all the primary and replica shards are allocated.
cluster.routing.allocation.cluster _concurrent_rebalance Numeric value (by default 2) This restricts the number of concurrent shard balancing in cluster.
cluster.routing.allocation .balance.shard Float value (by default 0.45f) This defines the weight factor for shards allocated on every node.
cluster.routing.allocation .balance.index Float value (by default 0.55f) This defines the ratio of the number of shards per index allocated on a specific node.
cluster.routing.allocation .balance.threshold Non negative float value (by default 1.0f) This is the minimum optimization value of operations that should be performed.

Disk-based Shard Allocation:

Setting Possible value Description
cluster.routing.allocation .disk.threshold_enabled Boolean value (by default true) This enables and disables disk allocation decider.
cluster.routing.allocation .disk.watermark.low String value (by default 85%) This denotes maximum usage of disk; after this point, no other shard can be allocated to that disk.
cluster.routing.allocation .disk.watermark.high String value (by default 90%) This denotes the maximum usage at the time of allocation; if this point is reached at the time of allocation, then Elasticsearch will allocate that shard to another disk.
cluster.info.update.interval String value (by default 30s) This is the interval between disk usages checkups.
cluster.routing.allocation .disk.include_relocations Boolean value (by default true) This decides whether to consider the shards currently being allocated, while calculating disk usage.
elasticsearch - elasticsearch tutorial - elastic search - elasticsearch sort - elasticsearch list indexes - elasticsearch node

Discovery:

  • This module helps a cluster to discover and maintain the state of all the nodes in it. The state of cluster changes when a node is added or deleted from a cluster. The cluster name setting is used to create logical difference between different clusters. There are some modules which help you to use the APIs provided by cloud vendors and those are −
  • Azure discovery
  • EC2 discovery
  • Google compute engine discovery
  • Zen discovery

Gateway

  • This module maintains the cluster state and the shard data across full cluster restarts. Following are the static settings of this module −
Setting Possible value Description
gateway.expected_ nodes numeric value (by default 0) The number of nodes that are expected to be in the cluster for the recovery of local shards.
gateway.expected_ master_nodes numeric value (by default 0) The number of master nodes that are expected to be in the cluster before start recovery.
gateway.expected_ data_nodes numeric value (by default 0) The number of data nodes expected in the cluster before start recovery.
gateway.recover_ after_time String value (by default 5m) This specifies the time for which the recovery process will wait to start regardless of the number of nodes joined in the cluster.
  • gateway.recover_ after_nodes
  • gateway.recover_after_ master_nodes
  • gateway.recover_after_ data_nodes

HTTP

  • This module manages the communication between HTTP client and Elasticsearch APIs.
  • This module can be disabled by changing the value of http.enabled to false. The following are the settings (configured in elasticsearch.yml) to control this module −
Sr.No Setting & Description
1

http.port

This is a port to access Elasticsearch and it ranges from 9200-9300.

2

http.publish_port

This port is for http clients and is also useful in case of firewall.

3

http.bind_host

This is a host address for http service.

4

http.publish_host

This is a host address for http client.

5

http.max_content_length

This is the maximum size of content in an http request. Its default value is 100mb.

6

http.max_initial_line_length

This is the maximum size of URL and its default value is 4kb.

7

http.max_header_size

This is the maximum http header size and its default value is 8kb.

8

http.compression

This enables or disables support for compression and its default value is false.

9

http.pipelinig

This enables or disables HTTP pipelining.

10

http.pipelining.max_events

This restricts the number of events to be queued before closing an HTTP request.

Indices

  • This module maintains the settings, which are set globally for every index. The following settings are mainly related to memory usage −

Circuit Breaker

  • This is used for preventing operation from causing an OutOfMemroyError.
  • The setting mainly restricts the JVM heap size.
  • For example, indices.breaker.total.limit setting, which defaults to 70% of JVM heap.

Fielddata Cache

  • This is used mainly when aggregating on a field.
  • It is recommended to have enough memory to allocate it.
  • The amount of memory used for the field data cache can be controlled using indices.fielddata.cache.size setting.
elasticsearch - elasticsearch tutorial - elastic search - elasticsearch sort - elasticsearch list indexes - elasticsearch node

Node Query Cache

  • This memory is used for caching the query results.
  • This cache uses Least Recently Used (LRU) eviction policy. Indices.queries.cahce.size setting controls the memory size of this cache.

Indexing Buffer

  • This buffer stores the newly created documents in the index and flushes them when the buffer is full.
  • Setting like indices.memory.index_buffer_size control the amount of heap allocated for this buffer.
elasticsearch - elasticsearch tutorial - elastic search - elasticsearch sort - elasticsearch list indexes - elasticsearch node

Shard Request Cache

  • This cache is used to store local search data for every shard.
  • Cache can be enabled during the creation of index or can be disabled by sending URL parameter.
Disable cache - ?request_cache = true
Enable cache "index.requests.cache.enable": true
Clicking "Copy Code" button will copy the code into the clipboard - memory. Please paste(Ctrl+V) it in your destination. The code will get pasted. Happy coding from Wikitechy - elasticsearch - elasticsearch tutorial - elastic - elastic search - elasticsearch docker team

Indices Recovery

  • It controls the resources during recovery process. The following are the settings −
Setting Default value
indices.recovery.concurrent_streams 3
indices.recovery.concurrent_small_file_streams 2
indices.recovery.file_chunk_size 512kb
indices.recovery.translog_ops 1000
indices.recovery.translog_size 512kb
indices.recovery.compress true
indices.recovery.max_bytes_per_sec 40mb
elasticsearch - elasticsearch tutorial - elastic search - elasticsearch sort - elasticsearch list indexes - elasticsearch node

TTL Interval

  • Time to Live (TTL) interval defines the time of a document, after which the document gets deleted. The following are the dynamic settings for controlling this process −
Setting Default value
indices.ttl.interval 60s
indices.ttl.bulk_size 1000

Node

  • Each node has an option to be data node or not.
  • You can change this property by changing node.data setting.
  • Setting the value as false defines that the node is not a data node.

The modules in this section are:

Cluster-level routing and shard allocation

  • Settings to control where, when, and how shards are allocated to nodes.
  • Discovery

  • How nodes discover each other to form a cluster.
  • Gateway

  • How many nodes need to join the cluster before recovery can start.
  • HTTP

  • Settings to control the HTTP REST interface.
  • Indices

  • Global index-related settings.
  • Network

  • Controls default network settings.
  • Node client

  • A Java node client joins the cluster, but doesn’t hold data or act as a master node.
  • Painless

  • A built-in scripting language for Elasticsearch that’s designed to be as secure as possible.
  • Plugins

  • Using plugins to extend Elasticsearch.
  • Scripting

  • Custom scripting available in Lucene Expressions, Groovy, Python, and Javascript. You can also write scripts in the built-in scripting language, Painless.
  • Snapshot/Restore

  • Backup your data with snapshot/restore.
  • Thread pools

  • Information about the dedicated thread pools used in Elasticsearch.
  • Transport

  • Configure the transport networking layer, used internally by Elasticsearch to communicate between nodes.
  • Tribe nodes

  • A tribe node joins one or more clusters and acts as a federated client across them.
  • Cross cluster Search

  • Cross cluster search enables executing search requests across more than one cluster without joining them and acts as a federated client across them.

  • This tutorial provides some of the very important topics such as elasticsearch types , elasticsearch 5 , elasticsearch github , elasticsearch ui , elasticsearch wiki , elasticsearch indices , elasticsearch cluster , elasticsearch port , elasticsearch docs , elasticsearch configuration , elasticsearch terms , elasticsearch mapping , elasticsearch head , elasticsearch rest api , elasticsearch filter , elasticsearch search , elastic search query , elasticsearch database , elasticsearch marvel , elasticsearch gui , elasticsearch match , elasticsearch install , elasticsearch query example , elasticsearch api , elasticsearch update , kibana plugins , solr vs elasticsearch , logstash elasticsearch , elasticsearch kibana , elasticsearch vs solr , elasticsearch head plugin , elasticsearch logstash , elasticsearch aws , aws elastic search , elasticsearch php , elasticsearch example , elasticsearch logstash kibana , download elasticsearch , kibana demo , elasticsearch documentation , elasticsearch version , what is elasticsearch , elasticsearch plugin , elasticsearch log , elasticsearch download , elastic co

    Related Searches to Elasticsearch - Modules