Elasticsearch Integration

Monitor Elasticsearch cluster health, nodes, indices, shards, search performance and indexing performance from a single Watchlog agent.

You point the agent at one reachable endpoint. It discovers the rest of the cluster — every node, its roles and data tiers, every index and every shard — from the cluster itself. A ten-node cluster is configured exactly like a single-node one.

Supported Platforms

• Ubuntu
• Windows
• Source Code
• Docker Container
• Kubernetes

Supported Versions

Version differences are handled by capability detection rather than assumption. A field that does not exist on your version is reported as unavailable, never as a zero.

Required Privileges

Watchlog is read-only. Create a dedicated monitoring user with the narrowest privileges that cover what you want to see:

{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["*"],
      "privileges": ["monitor"]
    }
  ]
}

In Kibana, the built-in monitoring_user role already grants exactly this.

Administrator privileges are not required. If the user cannot read a particular API, only that section is disabled — the integration stays connected and everything else keeps working:

Authentication

Three modes are supported:

If both an API key and a username are configured, the API key wins.

TLS

For an https endpoint, certificate verification is on by default. Two knobs change that:

What Watchlog Collects

Cluster Health status (green / yellow / red), node and data-node counts, active, relocating, initializing and unassigned shards, unassigned primary shards, pending cluster tasks and how long they waited, active shard percentage, document and index counts, store size, elected master.

Nodes JVM heap used / committed / max and heap percentage, non-heap memory, thread and peak thread counts, garbage collection counts and pause time (young and old generation, plus GC time as a share of wall-clock time), process and host CPU, load averages, memory, swap, open and maximum file descriptors, per-data-path filesystem usage, transport and HTTP connections, uptime and restart detection.

Search and indexing Queries and fetches per second, query / fetch / scroll / suggest latency, concurrent searches and open contexts, GET rate, latency and miss ratio, documents indexed and deleted per second, indexing latency, indexing failures, indexing throttling.

Storage internals Segment counts and memory, merge rate, throughput and throttling, refresh and flush rate and latency, translog size and uncommitted operations, query cache and request cache hit rates, fielddata memory and evictions.

Thread pools and circuit breakers Every pool the node reports — search, write, get, management and the rest — with threads, queue, active, rejected and completed, plus saturation state. Every circuit breaker with estimated size, limit, utilisation and trips during the period.

Indices Per index: health, documents, deleted documents and ratio, store size, primary size, shard and replica counts, search and indexing rate and latency, segments, merge activity, refresh and flush, translog, cache hit rates, and growth over the selected range.

Shards Every shard copy with index, shard number, primary or replica, state, node, relocation target, documents and size. Unassigned shards carry Elasticsearch's own reason and, where available, a cached allocation explanation translated into a plain recommendation.

Cluster management Pending cluster tasks with priority and queue time, long-running tasks (reindex, update-by-query, delete-by-query, snapshot), and shard recovery with stage, files, bytes and progress.

Health Score and Insights

Watchlog computes a 0–100 health score server-side, with a written reason attached to every deduction. The judgements are deliberately context-aware, because the naive versions are wrong often enough to be useless:

• A yellow cluster is not automatically a problem. A cluster with one data node cannot allocate a replica anywhere, so yellow is its permanent, correct state — that costs 2 points, not 25. Only unassigned primary shards mean data is actually unavailable.
• Disk is judged against your own configured watermarks, read from cluster.routing.allocation.disk.watermark.*, not a hardcoded 85%. A cluster that moved flood stage to 97% is not warned at 90%.
• Heap is judged on sustained pressure, not a single spike — a garbage collection at the moment of sampling is normal.
• A thread-pool rejection outranks a busy moment, and an actual circuit-breaker trip outranks elevated utilisation. Busy threads mean the cluster is working; a rejection means it dropped a user's request.

Collection Intervals

Not every API runs every minute. Expensive calls are throttled so the agent stays light on a large cluster:

All of these are configurable. See Advanced Options.

Safety

This integration is monitoring-only, and that is enforced structurally rather than by convention: the Elasticsearch client accepts only GET and the single diagnostic POST that Cluster Allocation Explain requires. There is no code path through which it can express a mutation.

Watchlog never runs any of the following, on your behalf or otherwise:

DELETE or PUT of anything · _cluster/reroute · task cancellation · close or open index · _forcemerge · _cache/clear · _flush · cluster settings updates · index settings updates (including enabling a slow log) · replica count changes · watermark changes

GET /_cluster/state is never called at all — on a large cluster it is the classic way to knock over the master with monitoring.

Recommendations in the dashboard tell you what to change. You make the change.

Slow Operations

Elasticsearch writes slow searches and slow indexing operations to log files on each node's host, not to an API — and only when you have configured a threshold.

Watchlog never enables slow logging for you. It has a real cost, and the decision is yours. If no threshold is configured anywhere, the Slow Operations tab says so and tells you what to set.

To enable it, on the indices you actually want traced:

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "10s"
}

Then enable collection in the agent (see the platform guides). The agent must be running on a host that can read the node's log directory.

Data protection

Slow logs are the one place in this integration where customer data can appear, so they are treated as untrusted text:

• Query text is scrubbed on your host, before it leaves it. Values under credential-shaped keys (password, api_key, token, authorization, …) are replaced, not truncated — a prefix of an API key is still a fact about the key. Credential-shaped values are also removed wherever they appear: JWTs, bearer tokens, AWS access keys, PEM blocks, and connection strings with inline credentials.
• Document _source is not stored by default. A document body is customer data by definition. Watchlog keeps only the fact that a body existed and how large it was — an oversized document explains a slow index operation. Setting storeSource: true is opt-in, and even then the content is scrubbed and truncated hard.
• Document ids are hashed unless they are opaque. A UUID or an Elasticsearch auto-id is kept; an email address used as an _id is replaced with a stable digest.

Advanced Options

Every field below is optional. The defaults are what most clusters should run.

{
  "service": "elasticsearch",
  "monitor": true,
  "protocol": "https",
  "host": "es.internal",
  "port": "9200",
  "username": "watchlog_monitor",
  "password": "your_password",
  "apiKey": "",
  "verifyCertificate": true,
  "tls": { "ca": "/etc/ssl/certs/es-ca.pem" },
  "requestTimeoutMs": 15000,

  "advanced": {
    "enabled": true,
    "indices": true,
    "shards": true,
    "clusterStats": true,
    "pendingTasks": true,
    "recovery": true,
    "tasks": true,
    "dataStreams": true,
    "allocationExplain": true,

    "maxIndices": 200,
    "maxIndicesPerPayload": 200,
    "maxShards": 2000,
    "maxNodes": 200,

    "indexIntervalSeconds": 300,
    "shardIntervalSeconds": 300,
    "clusterStatsIntervalSeconds": 300,
    "metadataIntervalSeconds": 900,
    "recoveryIntervalSeconds": 300,
    "taskIntervalSeconds": 600,

    "slowSearchThresholdMs": 500,
    "slowIndexingThresholdMs": 500
  },

  "slowlog": {
    "enabled": false,
    "searchLogPath": "",
    "indexingLogPath": "",
    "logDirectories": [],
    "minDurationMs": 0,
    "maxEntriesPerCollection": 200,
    "maxQueryLength": 2000,
    "maxSourceLength": 512,
    "storeSource": false
  }
}

Large clusters

On a cluster with thousands of indices, three settings matter:

• maxIndices — how many indices carry detailed statistics. Every index still appears in the list with its size, health and document count; this caps how many get full metrics. Indices are ranked by store size.
• maxIndicesPerPayload — indices beyond this are sent as follow-up batches rather than one oversized payload.
• maxShards — caps the shard listing. Unassigned, initializing and relocating shards are always kept first; started shards are what gets truncated.

Raising indexIntervalSeconds and shardIntervalSeconds is the other lever if the agent is adding measurable load.

Multiple Clusters

Add one entry per cluster. Each is identified by its cluster UUID, so the same cluster reached through a different endpoint stays one integration, and a cluster that is rebuilt from scratch becomes a new one.

[
  {
    "service": "elasticsearch",
    "monitor": true,
    "host": "es-prod.internal",
    "port": "9200",
    "username": "watchlog_monitor",
    "password": "your_password"
  },
  {
    "service": "elasticsearch",
    "monitor": true,
    "protocol": "https",
    "host": "es-analytics.internal",
    "port": "9243",
    "apiKey": "VnVhQ2ZHY0JDZGJrUW0tZTVhT3g6dWkybHAyYXhUTm1zeWFrdzl0dk5udw=="
  }
]

Troubleshooting