Elasticsearch is often described as a search engine. The accurate sentence is that it is a distributed system for Lucene indexes. Each shard is a Lucene index that can live on a node, with replica shards for reads and for a copy if the primary dies. Queries fan out to shards, merge top-N, and return. Aggregations do a similar scatter-gather. This is why Elasticsearch can search terabytes, and why a bad shard plan can make a 3-node cluster crawl while CPU looks "fine" on average.
Primaries, replicas, and refresh
A document write goes to a primary shard (hashed by _id unless you route), then to replicas. Lucene's search is near-real-time: a refresh makes new docs visible, at a cost. The default one-second refresh is a product choice. Logging clusters often relax it; search-as-you-type cannot.
Replica count is both availability and read throughput. Replicas are not a backup: they are extra copies of the same failure if you delete by query. Snapshots to object storage are the backup. Cluster state (mappings, shard assignment) lives in a small set of master-eligible nodes. If you make every node master-eligible on a huge cluster, you will suffer. If you have two masters, you will split.
Shard math is capacity planning
Too few shards: a shard is huge, recoveries take hours, a node holding a hot shard is a hotspot. Too many shards: cluster state bloats, every search fans out to hundreds of tiny Lucene indexes, heap dies on overhead. Time-based indexes (one per day) plus 5 shards each plus 50 retention days is a shard explosion that Elastic has spent years warning about. Shrink, rollover, and data streams exist because humans will not pick the right number on day one and then never change it.
Mappings are a schema. Dynamic mapping that creates a new field per log line of JSON will explode the mapping and then the cluster. Explicit mappings are operational hygiene.
Failure modes of Lucene-at-scale
The concrete failure is a yellow cluster after a node death because there is no room to allocate replicas, while search still "works" on primaries until the next death. Mid-size steal: disk watermarks, awareness (zone), and capacity that includes a node out.
Operational gotcha: search.max_buckets and unbounded aggregations that OOM a data node. Another is a refresh-heavy indexing pipeline on the same nodes that serve aggregations. Split hot ingest and cold search if you must. Deep pagination with from+size is a quadratic tax; use search_after. Script queries in production are a CPU sink. Heap vs mmap for Lucene files: under-heap and you GC; over-heap and you have no page cache. Follow current Elastic guidance for your version; folklore from 2015 is harmful. Reindexing after a mapping mistake is a migration project. Test mappings on a sample. If you use Elasticsearch as a primary store, you will learn about Lucene's delayed durability and version conflicts. Many teams should not. Steal shard-count dashboards and the idea that the unit of failure is a shard, not a node.
What you can borrow
- Choose shard counts for growth and recovery time; too many shards is as bad as too few.
- Treat replicas as HA and read scale, snapshots as backup.
- Freeze mappings; dynamic fields from logs will bury the cluster.
- Split ingest-heavy and query-heavy workloads when they fight for the same heap.