Elasticsearch Architecture: Inverted Indexes, Shards, and Near-Real-Time Search

Two D-pillar articles pre-paid this one’s framing, and both are debts to acknowledge. Database replication called this stack “a full stack in which every index is both sharded and replicated”: the sorting line this cluster drew, running as a product. Database sharding called it “a search engine built from sharding and replication as first-class design,” and data partitioning (this batch’s previous article) just named the concept underneath both. The when-to-add decision and the indexing pipeline are the design article’s; this one owns the machinery: what a shard actually contains, how writes become searchable within a second, and what a query pays to cross the fleet.

Elasticsearch architecture is the internal design of a distributed search engine: documents are indexed into shards spread across a cluster’s nodes, each shard is replicated for throughput and survival, each copy is built from immutable segments of inverted-index structures, and every query fans out to one copy of every shard before the answers are merged into one ranked list. It is the partitioning-plus-replication architecture of this cluster’s D-series, specialized for one job: making text queries fast and ranked.

The unit model: cluster, node, index, shard

The units, from the bottom up. A node is one engine instance on one machine; the stack runs on a Java virtual machine, with all the memory-tuning that implies. A cluster is a fleet of nodes that coordinates itself: master-eligible nodes elect a coordinator that owns the cluster’s shared state (shard placement, node membership) the leader election pattern wearing search-engine clothes. An index is the logical namespace (the database-shaped thing applications talk to) and it is divided into shards: each shard is a complete, self-contained search engine instance in its own right, built on the Lucene library, holding a slice of the index and answering queries against it alone.

The division is the cluster’s D-series architecture made first-class. A document lands in exactly one primary shard, routed by a hash of its identifier: the partitioning rule, data partitioning‘s hash strategy applied at write time. Each primary shard has replica copies, the replication half of the sorting line; distributed so that no node holds a primary and its replica together. The result is the exact shape the cluster has been assembling since the replication article: every index partitioned and every partition replicated, with placement owned by the engine instead of bolted on around it.

Above the shards sits the mapping (the index’s schema) and it deserves its place in the unit model because it is the boundary of what can change. The mapping declares each field’s type: text, analyzed for relevance; keyword, matched exactly and aggregated; numbers, dates, and the rest. The analysis chain the design article covered runs on the text fields, and the mapping is where that chain is attached. The honest rule is the one the design article set for analysis: mapping changes that would reprocess stored documents are reindex changes (a field’s type, once written, is what its segments hold) so the mapping is versioned and deliberate, exactly like the schema it replaced.

The inverted index: sorted copies pointed at words

Inside every shard, the searchable structure is the inverted index; two coordinated structures that are exactly the sorted-copy idea database indexing teaches, pointed at words. The term dictionary is the sorted vocabulary of every token in the shard; the same balanced, sorted structure a B-tree index uses, answering “does this term exist, and where does its list begin.” The postings lists hang off it: for each term, a sorted list of the documents containing it, with per-document frequencies and positions. A query looks up its terms in the dictionary and merges their postings; sorted document lists intersect like sorted anything, which is why the sorting is the point.

The postings answer “which documents,” but search must also rank and facet, so a second structure completes the shard. Doc values store per-document fields column-wise (the sorting keys, the category tags, the price bands) so ranking by field and counting by facet read straight sequences instead of decoding documents. Scoring leans on term statistics, the rare-terms-count-more intuition the design article introduced, and the subtlety this stack adds is statistical: the frequencies are shard-local by default, so a term rare in one shard scores differently than the same term rare across the index; the reason a global-statistics pass exists for the queries where it matters, and a hint of the fan-out costs the distributed section prices.

The positions in the postings earn their keep on the queries a word-bag cannot answer. “Edge network” and “network edge” contain the same terms and mean different things; a phrase or proximity query resolves them by checking whether the terms sit near each other in the document, and that check is what the position lists are for. The price is weight (positions make the postings heavier) and the decision is per-field: analyzed text carries them, exact-match keywords usually do not. The same per-field economics governs the whole structure, and the memory budget is where it surfaces: the dictionary’s pages live in the operating system’s page cache in a well-sized cluster, which is why heap-versus-cache is the first tuning conversation every deployment has.

Segments: immutability and the refresh

A write, followed end to end, explains the near-real-time clock. The document routes to its primary shard, where it lands in two places at once: an in-memory indexing buffer, and the translog; the shard’s write-ahead log, the same ordered-log discipline replication uses, pointed at durability. The write is acknowledged when the primary and its replicas have both applied it; synchronous replication by default, because an acknowledged write that vanishes on the next node crash is not a search engine’s idea of an acknowledgment. But none of that makes the document searchable yet; visibility runs on a different clock.

That clock is the refresh: periodically (by default about once a second) the in-memory buffer is turned into a new segment, a small, immutable inverted index, and the segment is opened for search. From one refresh to the next, writes are durable but invisible; after it, they are visible to every query against the shard. This is the precise meaning of near real time search: not “instant,” but “visibility latency bounded by a tunable interval,” with durability already guaranteed behind it by the translog. The two guarantees run independently, and confusing them is the classic deployment surprise; a team that raises the refresh interval to save resources has traded visibility latency, not durability, and the recovery story says so.

Immutability is the design choice underneath the whole clock. A segment, once written, never changes, so reads need no locks against writes, the operating system’s page cache can hold segments without coordination, and concurrency becomes a solved problem. The price is that updates and deletes are new versions and tombstones rather than edits: the old document stays in its segment, hidden by the tombstone, until a background merge folds small segments into larger ones and physically applies the deletions, the moment storage reclaims. The flush completes the lifecycle: segments are persisted to long-term storage and the translog is cleared, the durable point between refreshes. Immutability is why the refresh is cheap, why merges exist, and why this engine’s storage story is a ladder of structures rather than a mutable heap.

The knobs behind the clock deserve their honest summary, because they are the first thing a team tunes. A longer refresh interval means fewer, larger segments, less merge churn, and faster indexing: paid for in visibility latency, never in durability. The translog’s durability settings decide what a hard crash costs in replay time, with the same fsync trade redis architecture prices for its own append log. And the merge policy decides when small segments fold into large ones, usually left alone, occasionally tuned for write-heavy indexing bursts. Near-real-time is not a flaw being apologized for; it is a dial with a price tag, and the tag is visible.

Shards and replicas: partitioning and replication as first-class design

The numbers are set differently, and the asymmetry is the most operational fact in this stack. An index is created with a fixed count of primary shards, and it is fixed because the routing rule is a hash of the document ID modulo that count: change the count, and every document’s home changes, the resharding bill database sharding priced in full. This engine’s answer to that bill is the design article’s blue-green pattern: build a new index with the new count, reindex from the source, and cut queries over. Replica counts, by contrast, adjust on a running cluster; replicas exist per primary, and adding one is the snapshot-free, catch-up-heavy routine the replication article documented, because the engine performs the catch-up itself.

What each number buys is the cluster’s sorting line again. Primaries buy parallelism and capacity: the index’s data is partitioned across them, and a query touches all of them concurrently, the fan-out the next section prices. Replicas buy throughput and survival: each is a full copy that serves reads alongside the primary, and when a node dies, the surviving replicas are promoted per-shard: the partitioned-and-replicated composition of the partitioning article, running as a product feature. The honest sizing follows: too few primaries caps parallelism and per-node data volume; too many taxes every query with fan-out to tiny shards and pays segment overhead in each. The famous mistake is picking a large number early out of caution; the correct size comes from the data volume per node and the write rate, not from a default.

Node loss is where the two numbers cash out, and the cluster’s health vocabulary describes it. A node dies: its shard copies are gone, and the cluster re-creates them elsewhere; replicas promote to primary for the shards it led, new replicas catch up from survivors, and the cluster is briefly degraded rather than broken, a state the health reporting marks until every primary has a full replica set again. The unrecoverable case is the one the sizing guidance exists to prevent: a primary lost with no surviving replica means data loss and an honest red status; the partitioned-and-replicated design survives N-1 failures per shard, and the sizing decides how much of the fleet N-1 means.

The distributed query: scatter-gather

Every search fans out. The node a client reaches becomes the coordinating node for that request, and the query scatters to one copy of each shard: a primary or a replica, rotated for load. Each shard runs the query against its local segments and returns its top matches: identifiers, scores, and sort keys, not documents. The coordinating node gathers these ranked lists, merges them into one ranking, and then a fetch phase retrieves the actual documents from the shards that own them. The response’s latency is the slowest shard’s response (the tail-latency discipline again, latency budgets applied to a fan-out) and one slow or missing shard is either waited on or failed over to its replica, depending on the consistency the request declares.

The shape has the costs the sharding article taught at the database level, plus two search-specific ones. The statistics hint from the inverted-index section lands here: scores are computed per-shard by default, and the engine offers a global-statistics round for the queries where the skew matters. And deep pagination is the pathological case: asking for the thousandth page forces every shard to rank everything up to that depth, because each holds only its share of the answer; the honest pagination is cursor-style, walking the ranking forward rather than jumping into it. None of these costs are secrets; they are the standing price of running the ranking across a partitioned fleet, and the design article’s pipeline is what keeps the fleet worth querying.

Aggregations walk the same path with the same fine print. Each shard computes partial aggregates (term counts, metric sums, percentile sketches) and the coordinating node merges them into the final answer, which is why facet counts appear in milliseconds over billions of documents and why, on large multi-shard indexes, the counts arrive with a statistical caveat: partials merged across shards are approximations unless the query pays for the exact round. The faceted navigation the design article promised is real and fast; the engineering honesty is knowing which of those numbers are exact and which are estimates wearing a chart.

FAQ

What is Elasticsearch architecture?

A distributed search engine: an index divided into shards across a cluster’s nodes, each shard a self-contained engine built on immutable inverted-index segments, each shard replicated for throughput and failover, with writes made durable by a per-shard log and made visible by a once-a-second refresh, and every query fanned out to all shards and merged into one ranking.

What is an inverted index?

A sorted vocabulary of terms (the term dictionary) with, per term, a sorted postings list of the documents containing it. Queries look up their terms and merge the lists, the same sorted-copy logic a database index applies to column values, pointed at words. Rankings and facets lean on term frequencies and columnar per-document values beside it.

Why is it “near real time” rather than real time?

Because visibility runs on a separate clock from durability. A write is durable the moment the translog records it, but searchable only after the next refresh turns the in-memory buffer into a segment, by default about a second later. The interval is tunable: raising it trades visibility latency for indexing efficiency, and never touches the durability guarantee.

How are shards and replicas configured?

Primary-shard count is fixed at index creation (the routing hash depends on it) and changing it means reindexing into a new index with the new count. Replica count is adjustable on a running cluster, trading storage and indexing work for read throughput and node-loss survival. The counts are sized from data volume per node and write rate, not from caution.

Can you change the number of primary shards later?

Not in place: the placement rule is a hash of the document ID modulo the shard count, so a different count rehomes every document. The supported path is the blue-green one (new index, reindex from the source, verify counts, cut over) which is the same pattern the search design article uses for drift recovery, because it is the same operation: a derived structure rebuilt from the truth.

Why does search get slower as the index grows?

Usually for one of three reasons, each visible in this article’s machinery: the shard count is too low for the data volume, so each fan-out target does more work; segment count has grown past the merge policy’s pace, so every query checks more structures; or the working set has left the page cache, so queries start reading disk. All three are capacity questions, not fate; the first two are reshaped by the reindex and tuning paths above, and the third is the memory budget the sizing section set out.

Related articles

• Next read: kafka architecture; the batch’s other deep dive: the durable log, partitioned and replicated, that carries the event streams feeding systems like this one.
• search engines in system design; the when and how: the decision, the pipeline, and the consistency contract this machinery serves.
• database indexing, the sorted-copy ancestor of the inverted index.
• database sharding: the fan-out, hot-partition, and resharding economics this engine inherits from the partitioned shape.
• database replication; the log, catch-up, and promotion mechanics each shard pair runs.
• data partitioning, the umbrella concept this stack implements as first-class design.