Data Partitioning: Vertical, Horizontal, and the Sharding Question

The cluster left this article its vocabulary debts in writing. Database replication called partitioning “the general concept underneath sharding, and the vocabulary around it.” Database sharding named it “the umbrella concept” with its vertical and horizontal forms. The sharding vs replication comparison drew the line this article answers to: “a partitioning scheme decides which machine holds a row; a replication scheme decides how many machines hold it”, and then handed the full taxonomy, including the partitioning-versus-sharding boundary, to this page. Consistent hashing added the placement angle: sharding is “the partitioning question, specialized.” This article pays the debts in order: the umbrella, the two axes, the vocabulary, and the strategies.

Data partitioning is the general act of dividing a dataset into parts, by rows, by columns, or by rule, so that each part can be stored, queried, or managed independently of the others. The division can happen inside one machine, where partitions buy locality and manageability, or across many, where it becomes the foundation of horizontal scale. Partitioning is what every mechanism in this cluster finally does; the differences are in the axis, the rule, and whether the parts live on one machine or a fleet.

The umbrella: what dividing a dataset buys

Partitioning buys four things, and the four explain why every mechanism in this cluster eventually partitions something. Locality: queries touch a partition instead of the whole dataset, so the scan or the index search happens in a small structure: the same logic database indexing applies at the column level, applied now at the storage level. Manageability: a partition can be backed up, archived, migrated, or dropped on its own schedule: bulk operations become per-partition operations, and the maintenance windows get small. Parallelism: independent parts can be read or written concurrently, on one machine’s cores or across a fleet. And scale: a dataset divided across units can exceed any one unit; the ceiling only sharding moves, but the concept that makes it movable is division itself.

The buys have prices, and the prices should be listed once at the umbrella level because every specialized article below pays them. Cross-partition questions: anything the partition rule cannot answer locally becomes fan-out, and the transaction boundary stops at the partition edge; a change spanning two partitions is no longer a local atomic operation, the boundary the cluster’s transactions article exists to price. Rebalancing: the rule that divided the data must also be able to move it, and the machinery for that ranges from an engine feature to a fleet project. And key permanence: the partition rule gets baked into data placement, so changing the rule later means moving everything that was placed by the old one. The four buys are real; none of them is free, and the strategies section prices them by name.

The sorting line the comparison article drew deserves its permanent place at the top of the umbrella, because every vocabulary confusion downstream traces back to it: a partitioning scheme decides which unit holds a row; a replication scheme decides how many units hold it. Partitioning is about division; replication is about copies; and the two compose (every shard can be replicated, every replica-set can be sharded) without ever being the same decision. The cluster’s articles on replication and sharding own the mechanisms; this article owns the concept both of them stand on.

Vertical partitioning: splitting by columns

Vertical partitioning divides by columns: the schema’s attributes, not its records. The wide table’s rarely-read columns move to their own structure: the product description’s full text, the user’s binary avatar, the audit detail nobody queries, leaving the hot, narrow core (the columns every request actually touches) compact enough to sit in memory. The read pays one extra hop when it needs the cold part, and nothing at all when it does not; the working set shrinks by exactly the width of what moved. It is normalization’s cousin at the storage layer: same instinct (group by access pattern) without the relational algebra.

The file-tier connection is direct, and this cluster already built it. The extreme vertical partition is moving a column out of the database entirely: big blobs belong in object storage with a key in the row, which is vertical partitioning across tiers, the bytes partitioned right out of the table. The same rule of thumb governs both decisions: partition by access pattern, so the hot core stays hot and the cold bulk stops taxing the buffer pool, the log, and the backups with every write.

One vocabulary trap needs disarming, because the words collide: vertical partitioning is not vertical scaling. Vertical scaling makes one machine bigger (more RAM, more cores) and is the opposite axis of growth from dividing data; vertical partitioning divides a table’s columns, which can happen on one machine and has nothing to do with the machine’s size. The two share a word because both slice vertically on a diagram; they share nothing else, and conflating them produces design documents that talk past their readers.

What to move is an access-pattern audit, not a size sort. Log the queries that matter, group the columns by how often they are read together, and the split lines appear at the rare boundaries: the audit blob read by one screen in a hundred, the historical detail nobody joined in months. The warning is the mirror of the win: columns that are almost always read together should never be split, because vertical partitioning turns their single fetch into two fetches plus a join in application code. Split along the cold seams, and only along them.

Horizontal partitioning: splitting by rows

Horizontal partitioning divides by records; the rows distribute across partitions under a rule, and each partition holds the same schema with a subset of the data. The rule is the design decision everything else inherits: by ranges of a key (dates, IDs, geographic regions), by a hash of it (spreading evenly at the cost of range queries), by an explicit list (this tenant here, that product line there), or by placement machinery that decides and revises. The mechanics (the strategies in depth, the hot-partition problem, the rebalancing bill) are sharding’s to own; the point this article adds is that all of it is horizontal partitioning plus machinery, and the vocabulary should say so.

The single-machine form deserves its own paragraph, because it is where the wins are cheapest and most often skipped. A table partitioned by month answers a month’s query by pruning every other partition from the plan; the scan is a fraction of the dataset by construction. Retention becomes a drop: when the partition holding March is past its retention window, the archive operation detaches one structure instead of deleting millions of rows through the write path and the log. And each partition carries its own indexes, small and focused, the indexing article’s structures scaled to their slice. None of this needs a second machine; it needs a rule, declared to the engine, that the data will follow.

What horizontal partitioning fixes is the row-count ceiling: scans, index depth, memory for the working set, and write parallelism all stop growing with the whole dataset and start growing with one partition’s share. What it costs is the cross-partition question: any query without the partition key becomes fan-out (partial results from every partition, merged before the answer) the scatter-gather and tail-latency cost the sharding article priced in full. The two costs are the whole decision, and the cluster has already made it once at the mechanism level: sharding vs replication sorted when each axis of growth needs which answer.

Partitioning vs sharding: the vocabulary, sorted

The two words are used interchangeably and should not be, and the sorting is this article’s to give. Partitioning is the general concept: a dataset divided into parts by a rule, wherever those parts live, including inside a single database, where modern engines partition tables declaratively, pruning a query to the partitions it could possibly touch and archiving old partitions by dropping them. A partition, in the general sense, is a logical division: a container of rows, defined by rule, addressable on its own. Sharding is the specialization: horizontal partitioning where the parts live on different machines, with placement machinery added; a routing rule that gets each request to the right machine, and a rebalancing story for when the fleet changes. Every shard is a partition; not every partition is a shard.

The practical distinction is what changes when the boundary is crossed. A partition inside one machine needs no routing (the engine addresses it by rule) and rebalancing is a schema-level operation the engine runs. A shard across machines needs the whole fleet apparatus: the shard key carried by every hot query, the coordinator or client that routes, and the resharding project the sharding article called the heaviest routine task a fleet performs. The two also fail differently: a bad partition strategy wastes a machine’s memory; a bad shard strategy wastes a fleet’s latency, the cross-shard fan-out priced where the queries cannot follow the key.

The vocabulary extends one more step, and the cluster has already used it: partitioning and replication compose. A dataset partitioned three ways and replicated twice is six physical copies in three logical homes; the partition scheme still decides which home holds a row, and the replication scheme still decides how many copies of that home exist. The sorting survives composition, which is the test that it is a real sorting and not a mnemonic, and it is the architecture every sharded-and-replicated system in this cluster, up to and including the search stack this batch’s next article dissects, actually runs.

Partitioning strategies: choosing the rule

The strategies are few, and each optimizes a different shape of access. Range partitioning divides by ordered key intervals (dates are the canonical case) and it serves range queries by pruning to exactly the partitions the interval touches; its failure mode is the hotspot at the active edge, the newest range absorbing all the writes, the monotonic-key trap the sharding article named. Hash partitioning spreads keys evenly by applying a hash to the key, buying balance and giving up order; a range query becomes every-partition fan-out, and the placement becomes a ring problem, the one consistent hashing exists to solve: membership changes moving a small share of keys instead of all of them.

List partitioning assigns by explicit membership (this partition holds these tenants, that partition holds that region) and it is the strategy behind the requirements that sound nothing like performance: regulatory locality, where data must live in the jurisdiction that created it, is list partitioning wearing a legal constraint; tenant isolation, where one customer’s traffic must not sit on the same units as another’s noisy neighbor, is list partitioning wearing an operational one. And directory partitioning keeps an explicit map from key to partition; the most flexible rule and the most machinery, buying arbitrary placement (migrations become map edits) at the price of a component that must be available, consistent, and honest.

Choosing among them is the same discipline this cluster has applied everywhere else: index to the access pattern. A workload whose queries are time-shaped earns range partitioning and pays the hotspot with jittered writes or a pre-split active range. A workload whose queries are key-shaped and uniform earns hashing and never sees its own boundaries. A workload whose partitions are organizational or legal earns lists and directories, and pays in the machinery those imply. The strategy that fits is the one whose failure mode the workload can afford.

Strategies compose, and skew is the standing enemy. Real systems run compound rules (hash by tenant to spread, range by time inside each tenant’s slice for pruning and retention) combining balance from the hash with the lifecycle and range-query wins from the range. Whatever the rule, the partitions must be watched for skew: a partition growing faster than its siblings, or absorbing a disproportionate share of the traffic, means the rule has met a workload it did not model: a new whale tenant, a viral region, a monotonic edge. The fix menu is short; split the hot partition, add it to its own list, or rehash the namespace, and every item on it is cheaper the earlier the skew is caught, which is why per-partition size and rate belong on the dashboard next to the lag metrics the rest of this cluster already monitors.

FAQ

What is data partitioning?

Dividing a dataset into parts by a rule (rows or columns, ranges or hashes or lists) so each part can be stored, queried, and managed independently. It is the general concept under sharding and several single-node database features; the division buys locality, manageability, parallelism, and, when the parts span machines, scale past one unit.

What is the difference between partitioning and sharding?

Scope and machinery. Partitioning is any rule-based division of the data, including inside a single database; sharding is horizontal partitioning across machines, with placement machinery added; routing that finds the right machine, and rebalancing that moves the boundaries. Every shard is a partition; not every partition is a shard, and only shards need the fleet apparatus.

What is the difference between horizontal and vertical partitioning?

Rows versus columns. Horizontal partitioning splits the records under a rule, so each partition holds a subset of the rows with the full schema; vertical partitioning splits the attributes, so a hot narrow core separates from cold wide columns, at the extreme, blobs moving out of the table entirely. Most scale conversations are about the horizontal form; most working-set wins come from the vertical one.

Can you partition within a single machine?

Yes: modern engines partition tables declaratively inside one database, pruning queries to the relevant partitions and turning archive operations into partition drops. It buys manageability and locality without any routing, and it is the honest first step before sharding: the division is designed and proven while the parts still share a machine.

How does partitioning relate to replication?

They compose and never substitute. Partitioning decides which unit holds a row; replication decides how many units hold it. A three-way partitioned, twice-replicated dataset is six copies in three homes, and every sharded-and-replicated system in this cluster, caching fleets included, is exactly that shape wearing different names.

What is partition pruning?

The engine’s trick of excluding partitions from a query plan because the rule proves they cannot match: a date-ranged query touches only the partitions whose ranges intersect it, and the rest of the dataset never enters the plan. Pruning is the read-side payoff of declaring a partition rule to the engine; the same rule that makes writes land in the right partition makes reads skip every other one.

Related articles

• Next read: Elasticsearch architecture; the concept running in one system: an engine where every index is both partitioned and replicated, first-class.
• database sharding; the specialization this article keeps deferring to: strategies in depth, hot shards, resharding, the fleet apparatus.
• sharding vs replication, the decision article that drew the sorting line this page canonizes.
• consistent hashing; the placement scheme hash partitioning leans on, and the reason membership changes stay cheap. The scheme is why a rebalance moves a sliver instead of the whole key space.
• database replication, the other half of the sorting: copies, lag, and the how-many-units question. Sharding splits the writes; replication copies them, and the two compose.
• blob and object storage; vertical partitioning’s extreme: the column that moves out of the table entirely.