Split Brain

Split brain happens when nodes in a cluster lose contact with each other but stay up, and each side concludes the other is dead. Both promote themselves to primary and keep accepting writes, producing two conflicting versions of the truth.

Why it matters

It is one of the nastiest outage types because everything looks healthy from each side. Reconciling divergent data can mean choosing one side and discarding the other's writes, which is real data loss. The standard defenses are a quorum rule (only the side with a majority may act) and fencing, which forcibly stops the old primary from touching shared resources.

Example

A three-node database cluster splits so that node A is isolated from nodes B and C:

A | B C

With a majority quorum of 2 of 3, B and C elect a leader and continue. Node A cannot reach a majority, so it must stop accepting writes. In a two-node cluster with no tiebreaker, neither side can ever prove it has a majority, which is why two-node setups need a witness.

Common mistakes

• Running an even number of voting nodes with no tiebreaker.
• Automatic failover without fencing, so the old primary keeps writing after promotion.
• Assuming a ping from the monitor proves the node is reachable by its peers.

Related

• Quorum
• Failover
• Active-passive
• CAP theorem

This page is general information written by our editorial team and updated when documentation changes. Servers and providers differ, so test changes safely first. See our editorial policy and disclaimer.