The CAP theorem states that a distributed system cannot simultaneously guarantee consistency, availability, and partition tolerance. In practice, network partitions happen, so the real choice is between consistency and availability. For a financial transaction database, this is not a difficult choice, but it is worth understanding what each option actually means operationally.
What Choosing Availability Means for a Financial System ¶
A system that prioritizes availability during a partition continues accepting writes on both sides of the partition. When the partition heals, the two sides must be reconciled. For a financial ledger, reconciliation means resolving conflicting transaction histories, which may include transactions that both sides accepted as committed but that are mutually inconsistent. This is the problem that application-level reconciliation jobs exist to solve.
What Choosing Consistency Means Operationally ¶
A system that prioritizes consistency during a partition stops accepting writes on the side that cannot confirm replication. Reads from local replicas may remain available. When the partition heals, the system resumes writes automatically. The operational consequence is a write outage for the duration of the partition. For most network partitions in a well-maintained Japanese data center environment, that duration is measured in seconds to minutes, not hours.
Grit Core Lab's Partition Behavior ¶
Grit Core Lab follows the CP side of the CAP theorem. If the leader cannot confirm replication to a follower in the other region within the configured timeout (default: 500 ms), it stops accepting writes. Reads from local replicas remain available. The leader resumes writes automatically when replication is confirmed. This behavior is not configurable: a financial transaction database that can be set to accept writes without replication confirmation is not a safe choice for a settlement ledger.
Monitoring Partition Events ¶
Grit Core Lab exposes a metrics endpoint (compatible with Prometheus) that reports replication lag, leader election events, and write suspension events. A write suspension event indicates that the system has stopped accepting writes due to a replication timeout. Alerting on this metric gives the operations team immediate visibility into partition events without requiring manual log inspection.
The technical questions section covers the specific behavior of Grit Core Lab during a network partition between the Osaka and Tokyo regions, including the recovery sequence after the partition heals.