Crash recovery time is one of the least-discussed performance characteristics of a database, and one of the most consequential for financial systems. A database that takes ten minutes to recover after a node failure is unavailable for ten minutes. For a settlement system processing time-sensitive transactions, that is not acceptable.

Why Recovery Time Is Often Unbounded in Practice

Many databases recover by replaying the write-ahead log from the last checkpoint. If the checkpoint interval is long, or if the write volume between checkpoints is high, the replay can take minutes. Some databases also perform consistency checks on the data files during recovery, which adds time proportional to the size of the database. Neither of these properties is bounded in the general case: recovery time grows with write volume and database size.

How Grit Core Lab Bounds Recovery to Under 800 ms

Grit Core Lab uses a continuous checkpointing approach: the storage engine writes a checkpoint record to the WAL at regular intervals, and the checkpoint includes a reference to the current state of all in-memory data structures. On recovery, the engine reads the most recent checkpoint record and replays only the log entries that follow it. The number of entries to replay is bounded by the checkpoint interval, which is fixed at 200 ms. Under a sustained write load of 80,000 transactions per second, the maximum number of entries to replay is the volume written in 200 ms, which is a bounded and predictable quantity.

Measuring Recovery Time in Your Own Cluster

The fault injection suite available to on-premises customers includes a recovery time measurement scenario. It writes a sustained load to the cluster, injects a leader failure, and measures the time from the failure event to the first successful write on the new leader. The measurement is repeatable and can be run against your specific hardware configuration. On NVMe storage with a 10 Gbps inter-node network, the measured recovery time is consistently under 800 ms.

Recovery time is a property that should be measured on your hardware before a production deployment, not assumed from vendor documentation. The fault injection suite makes that measurement straightforward for on-premises customers.