ResBench: A Comprehensive Framework for Evaluating Database Resilience
Authors
Puyun Hu
Wei Pan
Xun Jian
Zeqi Ma
Tianjie Li
Yang Shen
Chengzhi Han
Yudong Zhao
Zhanhuai Li
Abstract
Existing database benchmarks primarily focus on performance under ideal running environments. However, in real-world scenarios, databases probably face numerous adverse events. Quantifying the ability to cope with these events from a comprehensive perspective remains an open problem. We provide the definition of database resilience to describe its performance when facing adversity and propose ResBench, a benchmark for evaluating database resilience. This framework achieves automation, standardization, and visualization of the testing process through clear hierarchical decoupling. ResBench simulates adverse events and injects them during normal transaction processing, utilizing a module to gather multiple metrics for the evaluation model. We assess database resilience across eight dimensions: throughput, latency, stability, resistance, recovery, disturbance period, adaptation capability and metric deviation. All the results are presented to users via a user-friendly graphical interface. We demonstrate the execution process and result interpretation of ResBench using two types of adversity datasets.
