Barbell: An Extensible Generator of On-Demand Loads for Interactive Cloud Services

Authors:

Yuxuan Liu (The Chinese University of Hong Kong, Shenzhen), Yeh-Ching Chung (The Chinese University of Hong Kong, Shenzhen), and Shuang Chen (Huawei Cloud)

Abstract:

Interactive cloud service developers and infrastructure de- signers heavily rely on load generators to optimize service performance and infrastructure architecture. However, existing load generators are far from meeting their demands, for two reasons. First, they can only generate loads of peak or stable intensity in request rate, whereas our analysis of real-world traces shows that complicated load fluctuations are common and traces are often dumped in server resource utilization. Second, they are tightly coupled with their designated service type. This blocks the load generator development from keeping up with the needs of emerging cloud services such as large language model (LLM) inference. To address these two issues, we propose an extensible load generator dubbed Barbell to generate load patterns on demand, with two innova- tions accordingly. First, we propose a multi-pass load intensity genera- tion method to generate loads with a configurable pipeline of modular passes. A segment-wise aggregation pass is devised to generate loads with segments of distinct patterns by merging segment-specific passes. A profile-guided conversion pass is designed to convert loads in resource utilization to request rates. Second, Barbell designs a decoupled architec- ture where the load generation is decoupled from the service type with hook and callback interfaces. Four service extension models are proposed to extend to services with various types of clients and protocols. Evaluation results show that Barbell achieves comparable accuracy of performance measures obtained by service type-specific load generators. It can generate loads that mimic real load patterns in the cloud. We have extended Barbell to six popular interactive services, including LLM inference, with only a few hundred lines of code for each service type.

My Duties

I did almost everything mentioned in this paper, from system design to implementation, and evaluation. Dr. chen and some other staff from Huawei Cloud helped me on the workload characterization session and the implementation of vLLM service extension.

Recommended citation:
Download Paper | Download Slides