LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
Large Language Model (LLM) systems have been the frontier of AI in many application domains, leading to new challenges and opportunities for hyperparameter optimization (HPO) for the AutoML community. However, this type of system exhibits an unprecedented compound space of hyperparameter configuration from both the AI and non-AI components; rich and nonlinear implications from the fidelity factors; and diverse costs of measuring hyperparameter configurations, none of which have been fully captured in existing benchmarks. This paper presents the first (live) benchmark suite and datasets for HPO of real-world LLM systems, dubbed LLMSYS-HPOBench, covering data related to the inference object...
| Authors | Siyu Wu, Yulong Ye, Zezhen Xiang et al. |
|---|---|
| Primary category | cs.LG |
| Published | 8 May 2026 |
| Last updated at source | 8 May 2026 |
Connected in the knowledge graph
- KleinWu/llmsys-hpobenchDataset · cited by
Models and datasets referencing this paper
- KleinWu/llmsys-hpobenchDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.