ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models
Current heterogeneity-aware training systems for large models balance workloads by assigning unequal numbers of layers across pipeline stages, a strategy aimed at equalizing execution time through a unique pipeline configuration. However, this approach fails to address critical memory constraints and suffers from inherently coarse balancing granularity. In this paper, we propose Heterogeneous Pipeline Parallelism (HPP), a novel approach enabling efficient and fine-grained data balance across heterogeneous pipelines. To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, we design the ETH training ...
TLDR (Semantic Scholar, machine-generated): To address HPP's key challenges, prohibitive search overhead for optimal configurations and high communication overhead from asymmetric All-Reduce, the ETH training system is designed, which employs a hierarchical workload balance framework that jointly optimizes balance across heterogeneous pipelines and within individual pipeline stages.
| Authors | Mianjie Yu, Zhongmin Zhao, Zizhao Mo et al. |
|---|---|
| Primary category | cs.DC |
| Published | 15 October 2026 |
| Last updated at source | 15 October 2026 |
| Citations (Semantic Scholar) | 0 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 16 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.