Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training

概要（原文）

Following AI scaling trends, frontier models continue to grow in size and continue to be trained on larger datasets. Training these models requires huge investments in exascale computational resources, which has in turn driven developtment of distributed deep learning methods. Data parallelism is an essential approach to speed up training, but it requires frequent global communication between workers, which can bottleneck training at the largest scales. In this work, we propose a method called Pseudo-Asynchronous Local SGD (PALSGD) to improve the efficiency of data-parallel training. PALSGD is an extension of Local SGD (Stich, 2018) and DiLoCo (Douillard et al., 2023), designed to further reduce communication frequency by introducing a pseudo-synchronization mechanism. PALSGD allows the use of longer synchronization intervals compared to standard Local SGD. Despite the reduced communication frequency, the pseudo-synchronization approach ensures that model consistency is maintained, leading to performance results comparable to those achieved with more frequent synchronization. Furthermore, we provide a theoretical analysis of PALSGD, establishing its convergence and deriving its convergence rate. This analysis offers insights into the algorithm's behavior and performance guarantees. We evaluated PALSGD on image classification and language modeling tasks. Our results show that PALSGD achieves better performance in less time compared to existing methods like Distributed Data Parallel (DDP), and DiLoCo. Notably, PALSGD trains 18.4% faster than DDP on ImageNet-1K with ResNet-50, 24.4% faster than DDP on TinyStories with GPT-Neo-125M, and 21.1% faster than DDP on TinyStories with GPT-Neo-8M.

研究の要点

課題

大規模なデータ並列学習では、頻繁な全体通信がボトルネックになります。

手法

Pseudo-Asynchronous Local SGDは疑似同期を導入し、モデルの整合性を保ちながらLocal SGDやDiLoCoより長い同期間隔を利用します。

主結果

収束解析を示し、DDP比でImageNet-1K/ResNet-50では18.4%、TinyStoriesのGPT-Neo-125Mと8Mでは24.4%と21.1%の学習時間短縮を報告しています。

意義

通信効率の高い分散最適化に、理論保証と実際のend-to-end高速化を結び付けています。

限界

報告された実験は画像分類と小規模言語モデルが中心で、frontier model規模での挙動はこの結果だけでは確立されていません。

関連リンク

• OpenReview
• arXiv
• Code

引用

Hiroki Naganuma, Xinzhi Zhang, Man-Chung Yue, Ioannis Mitliagkas, Russell J. Hewett, Philipp Andre Witte, Yin Tat Lee. “Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training.” Transactions on Machine Learning Research, 2025.

@article{Naganuma2025PseudoAsynchronous,
  title = {Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training},
  author = {Hiroki Naganuma and Xinzhi Zhang and Man-Chung Yue and Ioannis Mitliagkas and Russell J. Hewett and Philipp Andre Witte and Yin Tat Lee},
  year = {2025},
  journal = {Transactions on Machine Learning Research},
  eprint = {2504.18454},
  archivePrefix = {arXiv},
  url = {https://openreview.net/forum?id=8VTrvS5vN7}
}