What do near-optimal learning rate schedules look like?

概要（原文）

A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor in the success or failure of the training process, but beyond having some kind of warmup and decay, there is no consensus on what makes a good schedule shape. To answer this question, we designed a search procedure to find the best shapes within a parameterized schedule family. Our approach factors out the schedule shape from the base learning rate, which otherwise would dominate cross-schedule comparisons. We applied our search procedure to a variety of schedule families on three workloads: linear regression, image classification on CIFAR-10, and small-scale language modeling on Wikitext103. We showed that our search procedure indeed generally found near-optimal schedules. We found that warmup and decay are robust features of good schedules, and that commonly used schedule families are not optimal on these workloads. Finally, we explored how the outputs of our shape search depend on other optimization hyperparameters, and found that weight decay can have a strong effect on the optimal schedule shape. To the best of our knowledge, our results represent the most comprehensive results on near-optimal schedule shapes for deep neural network training, to date.

研究の要点

課題

学習率スケジュールの形状は学習結果を左右しますが、各ワークロードで何が準最適かについて十分な合意がありません。

手法

基準学習率とスケジュール形状を分離し、線形回帰、CIFAR-10、Wikitext103でパラメータ化された形状群を探索します。

主結果

warmupとdecayは良い形状に一貫して現れ、一般的な形状が検証条件では最適でない場合や、weight decayが最適形状を大きく変える場合を確認しました。

意義

慣例だけに依存せず、学習率形状を選ぶための探索手順と実証的根拠を提供します。

限界

検証はパラメータ化した形状群と3つのワークロードが対象で、他の規模・課題への一般化には追加検証が必要です。

関連リンク

• arXiv

引用

Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg, George E. Dahl. “What do near-optimal learning rate schedules look like?.” Transactions on Machine Learning Research, 2026.

@article{Naganuma2026What,
  title = {What do near-optimal learning rate schedules look like?},
  author = {Hiroki Naganuma and Atish Agarwala and Priya Kasimbeg and George E. Dahl},
  year = {2026},
  journal = {Transactions on Machine Learning Research},
  eprint = {2603.10301},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10301}
}