Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark

旧プレプリント題名

• Revisiting Generalization Measures Beyond IID: An Empirical Study under Distributional Shift

概要（原文）

Predicting generalization from quantities available before target-test evaluation remains a central challenge in deep learning. The systematic benchmark of Jiang et al. (2020) evaluated many generalization measures, but it focused on independent and identically distributed (IID) settings. We revisit this problem for image classifiers evaluated under controlled corruptions and perturbations. Our study uses CIFAR-10-C/P, where the label space and task remain fixed while the input images are degraded or perturbed. This setting also allows us to revisit the robustness concerns raised by Dziugaite et al. (2020), who showed that the apparent reliability of generalization measures can depend strongly on experimental conditions. Our experiments show that the usefulness of generalization measures is strongly regime-dependent. In our exploratory decision analysis across three CNN-style architectures, sharpness- and input-gradient-based measures are among the leading individual signals, whereas family results are close and architecture dependent. Optimization-based measures, Information Criteria, and Sharpness-based measures provide additional regime-dependent signals in correlation or local-reliability analyses. Together, these findings suggest that model selection should not rely only on measures favored by IID evaluation. Instead, within the evaluated CIFAR-10-C/P setting and architectures, generalization measures should be treated as regime-dependent ranking signals whose utility must be evaluated for the intended corruption or perturbation setting.

関連リンク

• OpenReview
• arXiv v1 (previous title)

引用

Sora Nakai, Youssef Fadhloun, Kacem Mathlouthi, Kotaro Yoshida, Ganesh Talluri, Ioannis Mitliagkas, Hiroki Naganuma. “Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark.” Transactions on Machine Learning Research, 2026.

@article{Nakai2026Generalization,
  title = {Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark},
  author = {Sora Nakai and Youssef Fadhloun and Kacem Mathlouthi and Kotaro Yoshida and Ganesh Talluri and Ioannis Mitliagkas and Hiroki Naganuma},
  year = {2026},
  journal = {Transactions on Machine Learning Research},
  issn = {2835-8856},
  eprint = {2602.01718},
  archivePrefix = {arXiv},
  url = {https://openreview.net/forum?id=X4RoujAYnY}
}