Research Overview

Scalable Pretraining and Distributed Learning

Efficient training of large language and foundation models through distributed, semi-synchronous, and large-batch methods.

Topics: large language model pretraining · distributed deep learning · large-batch training · high-performance computing

Selected related publications

• Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training — TMLR 2025
• Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent — ICML 2026
• Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

Research notes

Optimization for Machine Learning

Optimization algorithms and training dynamics, including learning-rate schedules, non-Euclidean and spectral methods, Muon, and critical batch size analysis.

Topics: machine learning optimization · Muon optimizer · non-Euclidean optimization · learning-rate schedules · critical batch size

Selected related publications

• What do near-optimal learning rate schedules look like? — TMLR 2026
• Convergence Bound and Critical Batch Size of Muon Optimizer
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths — ICML 2024

Muon optimizer research note

Generalization and Foundation Models

Generalization, calibration, fairness, and model composition for pre-trained and foundation models under distribution shift.

Topics: foundation models · out-of-distribution generalization · model calibration · distribution shift · model merging

Selected related publications

• An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration — TMLR 2025
• Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark — TMLR 2026
• Empirical Study on Optimizer Selection for Out-of-Distribution Generalizations — TMLR 2023