Hiroki Naganuma

Day1 (7/6 Mon)

HND->GMP

• Is numerical optimization theory irrelevant to machine learning practice in 2026?
  • https://www.cs.ubc.ca/~schmidtm/Documents/2026_ICML_Tutorial.pdf
  • Mark Schmidt (UBC)

Day2 (7/7 Tue)

• Breakfast w/ Atish Agarwal (GDM)

• Coffee Chat w/ Matsunaga Daiki (KAIST)

• Poster Session
  • Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Data
    • URL: https://icml.cc/virtual/2026/poster/62214
    • Authors: Hee-Sung Kim ⋅ Hyeonseong Kim ⋅ Sungyoon Lee
    • Summary: This work introduces local inconsistency as a label-free generalization measure and uses it in IAM to improve generalization across supervised, semi-supervised, and self-supervised learning.
  • FOAM: Frequency and Operator-Error Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
    • URL: https://icml.cc/virtual/2026/poster/63117
    • Authors: Kyunghun Nam ⋅ Sumyeong Ahn
    • Summary: This work analyzes the efficiency-stability trade-off from stale Shampoo preconditioner updates and proposes FOAM to adapt damping and eigendecomposition frequency for faster wall-clock training.
  • Flat Minima and Generalization: Insights from Stochastic Convex Optimization
    • URL: https://icml.cc/virtual/2026/poster/61057
    • Authors: Matan Schliserman ⋅ Shira Vansover-Hager ⋅ Tomer Koren
    • Summary: This work shows in stochastic convex optimization that flat minima do not necessarily guarantee good generalization, while sharp minima can generalize optimally, and analyzes similar limits for SA-GD and SAM.
  • One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
    • URL: https://icml.cc/virtual/2026/poster/62737
    • Authors: Philip Zmushko ⋅ Egor Petrov ⋅ Nursultan Abdullaev ⋅ Khrushchev Mikhail ⋅ Samuel Horváth
    • Summary: This work shows that one-step gradient delay in asynchronous pipeline parallelism can be mitigated through optimizer choice and Error-Feedback, narrowing the gap to synchronous training up to 10B-parameter LLMs.
  • Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation
    • URL: https://icml.cc/virtual/2026/poster/66517
    • Authors: Hyunji Jung ⋅ Sungbin Shin ⋅ Namhoon Lee
    • Summary: This work uses basis rotation to correct gradient staleness that grows with pipeline depth in asynchronous pipeline parallelism, reaching comparable loss with fewer iterations on a 1B-parameter LLM.
  • On the Role of Batch Size in Stochastic Conditional Gradient Methods
    • URL: https://icml.cc/virtual/2026/poster/60616
    • Authors: Rustem Islamov ⋅ Roman Machacek ⋅ Aurelien Lucchi ⋅ Antonio Silveti-Falls ⋅ Eduard Gorbunov ⋅ Volkan Cevher
    • Summary: This work analyzes the interaction between batch size, step size, and noise in stochastic conditional gradient methods and provides guidance for batch and sequence-length schedules under fixed token budgets.
  • Why Do We Need Warm-up? A Theoretical Perspective
    • URL: https://icml.cc/virtual/2026/poster/63104
    • Authors: Foivos Alimisis ⋅ Rustem Islamov ⋅ Aurelien Lucchi
    • Summary: This work explains early-training curvature changes through generalized $(L_0, L_1)$-smoothness and shows how learning-rate warm-up naturally emerges and improves convergence over a fixed learning rate.
  • Clipping Makes Distributed and Federated Asynchronous SGD Robust to Stragglers
    • URL: https://icml.cc/virtual/2026/poster/65724
    • Authors: Samuel Erickson ⋅ Mikael Johansson
    • Summary: This work theoretically shows that gradient clipping can remove the harmful dependence on large straggler-induced delays in asynchronous SGD, including under heavy-tailed noise.
  • Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo
    • URL: https://icml.cc/virtual/2026/poster/66683
    • Authors: Chamin Hewa Koneputugodage ⋅ Thalaiyasingam Ajanthan ⋅ Sameera Ramasinghe ⋅ Hadi Mohaghegh Dolatabadi ⋅ Shamane Siriwardhana ⋅ Gil Avraham ⋅ Violetta Shevchenko ⋅ Karol Pajak ⋅ James Snewin ⋅ Alexander Long
    • Summary: This work relaxes DiLoCo outer synchronization into gossip-style mixing, separating non-blocking and blocking communication to balance compute utilization and stability in low-bandwidth settings.
  • Adaptive Momentum and Nonlinear Damping for Neural Network Training
    • URL: https://icml.cc/virtual/2026/poster/64813
    • Authors: Aikaterini Karoni ⋅ Rajit Rajpal ⋅ Benedict Leimkuhler ⋅ Gabriel Stoltz
    • Summary: This work introduces adaptive momentum and cubic damping based on per-parameter kinetic energy, matching or outperforming Adam in ViT, BERT, and GPT-2 training.
    • Project Page: https://github.com/rajit906/higher-order-damping-in-ml-v0
• Quick Chat
  • Params Raman (Meta)
  • Philip Zmushko
• Reception
  • Adam Ibrahim (Google)
  • Vincent Quirion (Mila)
  • Guy Wolf (Mila, UdeM)
• Sakana AI Social
  • Geewook Kim (KAIST, NAVER)
  • Tanaka Tsunehiko (IBM)
  • Yoshihiro Izawa (UTokyo)
  • Gouki Minegishi (UTokyo)
  • Masanari Oi (Science Tokyo)

Day3 (7/8 Wed)

• How Far Can Quadratics Take Us? Lessons for LLM Pretraining
  • https://icml.cc/virtual/2026/invited-talk/67264

• Chat w/ Charles (IFM)

• Controlled LLM Training on Spectral Sphere
  • URL: https://icml.cc/virtual/2026/poster/66212
  • Authors: Tian Xie ⋅ Haoming Luo ⋅ Haoyu Tang ⋅ Hu Yiwen ⋅ Jason Liu ⋅ Qingnan Ren ⋅ Yang Wang ⋅ Xin Zhao ⋅ Rui Yan ⋅ Bing Su ⋅ Chong Luo ⋅ Baining Guo
  • Summary: This work proposes the Spectral Sphere Optimizer, which imposes module-wise spectral constraints on weights and updates to improve LLM training stability beyond AdamW and Muon.
  • Slides: https://icml.cc/media/icml-2026/Slides/71055.pdf
  • Project Page: https://github.com/Unakar/Spectral-Sphere-Optimizer
• LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
  • URL: https://icml.cc/virtual/2026/poster/63818
  • Authors: Andrej Jovanović ⋅ Alex Iacob ⋅ Mher Safaryan ⋅ Ionut-Vlad Modoranu ⋅ Lorenzo Sani ⋅ Shen ⋅ Xinchi Qiu ⋅ Dan Alistarh ⋅ Nicholas Lane
  • Summary: This work combines low-rank optimization with infrequent synchronization in LoRDO, reducing communication by roughly 10x while preserving performance in distributed foundation-model training.
• On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
  • URL: https://icml.cc/virtual/2026/poster/64227
  • Authors: Enea Monzio Compagnoni ⋅ Rustem Islamov ⋅ Frank Proske ⋅ Aurelien Lucchi ⋅ Antonio Orvieto ⋅ Eduard Gorbunov
  • Summary: This work uses an SDE perspective under $(L_0,L_1)$-smoothness to analyze how batch noise, communication compression, and adaptive or normalized updates affect DCSGD and DSignSGD stability.
• Per-example Gradients: a New Frontier for Understanding and Improving Optimizers
  • URL: https://icml.cc/virtual/2026/poster/63313
  • Authors: Vincent Roulet ⋅ Atish Agarwala
  • Summary: This work shows how to access per-example and per-token gradient statistics at low cost, then uses them to revisit signSGD and Adam preconditioner design.
• Exploiting weight-space symmetries for approximating curvature
  • URL: https://icml.cc/virtual/2026/poster/60589
  • Authors: Artem Artemev ⋅ Rui Xia ⋅ Benjamin M. Boyd ⋅ Youjing Yu ⋅ Felix Dangel ⋅ Guillaume Hennequin ⋅ Alberto Bernacchia
  • Summary: This work exploits weight-space symmetries that leave the loss invariant to build structured Hessian approximations from a single gradient and connect them to Shampoo- and Muon-like curvature estimates.
• OLion: Approaching the Hadamard Ideal by Intersecting Spectral and L inf Implicit Biases
  • URL: https://icml.cc/virtual/2026/poster/62586
  • Authors: Zixiao Wang ⋅ Yifei Shen ⋅ Huishuai Zhang
  • Summary: This work proposes OLion, a lightweight optimizer that combines spectral control from orthogonalized updates with $\ell_\infty$-style coordinate control from sign updates to match or outperform AdamW and Muon.
• GradientStabilizer: Fix the Norm, Not the Gradient
  • URL: https://icml.cc/virtual/2026/poster/63695
  • Authors: Tianjin Huang ⋅ Zhangyang “Atlas” Wang ⋅ Haotian Hu ⋅ Zhenyu Zhang ⋅ Gaojie Jin ⋅ Xiang Li ⋅ Li Shen ⋅ Jiaxing Shang ⋅ Tianlong Chen ⋅ Ke Li ⋅ Lu Liu ⋅ Qingsong Wen ⋅ Shiwei Liu
  • Summary: This work stabilizes update magnitudes using running gradient-norm statistics while preserving gradient direction, reducing training instability with less threshold tuning than clipping.
• Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability
  • URL: https://icml.cc/virtual/2026/poster/65256
  • Authors: Vincent Bürgin ⋅ Daniel Herbst ⋅ Ya-Wei Eileen Lin ⋅ Stefanie Jegelka
  • Summary: This work analyzes approximate equivalent solutions and linear mode connectivity through neuron identifiability and effective function classes beyond structural symmetries alone.
  • Project Page: https://github.com/Vuenc/neuron-identifiability
• WildCat: Near-Linear Attention in Theory and Practice
  • URL: https://icml.cc/virtual/2026/poster/61920
  • Authors: Tobias Schröder ⋅ Lester Mackey
  • Summary: This work approximates attention with a weighted coreset selected by randomly pivoted Cholesky, combining theoretical guarantees with near-linear runtime and high accuracy.
  • Project Page: https://github.com/microsoft/wildcat
• AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
  • URL: https://icml.cc/virtual/2026/poster/61050
  • Authors: Guoxia Wang ⋅ Shuai Li ⋅ Congliang Chen ⋅ Jinle Zeng ⋅ Jiabin Yang ⋅ Dianhai Yu ⋅ Yanjun Ma ⋅ Li Shen
  • Summary: This work uses per-tensor adaptive clipping based on historical clipped gradient norms to suppress loss spikes and optimizer-state contamination in LLM pretraining.
• MODEL MERGING SCALING LAWS IN LARGE LANGUAGE MODELS
  • URL: https://icml.cc/virtual/2026/poster/63590
  • Authors: Yuanyi Wang ⋅ Yanggan Gu ⋅ Yiming Zhang ⋅ Qi Zhou ⋅ Zhaoyi Yan ⋅ Congkai Xie ⋅ Xinyao Wang ⋅ Jianbo Yuan ⋅ Hongxia Yang
  • Summary: This work establishes power laws over model size and expert count to make the gains and stopping points of LLM model merging more predictable.
  • Project Page: https://infix.io/research/MergingScalingLaw
• Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
  • URL: https://icml.cc/virtual/2026/poster/60685
  • Authors: Zhijun Tu ⋅ Hanting Chen ⋅ Jian Li ⋅ Yuanyuan Xi ⋅ Siqi Liu ⋅ Chuanjian Liu ⋅ Jie Hu ⋅ Yunhe Wang
  • Summary: This work progressively converts pretrained LLMs from full precision to 1-bit representations in both forward and backward passes, avoiding costly training from scratch.
• A Geometry-Based View of Mahalanobis OOD Detection
  • URL: https://icml.cc/virtual/2026/poster/62433
  • Authors: Denis Janiak ⋅ Jakub Binkowski ⋅ Tomasz Kajdanowicz
  • Summary: This work shows that Mahalanobis OOD detection strongly depends on feature geometry and improves it using within-class spectral structure and local intrinsic dimensionality.
• When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning
  • URL: https://icml.cc/virtual/2026/poster/63669
  • Authors: Haoran Zhao ⋅ Caren Han ⋅ Eduard Hovy
  • Summary: This work attributes instability in rank-1 LoRA for vision-language models to misalignment with the modality-gap direction and stabilizes low-rank fine-tuning with Gap-Init.
• Path-conditioned training: a principled way to rescale ReLU neural networks
  • URL: https://icml.cc/virtual/2026/poster/61112
  • Authors: Arthur Lebeurrier ⋅ Titouan Vayer ⋅ Rémi Gribonval
  • Summary: This work uses the rescaling symmetries of ReLU networks through a path-lifting framework and accelerates training via kernel alignment in path space.
  • Project Page: https://github.com/Artim436/pathcond
• Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction
  • URL: https://icml.cc/virtual/2026/poster/61601
  • Authors: Jatin Chhugani ⋅ Geonhwa Jeong ⋅ Bor-Yiing Su ⋅ Yunjie Pan ⋅ Hanmei Yang ⋅ Aayush Ankit ⋅ Jiecao Yu ⋅ Summer Deng ⋅ Yunqing Chen ⋅ Nadathur Satish ⋅ Changkyu Kim
  • Summary: This work reduces MXFP4 quantization error with Overflow-Aware Scaling and Macro Block Scaling, approaching NVFP4-level accuracy without hardware changes.
• PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training
  • URL: https://icml.cc/virtual/2026/poster/62288
  • Authors: Shenghao Yang ⋅ Zhichao Wang ⋅ Oleg Balabanov ⋅ N. Benjamin Erichson ⋅ Michael Mahoney
  • Summary: This work accelerates iterative matrix square-root and orthogonalization computations used in Shampoo and Muon through adaptive polynomial approximation and randomized sketching.
• LiMuon: Light and Fast Muon Optimizer for Large Models
  • URL: https://icml.cc/virtual/2026/poster/61819
  • Authors: Feihu Huang ⋅ Yuning Luo ⋅ Songcan Chen
  • Summary: This work proposes LiMuon, a large-model optimizer that uses variance reduction and randomized SVD to reduce memory usage and sample complexity relative to Muon.
• Lunch with
  • Mahdi and Vincent (Mila)
• Chat at Poster Session
  • Egor Shulgin (KAUST)
  • Xi Wang (JHU, MSR AI Frontier)
• Meta Social
  • Wonbong (Won) JANG (Meta, UCL)
  • Hangyul Yoon (KAIST, MDPHD)

Day4 (7/9 Thu)

• Morning Session
  • MixQuant: Pushing the Limits of Block Rotations in Post-Training Quantization
    • URL: https://icml.cc/virtual/2026/poster/61670
    • Authors: Sai Sanjeet ⋅ Ian Colbert ⋅ Pablo Monteagudo-Lago ⋅ Giuseppe Franco ⋅ Yaman Umuroglu ⋅ Nicholas Fraser
    • Summary: This work improves INT4 post-training quantization with MixQuant, which redistributes activation mass by permutation before block Hadamard rotations.
    • Project Page: https://github.com/Xilinx/brevitas/tree/dev/src/brevitas_examples/papers/perq
  • Float8@2bits: Entropy Coding Enables Data-Free Model Compression
    • URL: https://icml.cc/virtual/2026/poster/66714
    • Authors: Patrick Putzky ⋅ Martin Genzel ⋅ Mattes Mollenhauer ⋅ Sebastian Schulze ⋅ Thomas Wollmann ⋅ Stefan Dietzel
    • Summary: This work proposes EntQuant, which decouples numerical precision from storage cost with entropy coding to compress 70B models data-free at extremely low bit rates.
    • Project Page: https://github.com/merantix-momentum/entquant
  • From Flat Facts to Sharp Hallucinations: Detecting Stubborn Errors via Gradient Sensitivity
    • URL: https://icml.cc/virtual/2026/poster/62350
    • Authors: Liew Yee Zhing ⋅ Andrew Tan ⋅ Anwar Majeed
    • Summary: This work detects stubborn high-confidence hallucinations in LLMs using EPGS, a gradient-sensitivity signal under embedding perturbations.
    • Project Page: https://github.com/potato0o/Stubborn_Hallucinations
  • Compute When Worth It: Risk Control for Reasoning on a Compute Budget
    • URL: https://icml.cc/virtual/2026/poster/61680
    • Authors: Xi Wang ⋅ Anushri Suresh ⋅ Alvin Zhang ⋅ Rishi More ⋅ William Jurayj ⋅ Mehrdad Farajtabar ⋅ Daniel Khashabi ⋅ Eric Nalisnick
    • Summary: This work formulates token-budget decisions for reasoning LLMs as distribution-free risk control, saving compute while respecting a target error rate.
  • Anatomy of Massive Activations and Attention Sinks
    • URL: https://icml.cc/virtual/2026/poster/63610
    • Authors: Shangwen Sun ⋅ Alfredo Canziani ⋅ Yann LeCun ⋅ Jiachen Zhu
    • Summary: This work gives a unified inference-time account of massive activations and attention sinks in Transformers and analyzes the roles of normalization and head dimension.
    • Project Page: https://github.com/savinasun/SpikeSparseSink
  • FPTQuant: Function-Preserving Transforms for LLM Quantization
    • URL: https://icml.cc/virtual/2026/poster/66544
    • Authors: Boris van Breugel ⋅ Yelysei Bondarenko ⋅ Paul Whatmough ⋅ Markus Nagel
    • Summary: This work uses function-preserving Transformer transforms to make activation distributions more quantization-friendly, enabling static INT4 quantization with negligible inference overhead.
  • https://eshyperscale.github.io/
  • Zeroth-Order Optimization at the Edge of Stability
    • URL: https://icml.cc/virtual/2026/poster/61252
    • Authors: Minhak Song ⋅ Liang Zhang ⋅ Bingcong Li ⋅ Niao He ⋅ Michael Muehlebach ⋅ Sewoong Oh
    • Summary: This work shows that the stability of two-point-estimator zeroth-order methods depends on the full Hessian spectrum and analyzes their edge-of-stability behavior in deep learning.
  • RubricRobustness: A Simple Framework for Evaluating the Robustness of Rubrics-Based Benchmarks
    • URL: https://icml.cc/virtual/2026/poster/65214
    • Authors: Manasi Sharma
    • Summary: This work tests the robustness of rubric-based benchmarks by applying negation, deletion, and irrelevant-addition perturbations to reveal weaknesses in LLM-as-a-judge evaluation.
  • Contribution Weights: A Geometrical Analysis of Self-Attention Transformers
    • URL: https://icml.cc/virtual/2026/poster/62621
    • Authors: Jake Cunningham ⋅ Nicola Muca Cirone
    • Summary: This work introduces Contribution Weights, combining attention weights, value magnitude, and output-direction alignment to analyze token influence and the functional role of attention sinks.
  • ECO: Quantized Training without Full-Precision Master Weights
    • URL: https://icml.cc/virtual/2026/poster/62753
    • Authors: Mahdi Nikdan ⋅ Amir Zandieh ⋅ Dan Alistarh ⋅ Vahab Mirrokni
    • Summary: This work proposes ECO, which updates quantized parameters directly without full-precision master weights by feeding quantization error back into optimizer momentum.
  • Geometric Rate-Distortion Invariance for Domain Generalization
    • URL: https://icml.cc/virtual/2026/poster/65890
    • Authors: Tong Liu ⋅ Sen Liang ⋅ Shuo Bai
    • Summary: This work improves domain generalization with RDI, which treats class-conditional representations as subspaces on a Grassmann manifold and combines alignment with complexity control.
• What will be left for us to work on?
  • https://icml.cc/virtual/2026/invited-talk/67274
• https://icml.cc/virtual/2026/oral/71156
  • Rustem Islamov ⋅ Michael Crawshaw ⋅ Jeremy Cohen ⋅ Robert Gower
• Afternoon session
  • Riemannian Dueling Optimization
    • URL: https://icml.cc/virtual/2026/poster/61755
    • Authors: Yuxuan Ren ⋅ Abhishek Roy ⋅ Shiqian Ma
    • Summary: This work extends comparison-oracle dueling optimization to Riemannian manifolds and establishes complexity results for RDNGD and projection-free RDFW.
  • Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in Reinforcement Learning
    • URL: https://icml.cc/virtual/2026/poster/63355
    • Authors: Ruijia Zhang ⋅ Jiacheng Zhu ⋅ Hanqing Zhu ⋅ Laixi Shi
    • Summary: This work derives orthonormal LoRA initialization for RLVR and improves training stability and reasoning performance with LoRA-RLPO and LoRA-RLMO.
    • Project Page: https://github.com/Richard-ZZZ/geometry-preserving-orthonormal-init-rlvr
  • Improved Convergence Analysis of Topology Dependence in Decentralized SGD
    • URL: https://icml.cc/virtual/2026/poster/61500
    • Authors: Yuki Takezawa ⋅ Anastasiia Koloskova ⋅ Sebastian Stich
    • Summary: This work analyzes topology dependence in Decentralized SGD through all eigenvalues of the mixing matrix, giving a more precise convergence picture than spectral-gap-only analyses.
  • L-SR1: Learned Symmetric-Rank-One Preconditioning
    • URL: https://icml.cc/virtual/2026/poster/60853
    • Authors: Gal Lifshitz ⋅ Shahar Zuler ⋅ Ori Fouks ⋅ Dan Raviv
    • Summary: This work augments classical SR1 with a lightweight trainable preconditioning unit to build a learned second-order optimizer aligned with the secant condition.
    • Project Page: https://gallif.github.io/lsr1/
  • Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
    • URL: https://icml.cc/virtual/2026/poster/60702
    • Authors: Haoming Meng ⋅ Anton Sugolov ⋅ Vardan Papyan
    • Summary: This work smooths layer-wise gradient updates across repeated blocks in Transformers and ResNets, acting as a preconditioning method that improves training and generalization.
  • An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
    • URL: https://icml.cc/virtual/2026/poster/64390
    • Authors: Michael Crawshaw ⋅ Chirag Modi ⋅ Mingrui Liu ⋅ Robert Gower
    • Summary: This work frames Muon and Adam-style methods as non-Euclidean gradient descent over layer-wise norms and aggregations, leading to more robust MuonMax and Momo variants.
  • An Embarrasingly Simple Way to Optimize Orthogonal Matrices at Scale
    • URL: https://icml.cc/virtual/2026/poster/61285
    • Authors: Adrián Javaloy ⋅ Antonio Vergari
    • Summary: This work introduces POGO, a GPU-friendly optimizer that preserves orthogonal constraints with only a few matrix products and scales orthogonal matrix optimization.
    • Project Page: https://github.com/adrianjav/pogo
  • https://icml.cc/virtual/2026/oral/71156
  • Understanding SAM through Minimax Perspective
    • URL: https://icml.cc/virtual/2026/poster/65197
    • Authors: Ying Chen ⋅ Aoxi Li ⋅ Javad Lavaei
    • Summary: This work analyzes SAM as a bilevel minimax problem and provides theoretical support for Multi-step SAM with larger radii and multiple inner updates.
  • Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
    • URL: https://icml.cc/virtual/2026/poster/64318
    • Authors: Dimitris Oikonomou ⋅ Nicolas Loizou
    • Summary: This work introduces stochastic Polyak step sizes into SAM-style updates, yielding an adaptive SAM scheduler that reduces learning-rate tuning.
  • General Analysis of LMO-based Optimizers: Beyond Bounded Variance
    • URL: https://icml.cc/virtual/2026/poster/61267
    • Authors: Egor Shulgin ⋅ Mohamed Awad ⋅ Peter Richtarik ⋅ Eduard Gorbunov
    • Summary: This work gives a unified expected-smoothness analysis of momentum LMO methods and derives batch-size scaling and optimal momentum rules for Muon and sign-based updates.
  • Lions and Muons: Optimization via Stochastic Frank-Wolfe
    • URL: https://icml.cc/virtual/2026/poster/62403
    • Authors: Maria-Eleni Sfyraki ⋅ Jun-Kun Wang
    • Summary: This work interprets Lion and Muon as special cases of Stochastic Frank-Wolfe, proving Frank-Wolfe gap convergence and developing variants robust to heavy-tailed noise.
  • Can Muon Fine-tune Adam-Pretrained Models?
    • URL: https://icml.cc/virtual/2026/poster/64467
    • Authors: Xingyu Qu ⋅ Peigeng Huang ⋅ Samuel Horváth
    • Summary: This work analyzes optimizer mismatch when fine-tuning Adam-pretrained models with Muon and shows that LoRA can mitigate the resulting degradation.
    • Project Page: https://github.com/XingyuQu/muon-finetune
  • Step-Size Stability in Stochastic Optimization: A Theoretical Perspective
    • URL: https://icml.cc/virtual/2026/poster/60591
    • Authors: Fabian Schaipp ⋅ Robert Gower ⋅ Adrien Taylor
    • Summary: This work quantifies step-size sensitivity across stochastic optimization methods and explains why adaptive step-size methods such as SPS and NGN are more robust than SGD.
  • Delving into Muon and Beyond: Deep Analysis and Extensions
    • URL: https://icml.cc/virtual/2026/poster/60694
    • Authors: Xianbiao Qi ⋅ Marco Chen ⋅ Jiaquan Ye ⋅ Yelin He ⋅ Rong Xiao
    • Summary: This work views Muon as an endpoint of spectral transformations and analyzes its stabilizing effects and limitations through RMS-normalized and spectral variants.
    • Project Page: https://github.com/Ocram7/BeyondMuon
  • A Geometry-Aware Efficient Algorithm for Compositional Entropic Risk Minimization
    • URL: https://icml.cc/virtual/2026/poster/66768
    • Authors: Xiyuan Wei ⋅ Linli Zhou ⋅ Bokun Wang ⋅ Chih-Jen Lin ⋅ Tianbao Yang
    • Summary: This work formulates compositional entropic risk minimization as a min-min dual problem and optimizes it efficiently with geometry-aware stochastic proximal mirror descent.
    • Project Page: https://github.com/Optimization-AI/SCENT
• Quick Chat
  • Xi Wang (JHU, MSR AI Frontier)
  • Atsushi Nitanda (NTU, A*STAR)

Thoughts

In the optimization sessions I attended, I noticed fewer Stanford and CMU affiliations than I expected, though this may simply reflect my own session selection.

The work that stood out to me came from a broad set of places, including UBassel, ISTA, EPFL, Flatiron, Mila, Vector, Meta, Google, and KAUST.

Several papers were close enough to my own research interests that they reminded me to be more careful about positioning, related work, and citations.

One recurring lesson was that strong research usually comes from deep domain fluency rather than frequent shallow pivots.

In optimization, many productive researchers seem to have spent time in leading groups, and informal conversations often revolve around research lineage, internships, and shared technical context.

Conversations with researchers from different academic environments also highlighted a trade-off between intense publication pressure and having enough space to pursue longer-term, more selective research directions.

---

Papers I Should Understand More Deeply

• Controlled LLM Training on Spectral Sphere
  • In practice, people working on Muon-style methods in frontier AI labs seem to be using Hyperbolic Sphere-style methods.
  • The Spectral Sphere Optimizer imposes module-wise spectral constraints on weights and updates to improve LLM training stability beyond AdamW and Muon.
• LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
  • LoRDO combines low-rank optimization with infrequent synchronization, reducing communication by roughly 10x while preserving performance in distributed foundation-model training.
• Factored Gossip DiLoCo: Reducing Blocking Communication within DiLoCo
  • Factored Gossip DiLoCo relaxes DiLoCo outer synchronization into gossip-style mixing and separates non-blocking from blocking communication, balancing compute utilization and stability in low-bandwidth settings.
• OLion: Approaching the Hadamard Ideal by Intersecting Spectral and L inf Implicit Biases
  • OLion combines spectral control from orthogonalized updates with $\ell_\infty$-style coordinate control from sign updates, matching or outperforming AdamW and Muon.
• Exploiting weight-space symmetries for approximating curvature
  • This work exploits weight-space symmetries that leave the loss invariant to build structured Hessian approximations from a single gradient and connect them to Shampoo- and Muon-like curvature estimates.
• Rethinking 1-bit Optimization Leveraging Pre-trained Large Language Models
  • This work progressively converts pretrained LLMs from full precision to 1-bit representations in both forward and backward passes, avoiding costly training from scratch.
• Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction
  • This work reduces MXFP4 quantization error with Overflow-Aware Scaling and Macro Block Scaling, approaching NVFP4-level accuracy without hardware changes.
• PRISM: Distribution-free Adaptive Computation of Matrix Functions for Accelerating Neural Network Training
  • PRISM accelerates iterative matrix square-root and orthogonalization computations used in Shampoo and Muon through adaptive polynomial approximation and randomized sketching.
• FPTQuant: Function-Preserving Transforms for LLM Quantization
  • FPTQuant uses function-preserving Transformer transforms to make activation distributions more quantization-friendly, enabling static INT4 quantization with negligible inference overhead.
• Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
  • This work smooths layer-wise gradient updates across repeated blocks in Transformers and ResNets, acting as a preconditioning method that improves training and generalization.

---

Small Research Ideas to Develop

• A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
  • Explanatory post
  • This work theoretically analyzes how pretraining shapes the inductive bias of fine-tuning.
  • It complements the questions below; one direction is to think about how to use this perspective to improve fine-tuning or optimizer switching.
• Can Muon Fine-tune Adam-Pretrained Models?
  • This work analyzes optimizer mismatch when fine-tuning Adam-pretrained models with Muon and shows that LoRA can mitigate the resulting degradation.
  • This is still exploratory; one question is how to mitigate this mismatch more systematically.
• ECO: Quantized Training without Full-Precision Master Weights
  • ECO updates quantized parameters directly without full-precision master weights by feeding quantization error back into optimizer momentum.
  • It would be interesting to test how this kind of error-feedback idea behaves in Muon-style methods.
• GradientStabilizer: Fix the Norm, Not the Gradient
  • This work stabilizes update magnitudes using running gradient-norm statistics while preserving gradient direction, reducing training instability with less threshold tuning than clipping.
  • It raises questions about whether the stabilization is being done in the right norm space, and whether similar ideas can be extended to Muon or Hyperbolic Sphere Optimizer.
• AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
  • AdaGC uses per-tensor adaptive clipping based on historical clipped gradient norms to suppress loss spikes and optimizer-state contamination in LLM pretraining.
  • This also raises the question of whether the clipping is happening in the right norm space, and whether it can be extended to Muon or Hyperbolic Sphere Optimizer.

---

Acknowledgements

I want to thank my PhD supervisor Ioannis Mitliagkas (Mila, UdeM) and Irina Rish for supporting my participation in the ICML.