Hiroki Naganuma

ICLR2025

Day -1 (4/22 Tue)

HND->SIN

Day0 (4/23 Wed)

Exploring the city of Singapore

Day1 (4/24 Thur)

People

• Registration
  • Yutaro Yamada (SakanaAI)
• Poster presentation
  • Ryan, Zichu (Mila, UdeM)
  • Priya (Google DeepMind), Frank (U Tübingen), Runa (U Cambridge)
• Lunch
  • Priya (Google DeepMind), Frank (U Tübingen), Runa (U Cambridge)

• Quick chat

  • Shu Ishida (Oxford -> Autodesk)

• Booth

  • Shopify
    • Full remote
    • Applied ML Engineering role is open, No research scientist role
  • Google
    • Research role is open only in Japan (in Asia)
  • Apple
    • Research role is open only in US or Europe
  • Jane Street
    • roles in NY is open
• Reception
  • Atushi Nitanda (A*STAR)
  • Takuya Akiba (SakanaAI)
• Quick chat
  • Koichi Nametaka (Vector Institute)
  • Tatsunori Taniai (OMRON SINIC X)
  • Takahiro Kawashima (ZOZO Research)
  • Ryotaro Shimizu (UCSD)

Poster

• Sharpness-Aware Minimization: General Analysis and Improved Rates
• Understanding Optimization in Deep Learning with Central Flows
• Test-time Adaptation for Regression by Subspace Alignment
• What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

🎯 Catch us at our spotlight poster during the first poster session at #ICLR2025!
📍Poster Session 1, Apr 24, 10:00, #357
📄 Paper: “What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis.”
🔗Link: https://t.co/1N9V6V0CxB
🇸🇬 See you in Singapore!
8/8 pic.twitter.com/lTq0N6Lz0i

— Weronika Ormaniec (@wormaniec) April 22, 2025

• Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine

multimodalモデルの学習で、普通はペア化されたデータが必要だが、unimodalに事前学習されたエンコーダを使って効率的に学習する方法。周波数ドメインで特徴を表現するのがキモか？ pic.twitter.com/4OtnXaJ7dY

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis

transformerのself attentionのQKV重みのHessianを分析すると、MLPよりも、データへの依存性が高い。 pic.twitter.com/CedRLMTR17

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• Do Stochastic, Feel Noiseless: Stable Stochastic Optimization via a Double Momentum Mechanism

勾配とモデルパラメータの両方にmomentum機構を使うoptimizer pic.twitter.com/3I08kgLqJV

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation

Hyperfittingという現象。事前学習したモデルを少数データでfinetuneすると、学習lossがゼロ、val lossが高くなる現象がある。実はこのとき、val lossが高いけど、画像を生成してみると、見た目的には（fine tuning前より）良かったりする。 pic.twitter.com/WKBNEu9Kku

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• A Geometric Framework for Understanding Memorization in Generative Models

生成モデルのmemorization問題を、data manifoldの、サンプルx周辺のlocal intrinsic dimension LID(x)の低さで説明。図のxがサンプル、青がdatasetが持つ真の多様体、ピンクが学習された多様体。 pic.twitter.com/68KEUhptzP

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• Instance-dependent Early Stopping

「インスタンスレベルでのearly stopping。あるインスタンスを"マスター"にしたら、そのインスタンスの学習はやめてよい。」 pic.twitter.com/sxe8x64RUk

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

• Inverse Scaling: When Bigger Isn’t Better

Scalingすると悪くなる場合はあるのか？→そういうタスクを考えるコンペ開催して分析→結論：ある→と思ったらもっと大きくすると上がる？ pic.twitter.com/TmP876fnqh

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 24, 2025

Day2 (4/25 Fri)

People

• Google reception
  • Algoperf team
    • Priya, Runa, Frank Schneider
  • Meta @Montreal
    • Mohammad
      • The Pitfalls of Memorization: When Memorization Hurts Generalization
    • Reyhane
  • Google DeepMind
    • DiLoCo Team
      • Henry Prior
    • Bayesian, Second Order Optimization
      • Ehsan Amid

Poster

• Our poster presentation

Presenting our work, “Mastering Task Arithmetic: τJp as a Key Indicator for Weight Disentanglement,” this Friday, Apr 25, 3:00–5:30 p.m.
Interested in task arithmetic? Please stop by our poster!#ICLR25 @Mila_Quebec https://t.co/XzAAwwf42p

— Hiroki Naganuma (@_Hiroki11x) April 23, 2025

• Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

プロンプト入力をトークンとして入力するのでなく、Transformer内部の重み補正パラメータΔWに変換する。そのためのencoderを学習する。テスト時には、プロンプト→ΔW=Enc(プロンプト)→MHA後のLinear layer WにW+ΔWとして挿し込む。おもしろい！ pic.twitter.com/QXuGmuLF57

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 25, 2025

• Accelerating Training with Neuron Interaction and Nowcasting Networks

Optimization can be sped up by 50% using our NiNo model! It takes a history of parameter values and predicts future parameters leveraging "neural graphs".
Accelerating Training with Neuron Interaction and Nowcasting Networks: https://t.co/hd9GOXA3rH
code: https://t.co/hdQMTfxECd https://t.co/oF1hHbnrsF pic.twitter.com/JTQza4XB9W

— Boris Knyazev (@BorisAKnyazev) September 13, 2024

• Expressivity of Neural Networks with Random Weights and Learned Biases

重みはランダムでbiasだけ学習しても、結構表現力がある、という理論と実験による解析。ただしレイヤーの幅（チャンネル数）は十分大きい想定 pic.twitter.com/yhAVXBvftv

— 谷合 竜典 | Tatsunori Taniai @ ICLR2025 (@taniai_phd) April 25, 2025

Day3 (4/26 Sat)

People

Conference

• ByteDance Career
• Salesforce
• Jane Street
• Shopify
  • Kasia Krotki
• mbzuai institute of foundation models @Sunnyvale
  • Lead Pre-Training / Dynamics / Hongyi Wang
    • Prev intern at DeepSpeed Team @MSR
  • Director / Taylor Killian
    • UofT -> Apple -> MBZUAI
  • Open positions
    • Full time role / not postdoc

Salesforce Reception and Dinner

Researcher from Philipp Hennig Lab

• Lukas Tatzel
  • working on second order optimization
• Felix Dangel
  • author of backpack, now postdoc at vector institute with Roger Grosse

Mila - Vector folks

• Diego Calanzone
• Helen Zhang
• Pritam Sarkar

Others

• Hanyu Wang: from University of Maryland

Posters

• A Second-Order Perspective on Model Compositionality and Incremental Learning

• Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning

• Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture

Want to understand how parametrization affects generalization in deep learning?

Do not miss our spotlight poster #ICLR2025
-- today Hall 3 + Hall 2B #315@smoosavid can answer very deep questions!https://t.co/uIWCzzxnpp pic.twitter.com/OaDXQjxg33

— Antonio Orvieto (@orvieto_antonio) April 26, 2025

Day4 (4/27 Sun)

SIN->HND

Acknowlegements

I want to thank The ANRI Fellowship for supporting my participation in the ICLR2025.