Conjugate Gradient Method for Generative Adversarial Networks

概要（原文）

One of the training strategies of generative models is to minimize the Jensen--Shannon divergence between the model distribution and the data distribution. Since data distribution is unknown, generative adversarial networks (GANs) formulate this problem as a game between two models, a generator and a discriminator. The training can be formulated in the context of game theory and the local Nash equilibrium (LNE). It does not seem feasible to derive guarantees of stability or optimality for the existing methods. This optimization problem is far more challenging than the single objective setting. Here, we use the conjugate gradient method to reliably and efficiently solve the LNE problem in GANs. We give a proof and convergence analysis under mild assumptions showing that the proposed method converges to a LNE with three different learning rate update rules, including a constant learning rate. Finally, we demonstrate that the proposed method outperforms stochastic gradient descent (SGD) and momentum SGD in terms of best Frechet inception distance (FID) score and outperforms Adam on average. The code is available at \url{https://github.com/Hiroki11x/ConjugateGradient_GAN}.

研究の要点

課題

GANの学習はゲーム理論的な最適化問題であり、既存手法では安定性や最適性の保証が限られています。

手法

局所Nash均衡を解くために共役勾配法を用い、3種類の学習率更新則について収束性を解析します。

主結果

論文の仮定の下で局所Nash均衡への収束を示し、最良FIDでSGDとmomentum SGDを、平均ではAdamを上回りました。

意義

最適化の理論保証とGAN品質の実証的改善を結び付け、実装コードも公開しています。

限界

収束保証は論文中の仮定に依存し、実験比較だけで全てのGANアーキテクチャやデータセットでの性能を保証するものではありません。

関連リンク

• arXiv
• Poster
• Code

引用

Hiroki Naganuma, Hideaki Iiduka. “Conjugate Gradient Method for Generative Adversarial Networks.” International Conference on Artificial Intelligence and Statistics, 2023.

@inproceedings{Naganuma2023Conjugate,
  title = {Conjugate Gradient Method for Generative Adversarial Networks},
  author = {Hiroki Naganuma and Hideaki Iiduka},
  year = {2023},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  eprint = {2203.14495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.14495}
}