Conjugate Gradient Method for Generative Adversarial Networks

Abstract

One of the training strategies of generative models is to minimize the Jensen--Shannon divergence between the model distribution and the data distribution. Since data distribution is unknown, generative adversarial networks (GANs) formulate this problem as a game between two models, a generator and a discriminator. The training can be formulated in the context of game theory and the local Nash equilibrium (LNE). It does not seem feasible to derive guarantees of stability or optimality for the existing methods. This optimization problem is far more challenging than the single objective setting. Here, we use the conjugate gradient method to reliably and efficiently solve the LNE problem in GANs. We give a proof and convergence analysis under mild assumptions showing that the proposed method converges to a LNE with three different learning rate update rules, including a constant learning rate. Finally, we demonstrate that the proposed method outperforms stochastic gradient descent (SGD) and momentum SGD in terms of best Frechet inception distance (FID) score and outperforms Adam on average. The code is available at \url{https://github.com/Hiroki11x/ConjugateGradient_GAN}.

Research summary

Problem

GAN training is a difficult game-theoretic optimization problem for which existing methods provide limited stability or optimality guarantees.

Method

The paper applies a conjugate-gradient method to solve for a local Nash equilibrium and analyzes convergence for three learning-rate update rules.

Result

Under the paper's assumptions, the method converges to a local Nash equilibrium and improves best FID over SGD and momentum SGD while outperforming Adam on average.

Significance

It combines an optimization guarantee with empirical GAN-quality improvements and provides public implementation code.

Limitations

The convergence guarantee depends on stated assumptions, and the reported empirical comparisons do not establish performance for every GAN architecture or dataset.

Research links

• arXiv
• Poster
• Code

How to cite

Hiroki Naganuma, Hideaki Iiduka. “Conjugate Gradient Method for Generative Adversarial Networks.” International Conference on Artificial Intelligence and Statistics, 2023.

@inproceedings{Naganuma2023Conjugate,
  title = {Conjugate Gradient Method for Generative Adversarial Networks},
  author = {Hiroki Naganuma and Hideaki Iiduka},
  year = {2023},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  eprint = {2203.14495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.14495}
}