Generative Adversarial Nets

lan J.Goodfellow，Jean Pouget-Abadie，Mehdi Mirza，Bing Xu， David Warde-Farley,Sherjil Ozair Aaron Courville，Yoshua Bengiot
Département d’informatique et de recherche operationnelle
Universite de Montréal
Montreal, QC H3C 3J7

Abstract

We propose a new framework for estimating generative models via an adversar-ial process, in which we simultaneously train two models: a generative model Gthat captures the data distribution,and a discriminative model D that estimates the probability that a sample came from the training data rather than G.The train-ing procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training datadistribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation.There is no need for any Markov chains or unrolled approximate inference net-works during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

摘要

我们提出了一个新的框架来通过对抗过程来估计生成模型，在该框架中，我们同时训练两个模型：一个生成模型G捕获数据分布，一个鉴别模型D估计样本来自训练数据的概率 G的训练过程是使D犯错误的可能性最大化。 该框架对应于minimax两人游戏。 在任意函数G和D的空间中，存在唯一的解决方案，其中G恢复训练数据分布，并且D各处都等于1/2。 如果G和D由多层感知器定义，则整个系统可以通过反向传播进行训练，在训练或生成样本的过程中不需要任Markov 链或展开的近似推理网络。 实验通过对生成的样本进行定性和定量评估，证明了该框架的潜力。

1Introduction
The promise of deep learning is to discover rich, hierarchical models [2] that represent probability distributions over the kinds of data encountered in artificial intelligence applications, such as naturalimages, audio waveforms containing speech, and symbols in natural language corpora.So far, themost striking successes in deep learning have involved discriminative models，usually those thatmap a high-dimensional, rich sensory input to a class label [ 14,22]. These striking successes haveprimarily been based on the backpropagation and dropout algorithms, using piecewise linear units[19,9,10] which have a particularly well-behaved gradient . Deep generative models have had lessof an impact, due to the difficulty of approximating many intractable probabilistic computations thatarise in maximum likelihood estimation and related strategies, and due to difficulty of leveragingthe benefits of piecewise linear units in the generative context. We propose a new generative modelestimation procedure that sidesteps these difficulties.
In the proposed adversarial nets framework, the generative model is pitted against an adversary: adiscriminative model that learns to determine whether a sample is from the model distribution or thedata distribution. The generative model can be thought of as analogous to a team of counterfeiters,trying to produce fake currency and use it without detection，while the discriminative model isanalogous to the police, trying to detect the counterfeit currency.Competition in this game drivesboth teams to improve their methods until the counterfeits are indistiguishable from the genuinearticles.
This framework can yield specific training algorithms for many kinds of model and optimizationalgorithm. In this article, we explore the special case when the generative model generates samplesby passing random noise through a multilayer perceptron, and the discriminative model is also amultilayer perceptron. We refer to this special case as adversarial nets. In this case, we can trainboth models using only the highly successful backpropagation and dropout algorithms [17] andsample from the generative model using only forward propagation.No approximate inference or Markov chains are necessary.

1简介
深度学习的希望是发现丰富的，层次化的模型[2]，该模型表示人工智能应用中遇到的各种数据的概率分布，例如自然图像，包含语音的音频波形以及自然语言语料库中的符号。深度学习的成功涉及判别模型，通常是将高维度，丰富的感官输入映射到班级标签的模型[14,22]。这些惊人的成功主要是基于反向传播和辍学算法，使用分段线性单元[19,9,10]，它们具有特别良好的梯度。深度生成模型的影响较小，这是由于难以逼近许多在最大似然估计和相关策略中难以解决的难于计算的概率，以及由于难以在生成上下文中利用分段线性单元的优势。我们提出了一种新的生成模型估计程序，可以避免这些困难。
在提出的对抗网络框架中，生成模型与一个对手相对立：一种区分模型，该模型学习确定样本是来自模型分布还是来自数据分布。可以将生成模型视为一个伪造团队，试图制造假货币并在未经检测的情况下使用它，而区分模型类似于警察，试图发现假币。直到仿冒品与真品无法区分为止。
该框架可以针对多种模型和优化算法产生特定的训练算法。 在本文中，我们探讨了特殊情况，即生成模型通过使随机噪声穿过多层感知器来生成样本，而判别模型也是多层感知器。 我们将此特殊情况称为对抗网。 在这种情况下，我们可以只使用非常成功的反向传播和丢失算法来训练两个模型，也可以只使用正向传播从生成模型中采样。不需要近似推理或Markov 链。

2 Related work

An alternative to directed graphical models with latent variables are undirected graphical modelswith latent variables，such as restricted Boltzmann machines (RBMs)[27,16], deep Boltzmannmachines (DBMs)[26] and their numerous variants. The interactions within such models arerepresented as the product of unnormalized potential functions，normalized by a global summa-tion/integration over all states of the random variables. This quantity (the partition function) andits gradient are intractable for all but the most trivial instances, although they can be estimated byMarkov chain Monte Carlo (MCMC)methods. Mixing poses a significant problem for learningalgorithms that rely on MCMC[3,5].
Deep belief networks (DBNs)[16] are hybrid models containing a single undirected layer and sev-eral directed layers. While a fast approximate layer-wise training criterion exists, DBNs incur thecomputational difficulties associated with both undirected and directed models.
Alternative criteria that do not approximate or bound the log-likelihood have also been proposed,such as score matching [18] and noise-contrastive estimation(NCE) [13J.Both of these require thelearned probability density to be analytically specified up to a normalization constant.Note thatin many interesting generative models with several layers of latent variables (such as DBNs andDBMs), it is not even possible to derive a tractable unnormalized probability density. Some modelssuch as denoising auto-encoders [30] and contractive autoencoders have learning rules very similarto score matching applied to RBMs. In NCE, as in this work, a discriminative training criterion isemployed to fit a generative model.However, rather than fitting a separate discriminative model, thegenerative model itself is used to discriminate generated data from samples a fixed noise distribution.Because NCE uses a fixed noise distribution, learning slows dramatically after the model has learnedeven an approximately correct distribution over a small subset of the observed variables.
Finally, some techniques do not involve defining a probability distribution explicitly, but rather traina generative machine to draw samples from the desired distribution.This approach has the advantagethat such machines can be designed to be trained by back-propagation. Prominent recent work in thisarea includes the generative stochastic network (GSN) framework [5], which extends generalizeddenoising auto-encoders [4]: both can be seen as defining a parameterized Markov chain, i.e., onelearns the parameters of a machine that performs one step of a generative Markov chain.Comparedto GSNs, the adversarial nets framework does not require a Markov chain for sampling.Becauseadversarial nets do not require feedback loops during generation,they are better able to leveragepiecewise linear units [19，9,10],which improve the performance of backpropagation but haveproblems with unbounded activation when used ina feedback loop. More recent examples of traininga generative machine by back-propagating into it include recent work on auto-cncoding variationalBayes [20] and stochastic backpropagation [24].

2 相关工作
具有潜在变量的有向图形模型的替代方法是具有潜在变量的无向图形模型，例如受限的Boltzmann机器（RBM）[27,16]，深度Boltzmann机器（DBM）[26]及其众多变体。这些模型内的相互作用表示为未归一化的势函数的乘积，通过对随机变量所有状态的全局求和/归一化对其进行归一化。尽管可以通过 Markov 链蒙特卡罗（MCMC）方法估计它们，但除最琐碎的情况外，此数量（分配函数）及其梯度对于所有实例而言都是棘手的。对于依赖MCMC [3,5]的学习算法，混合提出了一个重大问题。
深度信念网络（DBN）[16]是包含单个无向层和多个有向层的混合模型。尽管存在快速的近似逐层训练准则，但DBN会招致与无向模型和有向模型都相关的计算困难。
还提出了一些不近似或不限制对数似然性的标准，例如得分匹配[18]和噪声对比估计（NCE）[13J]。这两种标准都需要对所获悉的概率密度进行解析，直至进行归一化请注意，在许多有趣的生成模型中，这些生成模型具有多层潜在变量（例如DBN和DBM），甚至不可能得出可控的未归一化概率密度。一些模型，例如去噪自动编码器[30]和压缩自动编码器，具有与应用于RBM的得分匹配非常相似的学习规则。在NCE中，像在本文中一样，采用判别训练准则来拟合生成模型。但是，生成模型本身不是用于拟合单独的判别模型，而是用于将样本中的生成数据区别为固定的噪声分布。在噪声分布的情况下，在模型学习到观察变量的一小部分上的近似正确分布之后，学习速度显着降低。
最后，一些技术不涉及明确定义概率分布，而是训练生成机器以从所需分布中提取样本。这种方法的优势在于可以设计这种机器通过反向传播进行训练。该领域最近的杰出工作包括生成随机网络（GSN）框架[5]，该框架扩展了广义降噪自动编码器[4]：两者都可以看作是定义了参数化的马尔可夫链，即，可以学习执行一个参数的机器的参数。与GSN相比，对抗网络不需要马尔可夫链进行采样。由于对抗网络在生成过程中不需要反馈环，因此它们能够更好地利用分段线性单元[19，9,10]，改进了反向传播的性能，但在反馈回路中使用时，存在无限激活的问题。通过反向传播训练生成机器的最新例子包括最近对自动变分贝叶斯编码的研究[20]和随机反向传播[24]。

3 Adversarial nets

The adversarial modeling framework is most straightforward to apply when the models are bothmultilayer perceptrons. To learn the generator’s distribution pg over data x,we define a prior oninput noise variables pz(z ), then represent a mapping to data space as G(z;09), where G is adifferentiable function represented by a multilayer perceptron with parameters 0g" We also define asecond multilayer perceptron D(a;0a ) that outputs a single scalar. D(z ) represents the probabilitythat a came from the data rather than pg. We train D to maximize the probability of assigning thecorrect label to both training examples and samples from G. We simultaneously train G to minimizelog(1 - D(G(z))):

In other words, D and G play the following two-player minimax game with value function V(G, D):

3个对抗网
当模型都是多层感知器时，对抗建模框架最容易应用。 为了了解发生器在数据x上的分布pg，我们定义了一个先验的输入噪声变量pz（z），然后将到数据空间的映射表示为G（z; 09），其中G是由参数为0g的多层感知器表示的微分函数 “我们还定义了第二个多层感知器D（a; 0a），它输出单个标量。D（z）表示a来自数据而不是pg的概率。我们训练D以使为两个训练示例分配正确标签的概率最大化 和G的样本。我们同时训练G以使对数最小（1-D（G（z）））

换句话说，D和G玩以下具有值函数V（G，D）的两人minimax游戏：

In the next section,we present a theoretical analysis of adversarial nets，essentially showing thatthe training criterion allows one to recover the data generating distribution as G and D are givenenough capacity, i.e., in the non-parametric limit.See Figure 1 for a less formal, more pedagogicalexplanation of the approach. In practice, we must implement the game using an iterative, numericalapproach. Optimizing D to completion in the inner loop of training is computationally prohibitiveand on finite datasets would result in overfitting. Instead, we alternate between k steps of optimizingD and one step of optimizing G. This results in D being maintained near its optimal solution, solong as G changes slowly enough. This strategy is analogous to the way that SMLPCD [31,29]training maintains samples from a Markov chain from one learning step to the next in order to avoidburning in a Markov chain as part of the inner loop of learning.The procedure is formally presentedin Algorithm 1.
In practice, equation 1 may not provide sufficient gradient for G to learn well. Early in learning.when G is poor, D can reject samples with high confidence because they are clearly different fromthe training data. In this case, log(1 - D(G(z))) saturates. Rather than training G to minimizelog(1 - D(G(z))) we can train G’ to maximize log D(G(z)). This objective function results in thesame fixed point of the dynamics of G and D but provides much stronger gradients early in learning.

在下一部分中，我们将对对抗网进行理论分析，本质上表明，当G和D具有足够的容量（即在非参数范围内）时，训练准则允许人们恢复数据生成的分布。对该方法进行正式的，更具教学性的解释。在实践中，我们必须使用迭代的数值方法来实现游戏。在训练的内部循环中将D优化到完成是计算上的问题，而在有限的数据集上将导致过度拟合。取而代之的是，我们在优化D的k个步骤和优化G的一个步骤之间进行交替。只要G改变得足够缓慢，就可以使D保持在最佳解附近。此策略类似于SMLPCD [31,29]训练从一个学习步骤到下一个学习步骤，从马尔可夫链中维护样本的方法，以避免在学习的内部循环中出现马尔可夫链中的灼伤。算法1。
在实践中，等式1可能无法为G提供足够的梯度来学习。在学习的早期。当G较差时，D可以以高置信度拒绝样本，因为它们明显不同于训练数据。在这种情况下，log（1-D（G（z）））饱和。与其训练G以使log（1-D（G（z）））最小，我们可以训练G’以使log D（G（z））最大化。该目标函数导致G和D动力学的固定点相同，但在学习早期就提供了更强的梯度。

855 / 5000
翻译结果
图1：通过同时更新判别分布（D，蓝色，虚线）来训练生成对抗网络，以便区分数据生成分布pm（G）中的样本与数据生成分布pm（G）中的样本 （绿色实线）。较低的水平线是从中采样z的域，在这种情况下是均匀的。 上面的水平线是a的一部分。 向上的箭头表示映射a = G（z）如何将非均匀分布pg施加到已转换的样本上。 G在pg的高密度区域中收缩，并在pg的低密度区域中扩展。（a）考虑收敛附近的对抗对：pg与piata相似，D是部分准确的分类器。（b）在算法的内循环中 训练D以从数据中区分样本，收敛到 D *(a)=

After an update to G, gradient of D has guided G(z) to flow to regions that are more likelyto be classified as data.(d) After several steps of training, if G and D have enough capacity, they will reach apoint at which both cannot improve because Pg = Pdata. The discriminator is unable to differentiate betweenthe two distributions, i.e.D(x) = 1/2.

更新G之后，D的梯度会引导G（z）流动到更有可能被归类为数据的区域。（d）经过几步训练，如果G和D具有足够的能力，它们将达到 两者都无法改善，因为Pg = Pdata。 判别器无法区分两个分布，即D（x）= 1/2

4 Theoretical Results
The generator G implicitly defines a probability distribution pg as the distribution of the samplesG(z )obtained when z ~ pz.Therefore, we would like Algorithm 1 to converge to a good estimatorof Pdata， if given enough capacity and training time. The results of this section are done in a non-parametric setting, e.g. we represent a model with infinite capacity by studying convergence in thespace of probability density functions.
We will show in section 4.1 that this minimax game has a global optimum for Pg = Pdata. We willthen show in section 4.2 that Algorithm 1 optimizes Eq 1, thus obtaining the desired result.

4理论结果
生成器G隐含地定义了概率分布pg为当z〜pz时获得的样本G（z）的分布。因此，如果给定足够的容量和训练时间，我们希望算法1收敛到Pdata的良好估计量。 本部分的结果是在非参数设置中完成的，例如 我们通过研究概率密度函数空间中的收敛性来表示具有无限容量的模型。
我们将在第4.1节中显示，此minimax游戏具有Pg = Pdata的全局最优值。 然后我们将在第4.2节中展示算法1优化方程式1，从而获得所需的结果。
就先到这儿吧，后面再说。