Principle:LaurentMazare Tch rs Generative Adversarial Network

Knowledge Sources	LaurentMazare_Tch_rs Generative Adversarial Nets The Relativistic Discriminator
Domains	Generative Modeling, Deep Learning
Last Updated	2026-02-08 00:00 GMT

Overview

Generative adversarial networks train two competing neural networks in a minimax game where a generator learns to produce realistic samples while a discriminator learns to distinguish real data from generated fakes.

Description

A GAN consists of two neural networks trained simultaneously in opposition:

• Generator (G): Takes a random noise vector $z$ sampled from a simple prior distribution (e.g., standard normal) and transforms it into a sample intended to resemble the real data distribution. The generator never sees real data directly; it learns entirely through the discriminator's feedback.

• Discriminator (D): Receives both real data samples and generated samples and outputs a probability that the input is real. It acts as a binary classifier trained to maximize classification accuracy.

The training proceeds as a minimax game: the discriminator tries to maximize its ability to distinguish real from fake, while the generator tries to minimize the discriminator's accuracy. At equilibrium, the generator produces samples indistinguishable from real data, and the discriminator outputs 0.5 for all inputs.

Relativistic GAN is a variant where the discriminator no longer estimates the absolute probability that an input is real. Instead, it estimates the probability that a real sample is more realistic than a randomly sampled fake. This modification provides gradient signal to the generator from both real and fake samples, leading to more stable training and better sample quality.

Key training challenges include mode collapse (generator producing limited variety), training instability (oscillation or divergence), and vanishing gradients (discriminator becoming too strong too early).

Usage

GANs are applied in image synthesis, image-to-image translation, super-resolution, data augmentation, style transfer, and any task requiring generation of realistic samples from a learned data distribution.

Theoretical Basis

Standard GAN Objective:

$\underset{G}{\min }\underset{D}{\max }V(D,G)={\mathbb{𝔼}}_{x\sim p_{data}}[\log D(x)]+{\mathbb{𝔼}}_{z\sim p_z}[\log (1-D(G(z)))]$

Optimal Discriminator:

For a fixed generator $G$, the optimal discriminator is:

$D^{*}(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}$

where $p_g$ is the implicit distribution defined by the generator.

Generator Loss (non-saturating variant):

In practice, the generator is trained to maximize $\log D(G(z))$ rather than minimize $\log (1-D(G(z)))$ to avoid vanishing gradients early in training:

${{ℒ}}_G=-{\mathbb{𝔼}}_{z\sim p_z}[\log D(G(z))]$

Relativistic Discriminator:

The relativistic average discriminator (RaD) modifies the discriminator output:

$D_{Ra}(x,\tilde{x})=\sigma (C(x)-{\mathbb{𝔼}}_{\tilde{x}}[C(\tilde{x})])$

where $C(\cdot )$ is the critic (discriminator before sigmoid), $x$ is a real sample, and $\tilde{x}=G(z)$ is a generated sample.

The relativistic losses become:

${{ℒ}}_D=-{\mathbb{𝔼}}_x[\log D_{Ra}(x,\tilde{x})]-{\mathbb{𝔼}}_{\tilde{x}}[\log (1-D_{Ra}(\tilde{x},x))]$

${{ℒ}}_G=-{\mathbb{𝔼}}_{\tilde{x}}[\log D_{Ra}(\tilde{x},x)]-{\mathbb{𝔼}}_x[\log (1-D_{Ra}(x,\tilde{x}))]$

Convergence:

Under ideal conditions (infinite capacity, sufficient training), the global optimum of the minimax game is achieved when $p_g=p_{data}$, at which point the Jensen-Shannon divergence between the two distributions is zero.

Related Pages

• Implementation:LaurentMazare_Tch_rs_GAN_Example