Generative Modeling via Drifting: Main Results Explained

This paper introduces Drifting Models, a new paradigm for generative modeling that shifts the iterative process from inference time to training time, enabling high-quality one-step generation.

Core Innovation: Training-Time Distribution Evolution

Unlike diffusion/flow models that perform iterative updates during inference, Drifting Models evolve the pushforward distribution during training:

Figure 1: Drifting Model Concept

• A network f maps prior distribution p_prior to pushforward distribution q
• Training iterates models {f_i} → sequences of pushforward distributions {q_i}
• Drifting field approaches zero when q matches p_data

Figure 2: Sample Drifting Mechanism

• Generated samples are attracted by data distribution (blue)
• Generated samples are repulsed by sample distribution (orange)
• Drift vector: V = V_p+ - V_q-

Training Objective

The key insight is using an antisymmetric drifting field where:

• V_{p,q}(x) = -V_{q,p}(x)
• When distributions match (p = q), drift becomes zero for all samples

Training loss minimizes drift:

L = E_ε[‖f_θ(ε) - stopgrad(f_θ(ε) + V_{p,q_θ}(f_θ(ε)))‖²]

Experimental Results

Toy Experiments Demonstrate Robustness

Figure 3: Distribution Evolution

• Shows 2D case where q evolves toward bimodal p
• No mode collapse even when initialized on single mode

Figure 4: Sample Evolution

• Loss (‖V‖²) decreases as distribution converges
• Demonstrates training stability

State-of-the-Art ImageNet Results

Table 5: Latent Space Generation (256×256)

Table 6: Pixel Space Generation (256×256)

Key Achievements

1. State-of-the-art 1-NFE performance: FID 1.54 (latent) and 1.61 (pixel) outperform all previous single-step methods
2. Competitive with multi-step models: Results rival even expensive multi-step diffusion models
3. Robust to mode collapse: Toy experiments show stability across different initializations
4. Beyond images: Successful application to robotics control tasks, matching/exceeding 100-NFE diffusion policy with 1-NFE

Technical Contributions

• Feature-space training: Uses self-supervised encoders (MAE, MoCo, SimCLR) for rich gradient signals
• Classifier-Free Guidance: Naturally supports CFG through training-time negative sampling
• Multi-scale features: Extracts drifting losses across multiple network stages
• Normalization techniques: Ensures stable training across different feature encoders

The work opens new possibilities for efficient generative modeling by reframing iterative neural network training as a distribution evolution mechanism, fundamentally different from the differential equation approach of diffusion/flow models.