What are Normalizing Flows?

This short tutorial covers the basics of normalizing flows, a technique used in machine learning to build up complex probability distributions by transforming simple ones.

Timestamps:
0:00 - Intro
0:33 - Bijective transformation
1:18 - Change of variables formula
2:08 - Jacobian determinant
4:28 - Generative model likelihood
5:49 - Comparison with VAEs & GANs
6:52 - NICE architecture: triangular Jacobian & coupling layers
9:23 - Scaling matrix
10:26 - Extensions

Papers to check out:
NICE: Non-linear Independent Components Estimation (https://arxiv.org/abs/1410.8516)
Density estimation using Real NVP (https://arxiv.org/abs/1605.08803)
Glow: Generative Flow with Invertible 1x1 Convolutions (https://arxiv.org/abs/1807.03039)
Variational Inference with Normalizing Flows (https://arxiv.org/abs/1505.05770)
Improving Variational Inference with Inverse Autoregressive Flow (https://arxiv.org/abs/1606.04934)
Masked Autoregressive Flow for Density Estimation (https://arxiv.org/abs/1705.07057)
MADE: Masked Autoencoder for Distribution Estimation (https://arxiv.org/abs/1502.03509)
Discrete Flows: Invertible Generative Models of Discrete Data (https://arxiv.org/abs/1905.10347)

Earlier work on flows:
A family of non-parametric density estimation algorithms (https://math.nyu.edu/faculty/tabak/publications/Tabak-Turner.pdf)

Additional reading:
https://deepgenerativemodels.github.io/notes/flow/
https://blog.evjang.com/2018/01/nf1.html
https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html
http://akosiorek.github.io/ml/2018/04/03/norm_flows.html

Special thanks to Alex Beatson, Geoffrey Roeder, Yaniv Ovadia, Sachin Ravi, and Ryan Adams for helpful feedback on this video.

Video style inspired by 3Blue1Brown

Music: Trinkets by Vincent Rubinetti

Links:
YouTube: https://www.youtube.com/ariseffai
Twitter: https://twitter.com/ari_seff
Homepage: https://www.ariseff.com

If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:
Venmo: https://venmo.com/ariseff
PayPal: https://www.paypal.me/ariseff