Neural Variational Inference: Blackbox Mode

In the previous post we covered Stochastic VI: an efficient and scalable variational inference method for exponential family models. However, there’re many more distributions than those belonging to the exponential family. Inference in these cases requires significant amount of model analysis. In this post we consider Black Box Variational Inference by Ranganath et al. This work just as the previous one comes from David Blei lab — one of the leading researchers in VI. And, just for the dessert, we’ll touch upon another paper, which will finally introduce some neural networks in VI.

Blackbox Variational Inference

As we have learned so far, the goal of VI is to maximize the ELBO L(Θ,Λ)$\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })$. When we maximize it by Λ$\mathrm{\Lambda }$, we decrease the gap between the marginal likelihood of the model considered logp(x∣Θ)$\log p(x\mid \mathrm{\Theta })$, and when we maximize it by Θ$\mathrm{\Theta }$ we acltually fit the model. So let’s concentrate on optimizing this objective:

L(Θ,Λ)=Eq(z∣x,Λ)[logp(x,z∣Θ)logq(z∣x,Λ)]$$\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })={\mathbb{E}}_{q(z\mid x,\mathrm{\Lambda })}[\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })]$$

Let’s find gradients of this objective:

ΛL(Θ,Λ)=Λ∫q(z∣x,Λ)[logp(x,z∣Θ)logq(z∣x,Λ)]dz=∫Λq(z∣x,Λ)[logp(x,z∣Θ)logq(z∣x,Λ)]dz∫q(z∣x,Λ)Λlogq(z∣x,Λ)dz=Eq[Λq(z∣x,Λ)q(z∣x,Λ)logp(x,z∣Θ)q(z∣x,Λ)]∫q(z∣x,Λ)Λq(z∣x,Λ)q(z∣x,Λ)dz=Eq[Λlogq(z∣x,Λ)logp(x,z∣Θ)q(z∣x,Λ)]∫Λq(z∣x,Λ)dz=Eq[Λlogq(z∣x,Λ)logp(x,z∣Θ)q(z∣x,Λ)]Λ∫q(z∣x,Λ)dz=1=Eq[Λlogq(z∣x,Λ)logp(x,z∣Θ)q(z∣x,Λ)]$$\begin{array}{rl}{\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })& ={\mathrm{}}_{\mathrm{\Lambda }}\int q(z\mid x,\mathrm{\Lambda })[\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })]dz\\ & =\int {\mathrm{}}_{\mathrm{\Lambda }}q(z\mid x,\mathrm{\Lambda })[\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })]dz\int q(z\mid x,\mathrm{\Lambda }){\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })dz\\ & ={\mathbb{E}}_q[\frac{{\mathrm{}}_{\mathrm{\Lambda }}q(z\mid x,\mathrm{\Lambda })}{q(z\mid x,\mathrm{\Lambda })}\log \frac{p(x,z\mid \mathrm{\Theta })}{q(z\mid x,\mathrm{\Lambda })}]\int q(z\mid x,\mathrm{\Lambda })\frac{{\mathrm{}}_{\mathrm{\Lambda }}q(z\mid x,\mathrm{\Lambda })}{q(z\mid x,\mathrm{\Lambda })}dz\\ & ={\mathbb{E}}_q[{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })\log \frac{p(x,z\mid \mathrm{\Theta })}{q(z\mid x,\mathrm{\Lambda })}]\int {\mathrm{}}_{\mathrm{\Lambda }}q(z\mid x,\mathrm{\Lambda })dz\\ & ={\mathbb{E}}_q[{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })\log \frac{p(x,z\mid \mathrm{\Theta })}{q(z\mid x,\mathrm{\Lambda })}]{\mathrm{}}_{\mathrm{\Lambda }}\stackrel{=1}{\stackrel{}{\int q(z\mid x,\mathrm{\Lambda })dz}}\\ & ={\mathbb{E}}_q[{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })\log \frac{p(x,z\mid \mathrm{\Theta })}{q(z\mid x,\mathrm{\Lambda })}]\end{array}$$

In statistics Λlogq(z∣x,Λ)${\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })$ is known as score function. For more on this “trick” see a blogpost by Shakir Mohamed. In many cases of practical interest logp(x,z,∣Θ)$\log p(x,z,\mid \mathrm{\Theta })$ is too complicated to compute this expectation in closed form. Recall that we already used stochastic optimization successfully, so we can settle with just an estimate of true gradient. We get one by approximating the expectation using Monte-Carlo estimates using L$L$ samples z(l)～q(z∣x,Λ)$z^{(l)}～q(z\mid x,\mathrm{\Lambda })$ (in practice we sometimes use just L=1$L=1$ sample. We expect correct averaging to happen automagically due to use of minibatches):

ΛL(Θ,Λ)≈1L∑l=1LΛlogq(z(l)∣x,Λ)logp(x,z(l)∣Θ)q(z(l)∣x,Λ)$${\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })\approx \frac{1}{L}\sum _{l=1}^L{\mathrm{}}_{\mathrm{\Lambda }}\log q(z^{(l)}\mid x,\mathrm{\Lambda })\log \frac{p(x,z^{(l)}\mid \mathrm{\Theta })}{q(z^{(l)}\mid x,\mathrm{\Lambda })}$$

For model parameters Θ$\mathrm{\Theta }$ gradients look even simpler, as we don’t need to differentiate w.r.t. expectation distribution’s parameters:

ΘL(Θ,Λ)=EqΘlogp(x,z∣Θ)≈1L∑l=1LΘlogp(x,z(l)∣Θ)$$\begin{array}{rl}{\mathrm{}}_{\mathrm{\Theta }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })& ={\mathbb{E}}_q{\mathrm{}}_{\mathrm{\Theta }}\log p(x,z\mid \mathrm{\Theta })\approx \frac{1}{L}\sum _{l=1}^L{\mathrm{}}_{\mathrm{\Theta }}\log p(x,z^{(l)}\mid \mathrm{\Theta })\end{array}$$

We can even “naturalize” these gradients by premultiplying by the inverse Fisher Information Matrix I(Λ)1$\mathcal{I}(\mathrm{\Lambda }{)}^1$. And that’s it! Much simpler than before, right? Of course, there’s no free lunch, so there must be a catch… And there is: performance of stochastic optimization methods crucially depends on the variance of gradient estimators. It makes perfect sense: the higher the variance — the less information about the step direction we get. And unfortunately, in practice the aforementioned estimator based on the score function has impractically high variance. Luckily, in Monte Carlo community there are many variance reductions techniques known, we now describe some of them.

The first technique we’ll describe is Rao-Blackwellization. The idea is simple: if it’s possible to compute the expectation w.r.t. some of random variables, you should do it. If you think of it, it’s an obvious advice as you essentially reduce amount of randomness in your Monte Carlo estimates. But let’s put it more formally: we use chain rule to rewrite joint expectation as marginal expectation of conditional one:

EX,Yf(X,Y)=EX[EY∣Xf(X,Y)]$${\mathbb{E}}_{X,Y}f(X,Y)={\mathbb{E}}_X[{\mathbb{E}}_{Y\mid X}f(X,Y)]$$

Let’s see what happens with variance (in scalar case) when we estimate expectation of EY∣Xf(X,Y)${\mathbb{E}}_{Y\mid X}f(X,Y)$ instead of expectation of f(X,Y)$f(X,Y)$:

VarX(EY∣Xf(X,Y))=E(EY∣Xf(X,Y))2(EX,Yf(X,Y))2=VarX,Y(f(X,Y))EX(EY∣Xf(X,Y)2(EY∣Xf(X,Y))2)=VarX,Y(f(X,Y))EXVarY∣X(f(X,Y))$$\begin{array}{rl}{\text{Var}}_X({\mathbb{E}}_{Y\mid X}f(X,Y))& =\mathbb{E}({\mathbb{E}}_{Y\mid X}f(X,Y){)}^2({\mathbb{E}}_{X,Y}f(X,Y){)}^2\\ & ={\text{Var}}_{X,Y}(f(X,Y)){\mathbb{E}}_X({\mathbb{E}}_{Y\mid X}f(X,Y{)}^2({\mathbb{E}}_{Y\mid X}f(X,Y){)}^2)\\ & ={\text{Var}}_{X,Y}(f(X,Y)){\mathbb{E}}_X{\text{Var}}_{Y\mid X}(f(X,Y))\end{array}$$

This formula says that Rao-Blackwellizing an estimator reduces its variance by EXVarY∣X(f(X,Y))${\mathbb{E}}_X{\text{Var}}_{Y\mid X}(f(X,Y))$. Indeed, you can think of this term as of a measure of how much information Y$Y$ contains about X$X$ that’s relevant to computing f(X,Y)$f(X,Y)$. Suppose Y=X$Y=X$: then you have EXf(X,X)${\mathbb{E}}_{X}f(X,X)$, and taking expectation w.r.t. Y$Y$ does not reduce amount of randomness in the estimator. And this is what the formula tells us as VarY∣Xf(X,Y)${\text{Var}}_{Y\mid X}f(X,Y)$ would be 0 in this case. Here’s another example: suppose f$f$ does not use X$X$ at all: then only randomness in Y$Y$ affects the estimate, and after Rao-Blackwellization we expect the variance to drop to 0. And the formula agrees with out expectations as EXVarY∣Xf(X,Y)=VarYf(X,Y)${\mathbb{E}}_X{\text{Var}}_{Y\mid X}f(X,Y)={\text{Var}}_{Y}f(X,Y)$ for any X$X$ since f(X,Y)$f(X,Y)$ does not depend on X$X$.

Next technique is Control Variates, which is slightly less intuitive. The idea is that we can add zero-mean function h(X)$h(X)$ that’ll preserve the expectation, but reduce the variance. Again, for a scalar case

Var(f(X)αh(X))=Var(f(X))2αCov(f(X),h(X))+α2Var(f(X))$$\text{Var}(f(X)\alpha h(X))=\text{Var}(f(X))2\alpha \text{Cov}(f(X),h(X))+{\alpha }^2\text{Var}(f(X))$$

Optimal α=Cov(f(X),h(X))Var(f(X))${\alpha }^{}=\frac{\text{Cov}(f(X),h(X))}{\text{Var}(f(X))}$. This formula reflects an obvious fact: if we want to reduce the variance, h(X)$h(X)$ must be correlated with f(X)$f(X)$. Sign of correlation does not matter, as α${\alpha }^{}$ will adjust. BTW, in reinforcement learning α$\alpha$ is called baseline.

As we already have learned, Eq(z∣x,Λ)Λlogq(z∣x,Λ)=0${\mathbb{E}}_{q(z\mid x,\mathrm{\Lambda })}{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })=0$, so the score function is a good candidate for h(x)$h(x)$. Therefore our estimates become

ΛL(Θ,Λ)≈1L∑l=1LΛlogq(z(l)∣x,Λ)°(logp(x,z(l)∣Θ)q(z(l)∣x,Λ)α)$${\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })\approx \frac{1}{L}\sum _{l=1}^L{\mathrm{}}_{\mathrm{\Lambda }}\log q(z^{(l)}\mid x,\mathrm{\Lambda })°(\log \frac{p(x,z^{(l)}\mid \mathrm{\Theta })}{q(z^{(l)}\mid x,\mathrm{\Lambda })}{\alpha }^{})$$

Where °$°$ is pointwise multiplication and α$\alpha$ is a vector of |Λ|$|\mathrm{\Lambda }|$ components with αi${\alpha }_i$ being a baseline for variational parameter Λi${\mathrm{\Lambda }}_i$:

αi=Cov(Λilogq(z∣x,Λ)(logp(x,z∣Θ)logq(z∣x,Λ)),Λilogq(z∣x,Λ))Var(Λilogq(z∣x,Λ)(logp(x,z∣Θ)logq(z∣x,Λ)))$${\alpha }_i^{}=\frac{\text{Cov}({\mathrm{}}_{{\mathrm{\Lambda }}_i}\log q(z\mid x,\mathrm{\Lambda })(\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })),{\mathrm{}}_{{\mathrm{\Lambda }}_i}\log q(z\mid x,\mathrm{\Lambda }))}{\text{Var}({\mathrm{}}_{{\mathrm{\Lambda }}_i}\log q(z\mid x,\mathrm{\Lambda })(\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })))}$$

Neural Variational Inference and Learning

Hoooray, neural networks! In this section I’ll briefly describe a variance reduction technique coined by A. Mnih and K. Gregor in Neural Variational Inference and Learning in Belief Networks. The idea is surprisingly simple: why not learn a baseline α$\alpha$ using a neural network?

ΛL(Θ,Λ)≈1L∑l=1LΛlogq(z(l)∣x,Λ)°(logp(x,z(l)∣Θ)q(z(l)∣x,Λ)αα(x))$${\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })\approx \frac{1}{L}\sum _{l=1}^L{\mathrm{}}_{\mathrm{\Lambda }}\log q(z^{(l)}\mid x,\mathrm{\Lambda })°(\log \frac{p(x,z^{(l)}\mid \mathrm{\Theta })}{q(z^{(l)}\mid x,\mathrm{\Lambda })}{\alpha }^{}\alpha (x))$$

Where α(x)$\alpha (x)$ is a neural network trained to minimize

Eq(z∣x,Λ)(logp(x,z(l)∣Θ)q(z(l)∣x,Λ)αα(x))2$${\mathbb{E}}_{q(z\mid x,\mathrm{\Lambda })}{(\log \frac{p(x,z^{(l)}\mid \mathrm{\Theta })}{q(z^{(l)}\mid x,\mathrm{\Lambda })}{\alpha }^{}\alpha (x))}^2$$

What’s the motivation of this objective? The gradient step of ΛL(Θ,Λ)${\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })$ can be seen as pushing q(z∣x,Λ)$q(z\mid x,\mathrm{\Lambda })$ towards p(x,z∣Θ)$p(x,z\mid \mathrm{\Theta })$. Since q$q$ has to be normalized like any other proper distribution, it’s actually pushed towards the true posterior p(z∣x,Θ)$p(z\mid x,\mathrm{\Theta })$. We can rewrite the gradient ΛL(Θ,Λ)${\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })$ as

ΛL(Θ,Λ)=Eq[Λlogq(z∣x,Λ)(logp(x,z∣Θ)logq(z∣x,Λ))]=Eq[Λlogq(z∣x,Λ)(logp(z∣x,Θ)logq(z∣x,Λ)+logp(x∣Θ))]$$\begin{array}{rl}{\mathrm{}}_{\mathrm{\Lambda }}\mathcal{L}(\mathrm{\Theta },\mathrm{\Lambda })& ={\mathbb{E}}_q[{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })(\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda }))]\\ & ={\mathbb{E}}_q[{\mathrm{}}_{\mathrm{\Lambda }}\log q(z\mid x,\mathrm{\Lambda })(\log p(z\mid x,\mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })+\log p(x\mid \mathrm{\Theta }))]\end{array}$$

While this additional logp(x∣Θ)$\log p(x\mid \mathrm{\Theta })$ term does not contribute to the expectation, it affects the variance on the estimator. Therefore, α(x)$\alpha (x)$ is supposed to estimate the marginal log-likelihood logp(x∣Θ)$\log p(x\mid \mathrm{\Theta })$.

The paper also lists several other variance reduction techniques that can be used in combination with the neural network-based baseline:

• Constant baseline — analogue of Control Variates, uses running average of logp(x,z∣Θ)logq(z∣x,Λ)$\log p(x,z\mid \mathrm{\Theta })\log q(z\mid x,\mathrm{\Lambda })$ as a baseline
• Variance normalization — normalizes the learning signal to unit variance, equivalent to adaptive learning rate
• Local learning signals — falls out of the scope of this post as requires it model-specific analysis and alternations, and can’t be used in Blackbox regime