Coursehero >> California >> UC Irvine >> ICS 2006

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

IS THIS A DRAFT VERSION. FINAL VERSION TO BE PUBLISHED AT NIPS 06 A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation Yee Whye Teh School of Computing National University of Singapore 3 Science Drive 2, Singapore 117543 tehyw@comp.nus.edu.sg Dave Newman Bren School of Information and Computer Science UC Irvine Irvine, CA 92697-3425 newman@uci.edu Max Welling Bren School of Information and Computer Science UC Irvine Irvine, CA 92697-3425 welling@ics.uci.edu Abstract Latent Dirichlet allocation (LDA) is a Bayesian network that has recently gained much popularity in applications ranging from document modeling to computer vision. Due to the large scale nature of these applications, current inference procedures like variational Bayes and Gibb sampling have been found lacking. In this paper we propose the collapsed variational Bayesian inference algorithm for LDA, and show that it is computationally ef cient, easy to implement and signi cantly more accurate than standard variational Bayesian inference for LDA. 1 Introduction Bayesian networks with discrete random variables form a very general and useful class of probabilistic models. In a Bayesian setting it is convenient to endow these models with Dirichlet priors over the parameters as they are conjugate to the multinomial distributions over the discrete random variables [1]. This choice has important computational advantages and allows for easy inference in such models. A class of Bayesian networks that has gained signi cant momentum recently is latent Dirichlet allocation (LDA) [2], otherwise known as multinomial PCA [3]. It has found important applications in both text modeling [4, 5] and computer vision [6]. Training LDA on a large corpus of several million documents can be a challenge and crucially depends on an ef cient and accurate inference procedure. A host of inference algorithms have been proposed, ranging from variational Bayesian (VB) inference [2], expectation propagation (EP) [7] to collapsed Gibbs sampling [5]. Perhaps surprisingly, the collapsed Gibbs sampler proposed in [5] seem to be the preferred choice in many of these large scale applications. In [8] it is observed that EP is not ef cient enough to be practical while VB suffers from a large bias. However, collapsed Gibbs sampling also has its own problems: one needs to assess convergence of the Markov chain and to have some idea of mixing times to estimate the number of samples to collect, and to identify coherent topics across multiple samples. In practice one often ignores these issues and collects as many samples as is computationally feasible, while the question of topic identi cation is often sidestepped by using just 1 sample. Hence there still seems to be a need for more ef cient, accurate and deterministic inference procedures. In this paper we will leverage the important insight that a Gibbs sampler that operates in a collapsed space where the parameters are marginalized out mixes much better than a Gibbs sampler that samples parameters and latent topic variables simultaneously. This suggests that the parameters and latent variables are intimately coupled. As we shall see in the following, marginalizing out the parameters induces new dependencies between the latent variables (which are conditionally independent given the parameters), but these dependencies are spread out over many latent variables. This implies that the dependency between any two latent variables is expected to be small. This is precisely the right setting for a mean eld (i.e. fully factorized variational) approximation: a particular variable interacts with the remaining variables only through summary statistics called the eld, and the impact of any single variable on the eld is very small [9]. Note that this is not true in the joint space of parameters and latent variables because uctuations in parameters can have a signi cant impact on latent variables. We thus conjecture that the mean eld assumptions are much better satis ed in the collapsed space of latent variables than in the joint space of latent variables and parameters. In this paper we leverage this insight and propose a collapsed variational Bayesian (CVB) inference algorithm. In theory, the CVB algorithm requires the calculation of very expensive averages. However, the averages only depend on sums of independent Bernoulli variables, and thus are very closely approximated with Gaussian distributions (even for relatively small sums). Making use of this approximation, the nal algorithm is computationally ef cient, easy to implement and signi cantly more accurate than standard VB. 2 Approximate Inference in Latent Dirichlet Allocation LDA models each document as a mixture over topics. We assume there are K latent topics, each being a multinomial distribution over a W word vocabulary. For document j, we rst draw a mixing proportion j = { jk } over K topics from a symmetric Dirichlet with parameter . For the ith word in the document, a topic zij is drawn with topic k chosen with probability jk , then word xij is drawn from the zij th topic, with xij taking on value w with probability kw . Finally, a symmetric Dirichlet prior with parameter is placed on the topic parameters k = { kw }. The full joint distribution over all parameters and variables is: p(x, z, , | , ) = (K ) ( )K j=1 D 1+njk K k=1 jk K k=1 (W ) ( )W W w=1 1+n kw kw (1) where njkw = #{i : xij = w, zij = k}, and dot means the corresponding index is summed out: n kw = j njkw , and njk = w njkw . Given the observed words x = {xij } the task of Bayesian inference is to compute the posterior distribution over the latent topic indices z = {zij }, the mixing proportions = { j } and the topic parameters = { k }. There are three current approaches, variational Bayes (VB) [2], expectation propagation [7] and collapsed Gibbs sampling [5]. We review the VB and collapsed Gibbs sampling methods here as they are the most popular methods and to motivate our new algorithm which combines advantages of both. 2.1 Variational Bayes Standard VB inference upper bounds the negative log marginal likelihood log p(x| , ) using the variational free energy: log p(x| , ) F( (z, , )) = Eq [ log p(x, z, , | , )] H( (z, , )) q q (2) with q (z, , ) an approximate posterior, H( (z, , )) = Eq [ log q (z, , )] the variational en q tropy, and q (z, , ) assumed to be fully factorized: q (z, , ) = ij q (zij | ij ) j q ( j | j ) k q ( k | k ) (3) q (zij | ij ) is multinomial with parameters ij and q ( j | j ), q ( k | k ) are Dirichlet with parameters k respectively. Optimizing F( ) with respect to the variational parameters gives us a set of j and q updates guaranteed to improve F( ) at each iteration and converges to a local minimum: q jk = + kw = + i ijk 1 (xij = w) ijk w (4) (5) kw ) (6) ij ijk exp ( jk ) + ( kxij ) ( where (y) = log (y) y is the digamma function and 1 is the indicator function. Although ef cient and easily implemented, VB can potentially lead to very inaccurate results. Notice that the latent variables z and parameters , can be strongly dependent in the true posterior p(z, , |x) through the cross terms in (1). This dependence is ignored in VB which assumes that latent variables and parameters are independent instead. As a result, the VB upper bound on the negative log marginal likelihood can be very loose, leading to inaccurate estimates of the posterior. 2.2 Collapsed Gibbs Sampling Standard Gibbs sampling, which iteratively samples latent variables z and parameters , , can potentially have slow convergence due again to strong dependencies between the parameters and latent variables. Collapsed Gibbs sampling improves upon Gibbs sampling by marginalizing out and instead, therefore dealing with them exactly. The marginal distribution over x and z is p(z, x| , ) = j (K ) (K +nj ) k ( +njk ) ( ) k (W ) (W +n k ) w ( +n kw ) ( ) (7) Given the current state of all but one variable zij , the conditional probability of zij is: p(zij = k|z ij , x, , ) = ( + n ij )( + n ijij )(W + n ij ) 1 jk kx k K k =1 ( + n ij )( + n ij ij )(W + n ij ) 1 jk k x k (8) where the superscript ij means the corresponding variables or counts with xij and zij excluded, and the denominator is just a normalization. The conditional distribution of zij is multinomial with simple to calculate probabilities, so the programming and computational overhead is minimal. Collapsed Gibbs sampling has been observed to converge quickly [5]. Notice from (8) that zij depends on z ij only through the counts n ij , n ijij , n ij . In particular, the dependence of zij on jk kx k any particular other variable zi j is very weak, especially for large datasets. As a result we expect the convergence of collapsed Gibbs sampling to be fast [10]. However, as with other MCMC samplers, and unlike variational inference, it is often hard to diagnose convergence, and a suf ciently large number of samples may be required to reduce sampling noise. The argument of rapid convergence of collapsed Gibbs sampling is reminiscent of the argument for when mean eld algorithms can be expected to be accurate [9]. The counts n ij , n ijij , n ij act as jk kx k elds through which zij interacts with other variables. In particular, averaging both sides of (8) by p(z ij |x, , ) gives us the Callen equations, a set of equations that the true posterior must satisfy: p(zij = k|x, , ) = Ep(z ij |x, , ) ( +n ij )( +n ijij )(W +n ij ) 1 jk kx k K ij ij ij 1 k =1 ( +njk )( +n k xij )(W +n k ) (9) Since the latent variables are already weakly dependent on each other, it is possible to replace (9) by a set of mean eld equations where latent variables are assumed independent and still expect these equations to be accurate. This is the idea behind the collapsed variational Bayesian inference algorithm of the next section. 3 Collapsed Variational Bayesian Inference for LDA We derive a new inference algorithm for LDA combining the advantages of both standard VB and collapsed Gibbs sampling. It is a variational algorithm which, instead of assuming independence, models the dependence of the parameters on the latent variables in an exact fashion. On the other hand we still assume that latent variables are mutually independent. This is not an unreasonable assumption to make since as we saw they are only weakly dependent on each other. We call this algorithm collapsed variational Bayesian (CVB) inference. There are two ways to deal with the parameters in an exact fashion, the rst is to marginalize them out of the joint distribution and to start from (7), the second is to explicitly model the posterior of , given z and x without any assumptions on its form. We will show that these two methods are equivalent. The only assumption we make in CVB is that the latent variables z are mutually independent, thus we approximate the posterior as: q (z, , ) = q ( , |z) ij q (zij | ij ) (10) where q (zij | ij ) is multinomial with parameters ij . The variational free energy becomes: F( (z) ( , |z)) = Eq(z) ( , |z) [ log p(x, z, , | , )] H( (z) ( , |z)) qq qq q =Eq(z) [Eq( , |z) [ log p(x, z, , | , )] H( ( , |z))] H( (z)) q q (11) We minimize the variational free energy with respect to q ( , |z) rst, followed by q (z). Since we do not restrict the form of q ( , |z), the minimum is achieved at the true posterior q ( , |z) = p( , |x, z, , ), and the variational free energy simpli es to: F( (z)) q q ( , |z) min F( (z) ( , |z)) = Eq(z) [ log p(x, z| , )] H( (z)) qq q (12) We see that CVB is equivalent to marginalizing out , before approximating the posterior over z. As CVB makes a strictly weaker assumption on the variational posterior than standard VB, we have F( (z)) F( (z)) q q q ( ) ( ) q qqq min F( (z) ( ) ( )) (13) and thus CVB is a better approximation than standard VB. Finally, we derive the updates for the variational parameters ij . Minimizing (12) with respect to ijk , we get ijk = q (zij = k) = exp Eq(z ij ) [p(x, z ij , zij = k| , )] K k =1 exp Eq(z ij ) [p(x, z ij , zij = k | , )] n 1 (14) Plugging in (7), expanding log ( +n) = l=0 log( + l) for positive reals and positive integers ( ) n, and cancelling terms appearing both in the numerator and denominator, we get ijk = exp Eq(z ij ) [log( +n ij ) + log( +n ijij ) log(W +n ij )] jk kx k K k =1 exp Eq(z ij ) [log( +n ij ) + log( +n ij ij ) log(W +n ij )] jk k x k (15) 3.1 Gaussian approximation for CVB Inference For completeness, we describe how to compute each expectation term in (15) exactly in the appendix. This exact implementation of CVB is computationally too expensive to be practical, and we propose instead to use a simple Gaussian approximation which works very accurately and which requires minimal computational costs. In this section we describe the Gaussian approximation applied to Eq [log( + n ij )]; the other jk two expectation terms are similarly computed. Assume that nj Notice 0. that n ij = jk 1 (zi j = k) is a sum of a large number independent Bernoulli variables 1 (zi j = k) each i =i with mean parameter i jk , thus it can be accurately approximated by a Gaussian. The mean and variance are given by the sum of the means and variances of the individual Bernoulli variables: Eq [n ij ] = jk i =i i jk Varq [n ij ] = jk i =i i jk (1 i jk ) (16) We further approximate the function log( + n ij ) using a second-order Taylor expansion about jk Eq [n ij ], and evaluate its expectation under the Gaussian approximation: jk Eq [log( + n ij )] log( + Eq [n ij ]) jk jk Varq (n ij ) jk 2( + Eq [n ij ])2 jk (17) Because Eq [n ij ] 0, the third derivative is small and the Taylor series approximation is very jk accurate. In fact, we have found experimentally that the Gaussian approximation works very well even when nj is small. The reason is that we often have i jk being either close to 0 or 1 thus the variance of n ij is small relative to its mean and the Gaussian approximation will be accurate. jk Finally, plugging (17) into (15), we have our CVB updates: ijk +Eq [n ij ] jk exp 2( +E +Eq [n ijij ] kx ij 2 q [njk ]) W +Eq [n ij ] k ij 1 Varq (n ij ) jk Varq (n ij ) kx 2( +Eq [n ij ])2 kx ij + Varq (n ij ) k 2( +Eq [n ij ])2 k (18) Notice the striking correspondence between (18), (8) and (9), showing that CVB is indeed the mean eld version of collapsed Gibbs sampling. In particular, the rst line in (18) is obtained from (8) by replacing the elds n ij , n ijij and n ij by their means (thus the term mean eld) while the jk kx k exponentiated terms are correction factors accounting for the variance in the elds. CVB with the Gaussian approximation is easily implemented and has minimal computational costs. By keeping track of the mean and variance of njk , n kw and n k , and subtracting the mean and variance of the corresponding Bernoulli variables whenever we require the terms with xij , zij removed, the computational cost scales only as O(K) for each update to q (zij ). Further, we only need to maintain one copy of the variational posterior over the latent variable for each unique document/word pair, thus the overall computational cost per iteration of CVB scales as O(M K) where M is the total number of unique document/word pairs, while the memory requirement is O(M K). This is the same as for VB. In comparison, collapsed Gibbs sampling needs to keep track of the current sample of zij for every word in the corpus, thus the memory requirement is O(N ) while the computational cost scales as O(N K) where N is the total number of words in the corpus higher than for VB and CVB. Note however that the constant factor involved in the O(N K) time cost of collapsed Gibbs sampling is smaller than those for VB and CVB. 4 Experiments We compared the three algorithms described in the paper: standard VB, CVB and collapsed Gibbs sampling. We used two datasets: rst is KOS (www.dailykos.com), which has J = 3430 documents, a vocabulary size of W = 6909, a total of N = 467, 714 words in all the documents and on average 136 words per document. Second is NIPS (books.nips.cc) with J = 1675 documents, a vocabulary size of W = 12419, N = 2, 166, 029 words in the corpus and on average 1293 words per document. In both datasets stop words and infrequent words were removed. We split both datasets into a training set and a test set by assigning 10% of the words in each document to the test set. In all our experiments we used K = 8 number of topics for KOS and K = 40 for NIPS. We ran each algorithm on each dataset multiple times with different random initializations. Performance was measured in two ways. First using estimates of the negative log marginal likelihood or variational bounds thereof on the training set, and secondly using perplexities on the test set. Expressions for the variational bounds are given in (2) for VB and (12) for CVB. The true negative log marginal likelihood was estimated using importance sampling (IS) where samples were drawn from the collapsed variational posterior at convergence [11]. Test set perplexities are computed as Perp(xtest ) = exp( N1 log p(xtest )). For both VB and CVB, p(xtest ) was computed as follows: test p(xtest ) = ij k jk kxtest ij + Eq [njk ] jk = K + Eq [nj ] + Eq [n kw ] kw = W + Eq [n k ] (19) Note that we used estimated mean values of jk and kw [11]. For collapsed Gibbs sampling, given S samples from the posterior, we used: p(xtest ) = 1 |S| S s jk s test kx s=1 ij k ij s jk = + ns jk K + ns j s = kw + ns kw W + ns k (20) Results are summarized in Figure 1. We show histograms of nal estimates of the negative log marginal likelihoods and test set perplexities for all algorithms and both datasets. To get a feeling of the convergence speeds we also plotted these quantities as functions of the number of iterations of the algorithms. Both VB and CVB converge in about the same number of iterations. These plots con rm our intuition that CVB provides much better approximations than VB both in terms of variational bounds, and in terms of test set perplexities. Though not shown, we have also applied the exact but much slower version of CVB without the Gaussian approximation. We found that the much faster Gaussian approximation gave identical results to the exact version. We have also studied the dependence of approximation accuracies on the number of documents in the corpus. To conduct this experiment we train on 90% of the words in a (growing) subset the corpus and test on the corresponding 10% left out words. In gure Figure 2 we show both test set perplexity and free energy as a function of J. We observe that as expected the variational methods improve as J increases. However, perhaps surprisingly, CVB does not suffer as much as VB for small values of J, even though one might expect that the Gaussian approximation becomes dubious in that regime. 5 Discussion We have described a collapsed variational Bayesian (CVB) inference algorithm for LDA. The algorithm is easy to implement, computationally ef cient and more accurate than standard VB. The central insight of CVB is that instead of assuming parameters to be independent from latent variables, we treat their dependence on the topic variables in an exact fashion. Because the factorization assumptions made by CVB are weaker than those made by VB, the resulting approximation is more accurate. Computational ef ciency is achieved in CVB with a Gaussian approximation, which was found to be so accurate that there is never a need for exact summation. The idea of integrating out parameters before applying variational inference has been independently proposed by [12]. Unfortunately, because they worked in the context of general conjugateexponential family models, the approach cannot be made computationally useful and was left as an exercise in the derivation of the variational updates. Nevertheless, we believe the insights of CVB can be applied to a wider class of discrete graphical models beyond LDA. Speci c examples include various extensions of LDA [4, 13] hidden Markov models with discrete outputs, and mixedmembership models with Dirichlet distributed mixture coef cients [14]. These models all have the property that they consist of discrete random variables with Dirichlet priors on the parameters, which is the property allowing us to use the Gaussian approximation. We are also exploring applying CVB to an even more general class of models, including mixtures of Gaussians, Dirichlet processes, and hierarchical Dirichlet processes. To conclude, it is useful to related CVB to other inference algorithms. Over the years a variety of inference algorithms has been proposed based on a combination of {maximize, sample, assume independent, marginalize out} applied to both parameters and latent variables. We summarize these algorithms in Table 1, and note that CVB is located in the marginalize out parameters and assume latent variables are independent cell. A Exact Computation of Expectation Terms in (15) We can compute the expectation terms in (15) exactly as follows. Consider Eq [log( + n ij )], jk which requires computing q (n ij ) (other expectation terms are similarly computed). Note that jk n ij = i =i 1 (zi j = k) is a sum of independent Bernoulli variables 1 (zi j = k) each with mean jk parameter i jk . De ne vectors vi jk = [(1 i jk ), i jk ] , and let vjk = v1jk vn j jk be 8.4 8.3 8.2 8.1 VB CVB 8.6 VB CVB 8.4 8.2 free energy 8 7.9 7.8 7.7 free energy 20 40 60 iteration 80 100 5 8 7.8 7.6 7.6 7.5 0 7.4 0 20 40 60 iteration 80 100 12 IS CVB VB 4.5 4 3.5 3 10 IS CVB VB 8 6 2.5 2 4 1.5 1 0.5 2 0 7.55 2600 2500 2400 2300 7.6 7.65 7.7 free energy/neg.log prob. 7.75 7.8 0 7.35 2600 7.4 7.45 7.5 7.55 free energy/neg.log prob. 7.6 7.65 VB CVB Gibbs 2400 VB CVB Gibbs 2200 perplexity 2200 2100 2000 1900 perplexity 20 40 60 iteration 80 100 4 Gibbs CVB VB 3.5 3 2000 1800 1600 1800 1700 0 12 1400 0 20 40 60 iteration 80 100 10 Gibbs CVB VB 8 2.5 2 1.5 1 6 4 2 0.5 0 1400 0 1740 1780 1820 1860 1900 perplexity 1940 1980 2020 1450 1500 1550 perplexity 1600 1650 1700 Figure 1: Left panels are for KOS and right ones for NIPS. First row: variational free energies as function of number of iterations of VB and CVB. Second row: histograms of converged variational free energies across random initializations for VB and CVB, and of estimated negative log marginal likelihoods for importance sampling. Third row: test set perplexities as function of number of iterations for VB, CVB and Gibbs. Fourth row: histograms of nal test set perplexities across random initializations. The number of initializations for Gibbs sampling are scaled down to 25 (KOS) and 9 (NIPS) respectively. 3600 3400 3200 3000 VB CVB Gibbs 9.4 9.2 9 8.8 free energy 8.6 8.4 8.2 8 7.8 500 1000 1500 number of documents 2000 2500 VB CVB perplexity 2800 2600 2400 2200 2000 1800 0 7.6 0 500 1000 1500 number of documents 2000 2500 Figure 2: Test-set perplexities (left) and free energy (right) as functions of the number of documents for KOS. Parameters maximize sample assume marginalize Latent variables independent out maximize Viterbi EM ? ME ME sample stochastic EM Gibbs sampling ? collapsed Gibbs assume independent variational EM ? VB CVB marginalize out EM any MCMC EP for LDA intractable Table 1: A variety of inference algorithms for graphical models. Note that not every cell is lled in (marked by ?) while some are simply intractable. ME is the maximization-expectation algorithm of [15] and any MCMC means that we can use any MCMC sampler for the parameters once latent variables have been marginalized out. ij the convolution of all vi jk . Finally let vjk be vjk deconvolved by vijk . Then q (n ij = m) will jk ij be the (m+1)st entry in vjk . The expectation Eq [log( +n ij )] can now be computed explicitly. jk This exact implementation requires an impractical O(n2 ) time to compute Eq [log( +n ij )]. At j jk the expense of complicating the algorithm implementation, this can be improved by sparsifying the vectors vjk (setting small entries to zero) as well as other computational tricks. We propose instead the Gaussian approximation of Section 3.1, which we have found to give extremely accurate results but with minimal implementation complexity and computational cost. References [1] D. Heckerman. A tutorial on learning with Bayesian networks. In M. I. Jordan, editor, Learning in Graphical Models. Kluwer Academic Publishers, 1999. [2] D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent Dirichlet allocation. JMLR, 3, 2003. [3] W. Buntine. Variational extensions to EM and multinomial PCA. In ECML, 2002. [4] M. Rosen-Zvi, T. Grif ths, M. Steyvers, and P. Smyth. The author-topic model for authors and documents. In UAI, 2004. [5] T.L. Grif ths and M. Steyvers. Finding scienti c topics. In PNAS, 2004. [6] L. Fei-Fei and P. Perona. A Bayesian hierarchical model for learning natural scene categories. In CVPR, 2005. [7] T. P. Minka and J. Lafferty. Expectation propagation for the generative aspect model. In UAI, 2002. [8] W. Buntine and A. Jakulin. Applying discrete PCA in data analysis. In UAI, 2004. [9] M. Opper and O. Winther. From naive mean eld theory to the TAP equations. In D. Saad and M. Opper, editors, Advanced Mean Field Methods : Theory and Practice. The MIT Press, 2001. [10] G. Casella and C. P. Robert. Rao-Blackwellisation of sampling schemes. Biometrika, 83(1):81 94, 1996. [11] M.J. Beal. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003. [12] J. Sung, Z. Ghahramani, and S. Choi. Variational Bayesian EM: A second-order approach. Unpublished manuscript, 2005. [13] W. Li and A. McCallum. Pachinko allocation: DAG-structured mixture models of topic correlations. In ICML, 2006. [14] E. M. Airoldi, D. M. Blei, E. P. Xing, and S. E. Fienberg. Mixed membership stochastic block models for relational data with application to protein-protein interactions. In Proceedings of the International Biometrics Society Annual Meeting, 2006. [15] M. Welling and K. Kurihara. Bayesian K-means as a maximization-expectation algorithm. In SIAM Conference on Data Mining, 2006.

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: