This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation A. L. Yuille Smith-Kettlewell Eye Research Institute, 2318 Fillmore Street, San Francisco, CA 94115, USA. Tel. (415) 345-2144. Fax. (415) 345-8455. Email [email protected] Abstract This paper introduces a class of discrete iterative algorithms which are provably convergent alternatives to belief propagation (BP) and generalized belief propa- gation (GBP). Our work builds on recent results by Yedidia, Freeman and Weiss (2000) who showed that the fixed points of BP and GBP algorithms correspond to extrema of the Bethe and Kikuchi free energies respectively. We obtain two algorithms by applying CCCP to the Bethe and Kikuchi free energies respectively (CCCP is a procedure, introduced here, for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part). We implement our CCCP algorithms on 2D and 3D spin glasses and compare their results to BP and GBP. Our simulations show that the CCCP algorithms are stable and converge very quickly (the speed of CCCP is similar to that of BP/GBP). Unlike CCCP, BP will often not converge for these problems (GBP usually, but not always, con- verges). The results found by CCCP applied to the Bethe or Kikuchi free energies are equivalent, or slightly better than, those found by BP or GBP respectively (when BP/GBP converge). Note that for these, and other problems, BP/GBP give very accurate results (see Yedidia et al 2000) and failure to converge is their major error mode. Finally, we point out that our algorithms have a large range of inference and learning applications. To appear in Neural Computation 1 Introduction Recent work by Yedidia, Freeman and Weiss (2000) unified two approaches to sta- tistical inference. They related the highly successful belief propagation (BP) al- gorithms (Pearl 1988) to variational methods from statistical physics and, in par- ticular, to the Bethe and Kikuchi free energies (Domb and Green 1972). These BP algorithms typically give highly accurate results (Yedidia et al 2000). But BP algorithms do not always converge and, indeed, failure to converge is their major error mode. This paper develops new algorithms which are guaranteed to converge to extrema of the Bethe or Kikuchi free energies and hence are alternatives to belief propagation. Belief propagation (Pearl 1988) is equivalent to the sum-product algorithm devel- oped by Gallager to decode Low Density Parity Codes (LDPC) (Gallager 1963). In recent years, see (Forney 2001) for a review, the coding community has shown great interest in sum-product algorithms and LDPC’s. It is predicted that this combi- nation will enable the coding community to design practical codes which approach the Shannon performance limit (Cover and Thomas 1991) while requiring only lim- ited computation. In particular, it has been shown that the highly successful turbo codes (Berrou et al 1993) can also be interpreted in terms of BP (McEliece...
View Full Document
This note was uploaded on 01/29/2012 for the course CAP 6617 taught by Professor Staff during the Fall '11 term at University of Florida.
- Fall '11