{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

bethe_nc02J - CCCP Algorithms to Minimize the Bethe and...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation A. L. Yuille Smith-Kettlewell Eye Research Institute, 2318 Fillmore Street, San Francisco, CA 94115, USA. Tel. (415) 345-2144. Fax. (415) 345-8455. Email [email protected] Abstract This paper introduces a class of discrete iterative algorithms which are provably convergent alternatives to belief propagation (BP) and generalized belief propa- gation (GBP). Our work builds on recent results by Yedidia, Freeman and Weiss (2000) who showed that the fixed points of BP and GBP algorithms correspond to extrema of the Bethe and Kikuchi free energies respectively. We obtain two algorithms by applying CCCP to the Bethe and Kikuchi free energies respectively (CCCP is a procedure, introduced here, for obtaining discrete iterative algorithms by decomposing a cost function into a concave and a convex part). We implement our CCCP algorithms on 2D and 3D spin glasses and compare their results to BP and GBP. Our simulations show that the CCCP algorithms are stable and converge very quickly (the speed of CCCP is similar to that of BP/GBP). Unlike CCCP, BP will often not converge for these problems (GBP usually, but not always, con- verges). The results found by CCCP applied to the Bethe or Kikuchi free energies are equivalent, or slightly better than, those found by BP or GBP respectively (when BP/GBP converge). Note that for these, and other problems, BP/GBP give very accurate results (see Yedidia et al 2000) and failure to converge is their major error mode. Finally, we point out that our algorithms have a large range of inference and learning applications. To appear in Neural Computation 1 Introduction Recent work by Yedidia, Freeman and Weiss (2000) unified two approaches to sta- tistical inference. They related the highly successful belief propagation (BP) al- gorithms (Pearl 1988) to variational methods from statistical physics and, in par- ticular, to the Bethe and Kikuchi free energies (Domb and Green 1972). These BP algorithms typically give highly accurate results (Yedidia et al 2000). But BP
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
algorithms do not always converge and, indeed, failure to converge is their major error mode. This paper develops new algorithms which are guaranteed to converge to extrema of the Bethe or Kikuchi free energies and hence are alternatives to belief propagation. Belief propagation (Pearl 1988) is equivalent to the sum-product algorithm devel- oped by Gallager to decode Low Density Parity Codes (LDPC) (Gallager 1963). In recent years, see (Forney 2001) for a review, the coding community has shown great interest in sum-product algorithms and LDPC’s. It is predicted that this combi- nation will enable the coding community to design practical codes which approach the Shannon performance limit (Cover and Thomas 1991) while requiring only lim- ited computation. In particular, it has been shown that the highly successful turbo codes (Berrou et al 1993) can also be interpreted in terms of BP (McEliece et al 1998). Although BP has only been proven to converge for tree-like graphical models (Pearl 1988) it has been amazingly
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern