Sum-Product The Algorithm
CSci 5512: Artificial Intelligence II Instructor: Arindam Banerjee
January 30, 2012
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Factor Graphs
Many problems deal with global function of many variables
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Factor Graphs
Many problems deal with global function of many variables Global function "factors" into product of local functions
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Factor Graphs
Many problems deal with global function of many variables Global function "factors" into product of local functions Efficient algorithms take advantage of such factorization
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Factor Graphs
Many problems deal with global function of many variables Global function "factors" into product of local functions Efficient algorithms take advantage of such factorization Factorization can be visualized as a factor graph
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example
Bipartite graph over variables and local functions
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example
Bipartite graph over variables and local functions Edge "is an argument of" relation
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example
Bipartite graph over variables and local functions Edge "is an argument of" relation Encodes an efficient algorithm
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Bayes Nets to Factor Graphs
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Bayes Nets to Factor Graphs
fA (x1 ) = p(x1 )
fB (x2 ) = p(x2 )
fC (x1 , x2 , x3 ) = p(x3 |x1 , x2 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Bayes Nets to Factor Graphs
fA (x1 ) = p(x1 )
fB (x2 ) = p(x2 )
fC (x1 , x2 , x3 ) = p(x3 |x1 , x2 ) fE (x3 , x5 ) = p(x5 |x3 )
fD (x3 , x4 ) = p(x4 |x3 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF)
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 ) Marginalization of joint distribution is a MPF problem
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 ) Marginalization of joint distribution is a MPF problem
Several other problems use MPF
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 ) Marginalization of joint distribution is a MPF problem
Several other problems use MPF
Prediction/Filtering in dynamic Bayes nets
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 ) Marginalization of joint distribution is a MPF problem
Several other problems use MPF
Prediction/Filtering in dynamic Bayes nets Viterbi decoding in hidden Markov models
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions
Many problems involve "marginalize product of functions" (MPF) Inference in Bayesian networks
Compute p(x1 |x4 , x5 ) Need to compute p(x1 , x4 , x5 ) and p(x4 , x5 ) Marginalization of joint distribution is a MPF problem
Several other problems use MPF
Prediction/Filtering in dynamic Bayes nets Viterbi decoding in hidden Markov models Error correcting codes
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions (Contd.)
The "not-sum" notation h(x1 , x2 , x3 ) =
x2 x1 ,x3
h(x1 , x2 , x3 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions (Contd.)
The "not-sum" notation h(x1 , x2 , x3 ) =
x2 x1 ,x3
h(x1 , x2 , x3 )
Recall g (x1 , x2 , x3 , x4 , x5 ) = fA (x1 )fB (x2 )fC (x1 , x2 , x3 )fD (x3 , x4 )fE (x3 , x5 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Marginalize Product of Functions (Contd.)
The "not-sum" notation h(x1 , x2 , x3 ) =
x2 x1 ,x3
h(x1 , x2 , x3 )
Recall g (x1 , x2 , x3 , x4 , x5 ) = fA (x1 )fB (x2 )fC (x1 , x2 , x3 )fD (x3 , x4 )fE (x3 , x5 )
Computing marginal function using not-sum notations gi (xi ) =
xi
g (x1 , x2 , x3 , x4 , x5 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
MPF using Distributive Law
We focus on two examples: g1 (x1 ) and g3 (x3 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
MPF using Distributive Law
We focus on two examples: g1 (x1 ) and g3 (x3 ) From distributive law
g1 (x1 ) = fA (x1 )
x1
fB (x2 )fC (x1 , x2 , x3 )
x3
fD (x3 , x4 )
x3
fE (x3 , x5 )
Instructor: Arindam Sum-Product Banerjee
The Algorithm
MPF using Distributive Law
We focus on two examples: g1 (x1 ) and g3 (x3 ) From distributive law
g1 (x1 ) = fA (x1 )
x1
fB (x2 )fC (x1 , x2 , x3 )
x3
fD (x3 , x4 )
x3
fE (x3 , x5 )
Also
g3 (x3 ) =
x3
fA (x1 )fB (x2 )fC (x1 , x2 , x3 )
x3
fD (x3 , x4 )
x3
fE (x3 , x5 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Message Passing Example: Computing g1 (x1 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Message Passing Example: Computing g2 (x2 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Local Transformation for Message Passing
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Product Rule:
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Product Rule: At a variable node, take the product of descendants
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Product Rule: At a variable node, take the product of descendants Sum-product Rule:
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Product Rule: At a variable node, take the product of descendants Sum-product Rule: At a factor node, take the product of f with descendants; then perform not-sum over the parent of f
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum-Product Algorithm
The overall strategy is simple message passing To compute gi (xi ), form a rooted tree at xi Apply the following two rules:
Product Rule: At a variable node, take the product of descendants Sum-product Rule: At a factor node, take the product of f with descendants; then perform not-sum over the parent of f
Known as the sum-product algorithm
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Computing All Marginals
Interested in computing all marginal functions gi (xi )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Computing All Marginals
Interested in computing all marginal functions gi (xi ) One option is to repeat the sum-product for every single node
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Computing All Marginals
Interested in computing all marginal functions gi (xi ) One option is to repeat the sum-product for every single node Complexity of O(n2 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Computing All Marginals
Interested in computing all marginal functions gi (xi ) One option is to repeat the sum-product for every single node Complexity of O(n2 ) Repeat computations can be avoided
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Computing All Marginals
Interested in computing all marginal functions gi (xi ) One option is to repeat the sum-product for every single node Complexity of O(n2 ) Repeat computations can be avoided Sum-product algorithm for general trees
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum Product Updates
Variable to local function: xf (x) =
hn(x)\f
hx
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Sum Product Updates
Variable to local function: xf (x) =
hn(x)\f
hx
Local function to variable: f x (x) =
x
y f (y )
y n(f )\{x}
The Sum-Product Algorithm
f (x)
Instructor: Arindam Banerjee
Example: Step 1
fA x1 (x1 ) = fA (x1 ) fB x2 (x2 ) = fB (x2 ) x4 fD (x4 ) = 1 x5 fE (x5 ) = 1
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example: Step 2
x1 fC (x1 ) = fA x1 (x1 ) x2 fC (x2 ) = fB x2 (x2 ) fD x3 (x3 ) =
x3
fD (x3 , x4 )x4 fD (x4 ) fD (x3 , x5 )x5 fE (x5 )
x3
fE x3 (x3 ) =
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example: Step 3
fC x3 (x3 ) =
x3
fC (x1 , x2 , x3 )x1 fC (x1 )x2 fC (x2 )
x3 fC (x3 ) = fD x3 (x3 )fE x3 (x3 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example: Step 4
fC x1 (x1 ) =
x1
fC (x1 , x2 , x3 )x2 fC (x2 )x3 fC (x3 ) fC (x1 , x2 , x3 )x1 fC (x1 )x3 fC (x3 )
x2
fC x2 (x2 ) =
x3 fD (x3 ) = fC x3 (x3 )fE x3 (x3 ) x3 fE (x3 ) = fC x3 (x3 )fD x3 (x3 )
Instructor: Arindam Banerjee The Sum-Product Algorithm
Example: Step 5
x1 fA (x1 ) = fC x1 (x1 ) x2 fB (x2 ) = fC x2 (x2 ) fD x4 (x4 ) =
x4
fD (x3 , x4 )x3 fD (x4 ) fD (x3 , x5 )x3 fE (x5 )
x5
fE x5 (x5 ) =
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Example: Termination
Marginal function is the product of all incoming messages g1 (x1 ) = fA x1 (x1 )fC x1 (x1 ) g2 (x2 ) = fB x2 (x2 )fC x2 (x2 ) g3 (x3 ) = fC x3 (x3 )fD x3 (x3 )fE x3 (x3 ) g2 (x2 ) = fD x4 (x4 ) g5 (x5 ) = fE x5 (x5 )
Instructor: Arindam Banerjee The Sum-Product Algorithm
Belief Propagation in Bayes Nets
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Belief Propagation in Bayes Nets
fA (x1 ) = p(x1 )
fB (x2 ) = p(x2 )
fC (x1 , x2 , x3 ) = p(x3 |x1 , x2 )
Instructor: Arindam Banerjee
The Sum-Product Algorithm
Belief Propagation in Bayes Nets
fA (x1 ) = p(x1 )
fB (x2 ) = p(x2 )
fC (x1 , x2 , x3 ) = p(x3 |x1 , x2 ) fE (x3 , x5 ) = p(x5 |x3 )
The Sum-Product Algorithm
fD (x3 , x4 ) = p(x4 |x3 )
Instructor: Arindam Banerjee