Bayes Naive Classifier
1
Bayesian recipe for classification
The Bayesian recipe is simple, optimal, and in principle, straightforward to apply We could design an optimal classifier if we knew:
P(i) (priors) p(x | i) (class-conditional densities)
We have some knowledge and training data {(xi,i)} Use the samples to estimate the unknown probability distributions x is typically high-dimensional Need to estimate P(x| ) from limited data
1
2
Naive Bayes Classifier
Along with decision trees, neural networks, nearest neighbor, one of the most practical learning methods. Categories {1, 2,..., c}, Feature/attribute vector x = [x1, x2, ..., xd ]t Naive Bayes assumption:
P ( x1 ,L , x d |
Naive Bayes classifier:
j
) =
i j
P (xi |
j
)
NB
= arg max
j
P (
)
i
P (xi |
j
)
Performs optimally under certain assumptions
3
Naive Bayes Classifier
Given training data set D Need to estimate probabilistic parameters, no need for complicated training process as in neural networks Estimate P(j) = nj/n (Maximum Likelyhood estimation) Estimate P(xi=aik| j)
ML Estimation Njik/nj discrete feature
2
Training Examples
4
5
Example
Consider PlayTennis problem, and new instance <Outlk=sun, Temp=cool, Humid=high, Wind=strong>
We estimate parameters P(yes) = 9/14, P(no) = 5/14 P(Wind=strong|yes) = 3/9 P(Wind=strong|no) = 3/5 ... We have P(y) P(sun|y) P(cool|y) P(high|y) P(strong|y) = .005 P(n) P(sun|n) P(cool|n) P(high|n) P(strong|n) = .021
Therefore this new instance is classified to "no"
3
Estimation of Probabilities from Small Samples
6
Estimate P(xi=aik| j) discrete feature
ML Estimation Njik/nj
Poor estimates when nj is small What if none of the training instances with category j have feature value xi=aik? P(xi=aik | j) = 0, which lead to P(j|..., xi=aik,...)=0 Typical solution is Bayesian estimate
7
Bayesian estimates for estimating k =P(Xi=k) from
data set Dj
P( X i = k | D j) =
N
jik
+ Mp k
nj + M
Laplace estimates: M=|Dm(Xi)|, Mpk=1 M: imaginary equivalent sample size pk: prior belief about k , summation to 1 The larger the equivalent sample size M, the more confident we are in our prior
4
8
Estimate density p(xi| j) continuous feature
, , or
Assume e.g. Gaussian distribution N(,2), then estimate Discretize into {1, ..., k} Equal-width interval: Width = (xmax xmin)/k
Convert x to i if x is in ith interval
9
Naive Bayes Classifier
Conditional independence assumption is often violated But it works surprisingly well anyway (Domingos and Pazzani, 1997) Successful applications:
Diagnosis Learn which news articles are of interest. Learn to classify web pages by topic. Learn to assign proteins to functional families
Performance often comparable to that of neural networks, decision tree, etc.
5
10
Learning to Classify Text
Learn which news articles are of interest Target concept Interesting? : Documents {+,-} Learning: Use training examples to estimate
P (+), P (- ), P (doc |+), P (doc |-)
What attributes shall we use to represent text
documents?
11
Text each Representation
Represent document by vector of words
one attribute per word position in document
P(doc| j ) = P(length (doc )| j ) P( X i = w k | j )
i =1
|doc|
We need a probability for each word occurrence in each position in the document: 2 x length x |vocabulary|
Too many probabilities to estimate!
Limited samples
6
12
Binary Independence Model
Given a vocabulary V: (w1, ..., w|V|) A document is a vector of binary features (X1, ..., X|V|) Xi is 1 if wi appears in the document, 0 otherwise
x P(doc| j ) = P( xi | j ) = jii (1 - ji )1- xi i =1 i =1 |V | |V |
^ji =
N ji + c1 Nj +c
N ji :# of documents in class j with word wi
Multi-variate Bernoulli Model The number of times a word occurs in a document is not captured
13
Multinomial Model
Assume that probability of encountering a specific word in a particular position is independent of the position, P(wk|j)
The number of probabilities to be estimated drops to 2 x |vocabulary|
Treat each document as a bag of words! Each document d results from |d| draws on a multinomial variable X with |V| values The number of times a word occurs in a document is captured
7
14
Multinomial Model
Assume that the lengths of documents are independent of class
P(d| j ) = P(| d | )
( N k )!
k
k
P(w N !
k k =1
|V |
k
| j )N k
N k is the # of occurences of w k in document d N jk + ck ^ P( w k | j ) = jk = N jk + c
k
N jk is the # of occurences of w k in documents in class j
15
Learn_naive_Bayes_text(Examples) 1. collect all words and other tokens that occur in Examples
Vocabulary: all distinct words and other tokens in Examples For each target value j do
2. calculate the required probability terms
docsj : subset of Examples for which the target value is j P(j) = |docsj|/|Examples| Textj : a single document created by co...