111
Mor ITI Naaman, PhD and Paul Kantor
Probability Notes
1. Probability
1.1.
Probability Intro and Motivation
Wikipedia:
A way of expressing knowledge or belief that an event will occur or has
occurred
If a fair coin is tossed, what is the probability of getting heads?
If a fair coin is tossed 10 times; what is the probability of getting 8 heads?
In order to clarify the relation between the actions and the things that happen
after those actions are taken, we are going to introduce one more term:
e xperiment. This does not mean quite the same thing as an experiment in a
chemistry lab. It is a technical way of referring to the action that has a
random outcome.
Outcomes, and Sample Spaces
An experiment governed by chance has several possible outcomes
The collection of all possible outcomes in an experiment is the sample
space
Example experiments:
Tossing a coin
Rolling a die
Rolling two dice
Getting ten result pages for query sleeping baby
Example outcomes:
Heads
The first die shows a 4, and the second shows a 5
10 specific web pages (query results) in a specific order
Corresponding Sample Spaces:
- - {Heads, Tails}
- - All possible combinations of two die {(1,1), (1,2), (2,1), (1,3),
(3,1),(6,6)}
- - All possible set of 10 different web pages
1.1.1. Probabilities In Web Search
More details later, but a couple of examples on how we are going to use it:
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
* Estimates on how many result pages will be returned for each query
* The Probability that a user who entered a given query will be s atisfied by a
specific document
E.g. for each particular page that it might deliver, what fraction of the users
will think "yes this is exactly what I want"? In this case we want to know
statistics about all of the users who might enter the query terms: flyin g,
lessons, Hawaii.
A third concept: the probability that a web page that has a particular set of
features (such as the words that are in it) will satisfy this particular user
who has just asked us a query. In this case we want to know more about the
particular user.
1.1.2. Outcomes and events
We will agree to use the general term Event = to mean a collection of
possible outcomes in an experiment .
A die rolls 5 or higher
It will rain tomorrow
A student will be late today
A student will get a grade better than B
More than 15 students will get a B
7 or more web pages are relevant to the query
1.1.3. Elementary Events
Elementary events can be counted, and are not composed of more elementary
events.
If the event getting an even number from the set of Integers E10 {2, 4,
6, 8, 10} elementary?
No! Seeing an even numbers from E10 is not elementary events, because it
can happen in several different ways. The set containing nothing but 2,
is elementary. You cant break it down any further. So the set containing the
number 2 is elementary in the set of numbers. But it is just one part of the
set
2 4 6 8 10
Which contains all of the possibilities that would be in the event.
Cards:
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
Lets consider a deck of cards. Is drawing a card with the number 8 an
elementary event (or outcome)?
No! You could have 8 of , 8 of , 8 of , 8 of .
But if you are look for the set containing only the {8 of } from that deck of
cards, that would be an elementary outcome, or elementary event.
Examples
Experiment is one coin toss:
What are the possible outcomes?
Answer {H} or {T}
Experiment is toss a coin 6 times.
What are the possible outcomes? E.g., (T,H,H,H,H,H). Compound event:
get more than 3 heads. How many possible outcomes are there all together?
2x2x2x2x2x2= 26 64
-
-
Experiment is take a ITI 111 exam
Elementary event: grade = 35 out of 36. Compound event: any
grade higher than 30.
Experiment is class taking ITI 111 exam. Outcome: a set of grades
by class members. The complete sample space is huge (with every
person getting every different possible grade). An example of a
compound event is specifying that specific numbers of students get
each grade, without telling who got which grade.
Thus an elementary event (outcome) is that the set of results is (Joe
gets; Ann gets 25, and so on 24, 24, 34, 19, 36, 35, ). A
compound event is : more than 10 students get an 25.
Class exercise:
Everyone has a coin
Throw the coin 6 times
What are possible outcomes? (H,T,H,H,H,T),
(T,T,H,H,T,T),(T,T,T,T,T,T),...
Count the number of heads
Event: got a sequence with 4 heads. Compound event: can include
outcomes like (T,H,H,H,H,T) or (H,H,H,T,T,H).
Out of the entire class, how many times out of the N tosses did we have a
heads count of 0, 1, 2, , 6? (probability of these different compound
events)
Events may be independent. What does that mean?
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
1.1.4. Independent Events
Events are independent if (and only if) the way that one turns out doesnt
affect the other. Examples that we can think of are:
The rainfall in New Jersey and the smog level in Los Angeles;
{A web page contains the word Cat} and {the page is longer than 5000
words} - -- (at least we have no reason to believe they are dependent!)
Whether the ITI 111 exam has an odd number of questions {the event has
outcomes YES, and NO} , and your score in the exam {a number between 0
and 100}.
1.1.5. Not independent events
When events are not independent, knowing somethin g about one of them
tells us something (but generally not the whole truth) about the other.
One event is document will help us learn how to train the cat and the other
is document contains the word cat. In fact, we expect that if the first
event is true, the second one is almost surely true. But the reverse is not so.
There might be lots of pages that contain the word cat, but are not about
training cats.
Are these t wo events independent:
{A web page contains the word Cat } &
{A web page contains t he word Dog}?
A sensible person might think that since dogs and cats are common pets, in
some countries, we might expect that having the word dog in a page makes it
more likely that we will find the word cat, in the same page. We can examine
this in some systems, to see whether it comes out as we would expect.
1.1.6. Exclusive Events
Two events (outcomes) E1 and E2 are exclusive if they cannot both happen in
the same experiment.
.
Toss a coin. You get heads or tailsnot BOTH.
Since the events are sets of elemntary outcomes, then saying two events are
exclusive means that the intersections of 2 sets of elementary outcomes is
the empty set: or { }.
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
E1 E2 =
We say that the probability of an event is a number which tells how often it
would occur (over a very long set of repetitions of the experiment). The
probability is a fraction: number of equally likely elementary outcomes that
belong to the event, divided by the total number of equally likely elementa ry
outcomes.
If two events are exclusive then the probability of the union of t hem, that is,
t he probability that one or the other occurs, is exactly the sum of their
probabilities. [This is the same fact as the fact that the size of the sum of two
non-overlapping sets is the sum of their sizes.]
Exclusive sets at Rutgers (you cant be both!):
Grads / undergrads
Commuters / those who live on campus
Students 18 and over, Students under 18
1.2.
Probability Notations and formulas:
Pr(E) = Probability of event E
0 Pr(E) 1 always
Pr(A) = 0 means that A is impossible
Pr(A) + Pr(not A) = 1
Pr(A and B)=Pr(A B) [Note that in the left hand side the letters A, B
refer to the experiments while on the right hand side they refer to the sets of
outcomes.
This is a problem that will bedevil us when talking about probability: we
may want to use the same notation to refer to the uncertain result of a
particular experiment, or we may want to use it to refer to a specific outcome
event. We will try to use it to refer to outcomes all the time (events) because
that is more consistent.
Note: this is a change from what I said in class. P kantor
1.3.
The probability that both of two independent outcomes
(events) will occur
If A, B are independent events: Pr(A B)=Pr(A)*Pr(B)
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
Note: Sometimes we write P(event ) and sometimes we write Pr{event} .
This is like talking two dialects of the same language.
1.4.
Complementary events
As we mentionedL If E is any event, and E is the complementary event (i.e.
not E), then
Pr{E}+Pr{E}=1
EXAMPLE:
Pr(a student got >18/36 in exam)
Pr(a student got <=18/36 in exam)
Pr(more than 3 heads) in a single toss of 5 coins/
Pr(3 heads or less) in a single toss of 5 coins/
Pr(heads) in one toss
Pr(tails) in one toss
1.5.
Conditional Probability
[WIKIPEDIA] The conditional probability of some event (outcome) A, given
the occurrence of some other event B. Conditional probability is written
Pr(A|B), and is read "the (conditional) probability of A, given B" or "the
probability of A under the condition B", or the probability of A given B.
Class experiment:
How many male?
How many female?
FEMALE/MALE EYE COLORthis is complementary because if youre
not male, youre female (for this example anyway)
Pr(M)=70%
Pr(F)=30%
Now we need a notation. We write Pr{A|B} for the probability that outcome
A will happen (for event A), given that outcome B happened (for event B).
NOTE: Hmmmm. This is a little confusing. If we want to be really careful
we say Pr{A=a|B=b}. This means, the probability that the outcome of event
A is the particular outcome a, given that the outcome of the event B is the
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
particular event b. In what follows we will not be this fussy. Why not?
Because it is pretty obvious that if we write Pr{blue|M} we d ont mean the
probability that the student is blue, given that her eyes Male.
Pr(blue|M)=25%
Pr(M|blue)=? are (not enough info at this point. We need both (A|B) and (B|A)
Pr(Blue)=10%
Pr(M|blue)=0.25(0.7)/0.4=0.4375
Not the perfect example because we assume these are independent: eye color
and gender (but in our class room the sample is small so there will be
differences).
Lets figure it out.
1.5.1. Joint Probability
[Wikipedia] probability of two events in conjunction. That is, it is the
probability of both events together. The joint probability of A and B is written
Pr(A B) or Pr(A,B).
When does Pr(A|B) = Pr(A)? When A and B are independent!
Pr(Class dismissed early | professor wearing red) = P r(Class dismissed early
overall).
But:
Pr(Class dismissed early | professor feeling sick) != P r(Class dismissed early
overall).
In case of independence:
P( A B) P( A)P(B)
Relation between Joint events and Conditional events:
Pr( A | B)
Pr( A B)
Pr( B)
Try it on the male/female eyes:
From above:
Pr(male|blue) =0.25(0.7)/0.4=0.4375
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
1.5.2. Very deep fact about conditional probability
There is an important relation between two conditional probabilities, which
we will get to. But the relation is not equality.
We can write: Pr(A|B) != Pr(B|A)
[the exclamation point in front of the = sign means NOT]
Lets work out an example. Sometimes tests are used to place students in
tracks. The test is supposed to find out whether a student is Gifted and
talented. But the test is not perfect, so we can ask: what is the real probability
that someone is gifted and talented (whatever that may mean, given that the
test says the person is.
To be specific, suppose that 1% of the population is gifted and talented
(GT, to save writing it out)..
That is:
Pr(GT)=1%, Pr(not- GT) = 99%
Let's also suppose that when doing the GT test, there is 1% chance of a false
positive (that is, the test says the person is GT, but the person is not really GT
they just happened to get lucky in guessing). This kind of result is called a
f alse positive:
We express this as the conditional probability of being scored as GT given
being ordinary.
Pr(GT|Ordinary) = 1%, Pr(not GT|ordinary) = 99%
Since the test is not perfect, it might also fail to detect people who really are
GT. For convenience we will suppose that this happens to 1% of the GT
people they get missed. [I actually believe we miss a lot more than that].
Pr(not- GT|really is GT) = 1%, Pr(GT|really is GT) = 99%
Now we can do a lot of calculat ion:
Of all the people in the population, the fraction of people who are gifted and
also test to be GT:
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
Pr(really are gifted, test as GT)=Pr(gifted)*Pr(pass|gifted) =0.01*0.99=0.0099
= 0.99%
Fraction of people who are ordinary and do not test as GT:
Pr(ordinary, not test as GT) = Pr(ordinary)*Pe( not test GT|ordinary) =
0.99*0.99= 98.01%
Fraction of people who are ordinary but test as GT:
Pr(ordinary, test as GT) = Pr(ordinary)*Pr(test GT|ordinary) = 0.99*0.01=
0.99%
Fraction of people who are GT and but test to not GT:
Pr(GT,not test to GT)=Pr(GT)*Pr(test not GT|really are GT)
=0.01*0.01=0.01%
Lets check that we have accounted for all of the people:
.99% + 98.01% + .99% + .01% = 100% exactly!
But now lets look just at the people who test out to be GT.
What fraction of the population are they:
Pr(test GT) = Pr(test GT|really GT)+Pr(test GT|ordinary) = 1.98%
Are all of these people really GT? No!. We know that only .99% of the
people in the population are GT and also test as GT. This is just a fraction o f
all the people who test as GT, because the rest of them are the false
positives.
So, what is the probability of the person actually being GT if they test as GT?
Pr(gifted|pass) = Pr(gifted,test GT)/Pr(test GT) = .99%/1.98% = 50% !!!!
Remember we had Pr(test GT|gifted) = 99%
In other words, fully half of the people who are selected by this test are
selected by mistake. How did it happen?? It is because there very few of them
to begin with. So it is as likely (in this particular example) that a person tests
GT because of an error in the test, as it is because the person really is GT.
The starting odds that person is GT, which is called the prior probability of
being GT is very low..
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
Another example: cancer screening: if your cancer test was p ositive, depends
on prior history of disease, you might be just fine
Another example that I work on as part of my own research is the problem of
screening for terrorist threats. Since only a tiny fraction of all travelers will
actually be terrorists, the proportion of real terrorists in the people who are
pulled aside for detailed inspection is still very very small.
Note: numbers from Sick/Well example from Wikipedia:
http://en.wikipedia.org/wiki/Conditional_probability#The_conditional_probab
ility_fallacy
So you can use that page as reference.
PART 2
Review of previous class:
Probability:
A way of expressing knowledge or belief that an event will occur or has
occurred
If a fair coin is tossed, what is the probability of getting heads?
If a fair coin is tossed 4 time; what is the probability of getting 3 heads?
Outcomes :
An experiment governed by chance has several possible outcomes
The collection of all possible outcomes in an experiment is the sample space
{H,T}
{(T,T,T,T}, (H,T,T,T),}
Outcomes and Internet experiments:
1. A specific page is the first search result for my query
2. A given set of Recommended tracks on Last.fm
3. A set of specific Web pages returned as search results
4. The user Facebook recommends for me to connect with
5. A specific set of friends with updated status on Facebook
Sample space : collection of all possible outcomes.
What are they in each case above?
Events:
A collection of possible outcomes in an experiment
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
From above:
1. A page that I care about.
2. A set where 3/5 songs are good recommendations
3. Precision > 50%
4. A user I actually want to connect with
5. The set includes someone I care about
Notation:
Pr(E) = Probability of event E
0 Pr(E) 1 always
Pr(E) = 0 means that A is impossible
Pr(A) + Pr(not A) = 1
Pr(A,B)=Pr(A B)
If A, B independent: Pr(A B)=Pr(A)*Pr(B)
Complementary events:
Pr{E}+Pr{E}=1
Assuming all outcomes are equally likely, Pr(event) = number of outcomes in
event / overall outcomes.
Conditional Probability
Is the probability of some event A, given the occurrence of some other event
B. Conditional probability is written Pr(A|B), and is read "the (conditional)
probability of A, given B" or "the probability of A under the condition B"
When does Pr(A|B) = Pr(A)? When A, B independent!
Joint Probability
probability of two events in conjunction. That is, it is the probability of both
events together. The joint probability of A and B is written Pr(A B) or
Pr(A,B).
In the case of independence:
P( A B) P( A)P(B)
Relation between Joint events and Conditional events:
P ( A | B)
03 Probability
P ( A B)
P ( B)
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
Fallacy of Cond. Prob: Pr(A|B) != Pr(B|A)
Shown with gifted kids test and whether or not the kid is gifted if they pass
the test.
Another previous example: cancer screening
Another example: Pr(rain falling | rain predicted) != Pr(rain pred | rain
falling)}
1.6.
Estimating Search Result Sizes With Probability
How many results for Africa Pink Flamingos?
A= {p: p has term Africa} = 318,000,000
K = Pink = 253,000,000
F Flamingos = 2,340,000
Assuming independence:
Pr( A K F ) Pr( A) Pr( K ) Pr( F )
Pr(A) = |A|/N where N is number of pages in our database.
Pr(K) = |K|/N
Pr(F) = |F|/N
Lets assume N=100,000,000,000 (100B)
Estimate:
| A K F | N *Pr( A K F ) N *Pr( A) Pr( K ) Pr( F )
N
| A | | K | | F | | A || K || F |
=1,000,0003*318*253*2.34/(1,000,000 2
NNN
N2
*100000*100000)= 1,000,000*318*253*2.34/100,000*100,000= 18
18???
But they are not independent!
Pr( A K F ) Pr( A) Pr( K ) Pr( F | A K )
Pr(F|A,K ) is hard so we can check Pr(F|K), or assume Pr(F|A) is Pr(F).
Pr( A K F ) Pr( A) Pr( K ) Pr( F | K )
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
| A K F | N *Pr( A) Pr( K )
Probability Notes
Pr( F K )
Pr( K )
Do Pr(A),Pr(K) the way we did before:
| A K F | N *Pr( A)*Pr( K )*
Pr( F K )
N *Pr( A)*Pr( F K )
Pr( K )
But we need an estimate for | F K | lets assume its 1,540,000 (We
can get an estimate by getting 100 pages of F and counting how many
have K). Then we get:
100B * 318/100,000 * 1.54/100,000 = 4897.2
Actual number: 424,000
Why are we off?
Wrong N?
Forget that A,K and A,F also not independent?
Note: we c an use these formulas to estimate the size of the Google
index! (N)
| A K F |
| A || K || F |
N2
=>
N
| A || K || F |
| AK F |
Try it with:
Rutgers Shoes Payphone
|R| =
|S| =
|P|=
|R,S,P|=
N= ?
You may try other combinations of three independent words
03 Probability
ITI 111
Mor Naaman, PhD and Paul Kantor
Probability Notes
PART 3
Review:
IF INDEPENDENT:
| A K F | N *Pr( A K F ) N *Pr( A) Pr( K ) Pr( F )
| A | | K | | F | | A || K || F |
=N
NNN
N2
If not, say F dependent on A,K:
Pr( A K F ) Pr( A) Pr( K ) Pr( F | A K )
And need to estimate some other terms as well.
In any case, you can find the N!
N
| A || K || F |
For any three independent terms.
| AK F |
1.7.
Bayes Theorem
The relation between Pr(A|B) and Pr(B|A).
Deriving Bayes:
1.
Pr( A | B)
Pr( A B)
Pr( B)
2. Pr( B | A)
Pr( B A) Pr( A B)
Pr( A)
Pr( A)
Multiply by Pr(A) in Eq Number 2:
Pr( A B) Pr( B | A) Pr( A)
Now plugging things in Eq. 1:
Pr( A | B)
03 Probability
Pr( B | A) Pr( A)
Pr( B)
ITI 111
Mor Naaman, PhD and Paul Kantor
And this is Bayes formula!
Try it on the eyes/gender example:
Pr(blue|female) = T
Then whats Pr(female|blue)?
03 Probability
Probability Notes