CSC 411 / CSC D11
Basic Probability Theory
5
Basic Probability Theory
Probability theory addresses the following fundamental question:
how do we reason?
Reasoning
is central to many areas of human endeavor, including philosophy (what is the best way to make
decisions?), cognitive science (how does the mind work?), artificial intelligence (how do we build
reasoning machines?), and science (how do we test and develop theories based on experimental
data?). In nearly all real-world situations, our data and knowledge about the world is incomplete,
indirect, and noisy; hence, uncertainty must be a fundamental part of our decision-making pro-
cess. Bayesian reasoning provides a formal and consistent way to reasoning in the presence of
uncertainty; probabilistic inference is an embodiment of common sense reasoning.
The approach we focus on here is
Bayesian
. Bayesian probability theory is distinguished by
defining probabilities as
degrees-of-belief
. This is in contrast to
Frequentist statistics
, where the
probability of an event is defined as its frequency in the limit of an infinite number of repeated
trials.
5.1
Classical logic
Perhaps the most famous attempt to describe a formal system of reasoning is classical logic, origi-
nally developed by Aristotle. In classical logic, we have some statements that may be true or false,
and we have a set of rules which allow us to determine the truth or falsity of new statements. For
example, suppose we introduce two statements, named
A
and
B
:
A
≡
“My car was stolen”
B
≡
“My car is not in the parking spot where I remember leaving it”
Moreover, let us assert the rule “
A
implies
B
”, which we will write as
A
→
B
. Then, if
A
is
known to be true, we may deduce logically that
B
must also be true (if my car is stolen then it
won’t be in the parking spot where I left it). Alternatively, if I find my car where I left it (“
B
is
false,” written
¯
B
), then I may infer that it was not stolen (
¯
A
) by the contrapositive
¯
B
→
¯
A
.
Classical logic provides a model of how humans might reason, and a model of how we might
build an “intelligent” computer. Unfortunately, classical logic has a significant shortcoming: it
assumes that all knowledge is absolute. Logic requires that we know some facts about the world
with absolute certainty, and then, we may deduce only those facts which must follow with absolute
certainty.
In the real world, there are almost no facts that we know with absolute certainty — most of
what we know about the world we acquire indirectly, through our five senses, or from dialogue with
other people. One can therefore conclude that most of what we know about the world is
uncertain
.
(Finding something that we know with certainty has occupied generations of philosophers.)