#### You've reached the end of your free preview.

Want to read all 370 pages?

**Unformatted text preview: **PROBABILITY
For the Enthusiastic Beginner David Morin
Harvard University © David Morin 2016d ISBN-10: 1523318678
ISBN-13: 978-1523318674
Printed by CreateSpace
Additional resources located at:
˜ djmorin/book.html Contents
Preface
1 2 Combinatorics
1.1 Factorials . . . . . . . . . . . . . . . .
1.2 Permutations . . . . . . . . . . . . . .
1.3 Ordered sets, repetitions allowed . . . .
1.4 Ordered sets, repetitions not allowed . .
1.5 Unordered sets, repetitions not allowed .
1.6 What we know so far . . . . . . . . . .
1.7 Unordered sets, repetitions allowed . . .
1.8 Binomial coeﬃcients . . . . . . . . . .
1.8.1 Coins and Pascal’s triangle . . .
1.8.2 (a + b) n and Pascal’s triangle .
1.8.3 Properties of Pascal’s triangle .
1.9 Summary . . . . . . . . . . . . . . . .
1.10 Exercises . . . . . . . . . . . . . . . .
1.11 Problems . . . . . . . . . . . . . . . .
1.12 Solutions . . . . . . . . . . . . . . . . vii .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. 1
2
3
7
12
14
20
21
29
29
31
33
35
35
36
41 Probability
2.1 Definition of probability . . . . . . . . . . . . . . . . . .
2.2 The rules of probability . . . . . . . . . . . . . . . . . . .
2.2.1 AND: The “intersection” probability, P( A and B)
2.2.2 OR: The “union” probability, P( A or B) . . . . .
2.2.3 (In)dependence and (non)exclusiveness . . . . . .
2.2.4 Conditional probability . . . . . . . . . . . . . . .
2.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 The art of “not” . . . . . . . . . . . . . . . . . . .
2.3.2 Picking seats . . . . . . . . . . . . . . . . . . . .
2.3.3 Socks in a drawer . . . . . . . . . . . . . . . . . .
2.3.4 Coins and dice . . . . . . . . . . . . . . . . . . .
2.3.5 Cards . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Four classic problems . . . . . . . . . . . . . . . . . . . .
2.4.1 The Birthday Problem . . . . . . . . . . . . . . .
2.4.2 The Game-Show Problem . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. 57
57
59
60
68
71
73
75
75
76
79
81
83
85
85
87 .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. 2.4.3 The Prosecutor’s Fallacy
2.4.4 The Boy/Girl Problem .
2.5 Bayes’ theorem . . . . . . . . .
2.6 Stirling’s formula . . . . . . . .
2.7 Summary . . . . . . . . . . . .
2.8 Exercises . . . . . . . . . . . .
2.9 Problems . . . . . . . . . . . .
2.10 Solutions . . . . . . . . . . . . .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. .
.
.
.
.
.
.
. 90
93
97
106
108
109
109
114 .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
. 133
133
141
146
150
155
163
164
165
168 4 Distributions
4.1 Discrete distributions . . . . . . . . . . . . . . . .
4.2 Continuous distributions . . . . . . . . . . . . . .
4.2.1 Motivation . . . . . . . . . . . . . . . . .
4.2.2 Probability density . . . . . . . . . . . . .
4.2.3 Probability equals area . . . . . . . . . . .
4.3 Uniform distribution . . . . . . . . . . . . . . . .
4.4 Bernoulli distribution . . . . . . . . . . . . . . . .
4.5 Binomial distribution . . . . . . . . . . . . . . . .
4.6 Exponential distribution . . . . . . . . . . . . . . .
4.6.1 Discrete case . . . . . . . . . . . . . . . .
4.6.2 Rates, expectation values, and probabilities
4.6.3 Continuous case . . . . . . . . . . . . . .
4.7 Poisson distribution . . . . . . . . . . . . . . . . .
4.7.1 Discrete case . . . . . . . . . . . . . . . .
4.7.2 Continuous case . . . . . . . . . . . . . .
4.8 Gaussian distribution . . . . . . . . . . . . . . . .
4.9 Summary . . . . . . . . . . . . . . . . . . . . . .
4.10 Exercises . . . . . . . . . . . . . . . . . . . . . .
4.11 Problems . . . . . . . . . . . . . . . . . . . . . .
4.12 Solutions . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. 182
182
184
184
186
189
191
192
193
196
196
199
202
207
207
209
215
221
222
222
227 5 Gaussian approximations
5.1 Binomial and Gaussian . . . . .
5.2 The law of large numbers . . . .
5.3 Poisson and Gaussian . . . . . .
5.4 Binomial, Poisson, and Gaussian .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. 250
250
256
260
263 3 Expectation values
3.1 Expectation value . . . . . . .
3.2 Variance . . . . . . . . . . . .
3.3 Standard deviation . . . . . .
3.4 Standard deviation of the mean
3.5 Sample variance . . . . . . . .
3.6 Summary . . . . . . . . . . .
3.7 Exercises . . . . . . . . . . .
3.8 Problems . . . . . . . . . . .
3.9 Solutions . . . . . . . . . . . .
.
.
.
.
.
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. .
.
.
. 5.5
5.6
5.7
5.8
5.9
6 7 The central limit theorem
Summary . . . . . . . .
Exercises . . . . . . . .
Problems . . . . . . . .
Solutions . . . . . . . . .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. .
.
.
.
. 264
269
270
270
272 Correlation and regression
6.1 The concept of correlation . . . . .
6.2 A model for correlation . . . . . . .
6.3 The correlation coeﬃcient, r . . . .
6.4 Improving the prediction for Y . . .
6.5 Calculating ρ(x, y) . . . . . . . . .
6.6 The standard-deviation box . . . . .
6.7 The regression lines . . . . . . . . .
6.8 Two regression examples . . . . . .
6.8.1 Example 1: Retaking a test .
6.8.2 Example 2: Comparing IQ’s
6.9 Least-squares fitting . . . . . . . . .
6.10 Summary . . . . . . . . . . . . . .
6.11 Exercises . . . . . . . . . . . . . .
6.12 Problems . . . . . . . . . . . . . .
6.13 Solutions . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
. 277
277
280
285
291
294
297
300
305
305
310
313
317
318
318
320 Appendices
7.1 Appendix A: Subtleties about probability
7.2 Appendix B: Euler’s number, e . . . . . .
7.2.1 Definition of e . . . . . . . . . .
7.2.2 Raising e to a power . . . . . . .
7.2.3 The infinite series for e x . . . . .
7.2.4 The slope of e x . . . . . . . . . .
7.3 Appendix C: Approximations to (1 + a) n
7.4 Appendix D: The slope of e x . . . . . . .
7.4.1 First derivation . . . . . . . . . .
7.4.2 Second derivation . . . . . . . . .
7.5 Appendix E: Important results . . . . . .
7.6 Appendix F: Glossary of notation . . . . . .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. .
.
.
.
.
.
.
.
.
.
.
. 335
335
339
339
341
344
346
346
350
350
353
356
359 Preface
This book is written for high school and college students learning about probability
for the first time. Most of the book is very practical, with a large number of concrete
examples and worked-out problems. However, there are also parts that are a bit
theoretical (at least for an introductory book), with many mathematical derivations.
All in all, if you are looking for a book that serves as a quick reference, this may not
be the one for you. But if you are looking for a book that starts at the beginning and
derives everything from scratch in a comprehensive manner, then you’ve come to
the right place. In short, this book will appeal to the reader who has a healthy level
of enthusiasm for understanding how and why the standard results of probability
come about.
Probability is a very accessible (and extremely fun!) subject, packed with challenging problems that don’t require substantial background or serious math. The
examples in Chapter 2 are a testament to this. Of course, there are plenty of challenging topics in probability that do require a more formal background and some
heavy-duty math. This will become evident in Chapters 4 and 5 (and the latter part
of Chapter 3). However, technically the only math prerequisite for this book is a
comfort with algebra. Calculus isn’t relied on, although there are a few problems
that do involve calculus. These are marked clearly.
All of the problems posed at the ends of the chapters have solutions included.
The diﬃculty is indicated by stars; most problems have two stars. One star means
plug and chug, while three stars mean some serious thinking. Be sure to give a solid
eﬀort when solving a problem, and don’t look at the solution too soon. If you can’t
solve a problem right away, that’s perfectly fine. Just set it aside and come back to
it later. It’s better to solve a problem later than to read the solution now. If you do
eventually need to look at a solution, cover it up with a piece of paper and read one
line at a time, to get a hint to get started. Then set the book aside and work things
out for real. That way, you can still (mostly) solve it on your own. You will learn
a great deal this way. If you instead head right to the solution and read it straight
through, you will learn very little.
For instructors using this book as the assigned textbook for a course, a set of
homework exercises is posted at ˜ djmorin/book.html.
A solutions manual is available to instructors upon request. When sending a request,
please point to a syllabus and/or webpage for the course.
The outline of this book is as follows. Chapter 1 covers combinatorics, which
is the study of how to count things. Counting is critical in probability, because
probabilities often come down to counting the number of ways that something can happen. In Chapter 2 we dive into actual probability. This chapter includes a large
number of examples, ranging from coins to cards to four classic problems presented
in Section 2.4. Chapter 3 covers expectation values, including the variance and
standard deviation. A section on the “sample variance” is included; this is rather
mathematical and can be skipped on a first reading. In Chapter 4 we introduce the
concept of a continuous distribution and then discuss a number of the more common probability distributions. In Chapter 5 we see how the binomial and Poisson
distributions reduce to a Gaussian (or normal) distribution in certain limits. We
also discuss the law of large numbers and the central limit theorem. Chapter 6 is
somewhat of a stand-alone chapter, covering correlation and regression. Although
these topics are usually found in books on statistics, it makes sense to include them
here, because all of the framework has been set. Chapter 7 contains six appendices.
Appendix C deals with approximations to (1 + a) n which are critical in the calculations in Chapter 5, Appendix E lists all of the main results we derive in the book,
and Appendix F contains a glossary of notation; you may want to refer to this when
starting each chapter.
A few informational odds and ends: This book contains many supplementary
remarks that are separated oﬀ from the main text; these end with a shamrock, “♣.”
The letters N, n, and k generally denote integers, while x and t generally denote
continuous quantities. Upper-case letters like X denote a random variable, while
lower-case letters like x denote the value that the random variable takes. We refer to the normal distribution by its other name, the “Gaussian” distribution. The
numerical plots were generated with Mathematica. I will sometimes use “they” as
a gender-neutral singular pronoun, in protest of the present failing of the English
language. And I will often use an “ ’s” to indicate the plural of one-letter items (like
6’s on dice rolls). Lastly, we of course take the frequentist approach to probability
in this introductory book.
I would particularly like to thank Carey Witkov for meticulously reading through
the entire book and oﬀering many valuable suggestions. Joe Swingle provided many
helpful comments and sanity checks throughout the writing process. Other friends
and colleagues whose input I am grateful for are Jacob Barandes, Sharon Benedict, Joe Blitzstein, Brian Hall, Theresa Morin Hall, Paul Horowitz, Dave Patterson,
Alexia Schulz, and Corri Taylor.
Despite careful editing, there is essentially zero probability that this book is
error free (as you can show in Problem 4.16!). If anything looks amiss, please check
the webpage ˜ djmorin/book.html for a list of typos,
updates, additional material, etc. And please let me know if you discover something that isn’t already posted. Suggestions are always welcome. David Morin
Cambridge, MA Chapter 1 Combinatorics
TO THE READER: This book is available as both a paperback and an eBook. I
have made a few chapters available on the web, but it is possible (based on past
experience) that a pirated version of the complete book will eventually appear on
file-sharing sites. In the event that you are reading such a version, I have a request:
If you don’t find this book useful (in which case you probably would have returned
it, if you had bought it), or if you do find it useful but aren’t able to aﬀord it, then
no worries; carry on. However, if you do find it useful and are able to aﬀord the
Kindle eBook (priced below $10), then please consider purchasing it (available
on Amazon). If you don’t already have the Kindle reading app for your computer,
you can download it free from Amazon. I chose to self-publish this book so that I
could keep the cost low. The resulting eBook price of around $10, which is very
inexpensive for a 350-page math book, is less than a movie and a bag of popcorn,
with the added bonus that the book lasts for more than two hours and has zero
calories (if used properly!).
– David Morin Combinatorics is the study of how to count things. By “things” we mean the various
combinations, permutations (diﬀerent orderings), subgroups, and so on, that can be
formed from a given set of objects/people/etc. For example, how many diﬀerent
outcomes are possible if you flip a coin four times? How many diﬀerent full-house
hands are there in poker? How many diﬀerent committees of three people can be
chosen from five people? What if we additionally designate one person as the committee’s president? Knowing how to count these types of things is critical for an
understanding of probability, because when calculating the probability of a given
event, we often need to count the number of ways that the event can happen.
The outline of this chapter is as follows. In Section 1.1 we introduce the concept of factorials, which are ubiquitous in the study of probability. In Section 1.2
we learn how to count the number of possible permutations (orderings) of a set of
objects. Section 1.3 covers the number of possible combined outcomes of a repeated
experiment, where each repetition has an identical set of possible results. Examples 2 Chapter 1. Combinatorics
include rolling dice and flipping coins. In Section 1.4 we learn how to count the
number of subgroups that can be formed from a given set of objects, where the order within the subgroup matters. An example is choosing a committee of people
in which all of the positions are distinct. Section 1.5 covers the related question of
the number of subgroups that can be formed from a given set of objects, where the
order within the subgroup doesn’t matter. An example is a poker hand; the order
of the cards in the hand is irrelevant. We find that the answer takes the form of a
binomial coeﬃcient. In Section 1.6 we summarize the various results we have found
so far. We discover that one result is missing from our counting repertoire, and we
remedy this in Section 1.7. In Section 1.8 we look at the binomial coeﬃcients in
more detail.
After learning in this chapter how to count all sorts of things, we’ll see in Chapter 2 how the counting can be used to calculate probabilities. It’s usually a trivial
step to obtain a probability once you’ve counted the relevant things, so the work we
do here will prove well worth it. 1.1 Factorials
Before getting into the discussion of actual combinatorics, we first need to look at a
certain quantity that comes up again and again. This quantity is called the factorial.
We’ll see throughout this chapter that when dealing with a situation that involves
an integer N, we often need to consider the product of the first N integers. This
product is called “N factorial,” and it is denoted by “N!”.1 For the first few integers,
we have:
1! = 1,
2! = 1 · 2 = 2,
3! = 1 · 2 · 3 = 6,
4! = 1 · 2 · 3 · 4 = 24,
5! = 1 · 2 · 3 · 4 · 5 = 120,
6! = 1 · 2 · 3 · 4 · 5 · 6 = 720. (1.1) As N increases, N! gets very large very fast. For example, 10! = 3, 628, 800, and
20! ≈ 2.43 · 1018 . In Chapter 2 we will introduce an approximation to N! called
Stirling’s formula. This formula makes it clear what we mean by the statement, “N!
gets very large very fast.”
We should add that 0! is defined to be 1. Of course, 0! doesn’t make much sense,
because when we talk about the product of the first N integers, it is understood that
we start with 1. Since 0 is below this starting point, it is unclear what 0! actually
means. However, there is no need to try too hard to make sense of it, because as
we’ll see below, if we simply define 0! to be 1, then a number of formulas turn out
to be very nice.
1I don’t know why someone long ago picked the exclamation mark for this notation. But just remember that it has nothing to do with the more common grammatical use of the exclamation mark for
emphasis. So try not to get too excited when you see “N!”! 1.2. Permutations 3 Having defined N!, we can now start counting things. With the exception of the
result in Section 1.3, all of the main results in this chapter involve factorials. 1.2 Permutations A permutation of a set of objects is a way of ordering them. For example, if we have
three people – Alice, Bob, and Carol – then one permutation of them is Alice, Bob,
Carol. Another permutation is Carol, Alice, Bob. Another is Bob, Alice, Carol. It
turns out that there are six permutations in all, as we will see below. The goal of this
section is to learn how to count the number of possible permutations. We’ll do this
by starting oﬀ with the very simple case where we have only one object. Then we’ll
consider two objects, then three, and so on, until we see a pattern. The route we
take here will be a common one throughout this book: Although many of the results
can be derived in a few lines of reasoning, we’ll take the longer route where we
start with a few...

View
Full Document

- Fall '13