Bayesian Inference
Last time we discussed the MLE as a systematic way to derive arbitrary estimators. The MLE is the classic approach to parameter
estimation. While its nice to have a recipe and it is possible to derive
some nice properties of the MLE (es
Welcome to ECE 3077
Intro to Probability and Statistics for ECEs
Instructor: Mark Davenport
Course goals
To develop a mathematical framework for modeling and
understanding uncertainty
To develop tools to reason about, analyze, and learn from
data in the
II. Discrete Random Variables
ECE 3077 Notes by M. Davenport, J. Romberg and C. Rozell
Discrete Random variables
Until now, we have discussed probability almost entire in the context of events, in which there are only two possibilities: either they
happen
Conditional Probability
Conditional probability gives us a systematic way to reason about
the outcome of an experiment based on partial information.
Examples:
A coin is ipped three times and two of the results are heads.
Whats the probability that the rs
Expectation of a random variable
Since random variables give us a way to talk quantitatively about
uncertain quantities, they should also give us a way to make predictions about future outcomes. After seeing a lot of trials we could nd
their average. Coul
The counting principle
Many counting problems can be naturally broken down into multiple
stages. If the outcomes at one stage do not aect the number of
possibilities at the subsequent stages, we can just multiply the number of possibilities at each stage
Joint probability mass functions of multiple
random variables
We are often interested in multiple random variables resulting
from the same experiment. For example, if you run a manufacturing
facility, X may represent the number of failures in a batch of c
Independence
Previously, we talked about conditional probability as a means to
incorporate partial information into a probability law about multiple
outcomes. In other words, if we know something about an event B,
how does that change our belief about A?
I. Introduction to Probability
ECE 3077 Notes by M. Davenport, J. Romberg and C. Rozell
Basic probability models
A probability model consists of an experiment which produces exactly one out of several mutually exclusive outcomes. The essential
elements ar
Independence of random variables
We say that random variables X and Y are independent if
pX,Y (x, y) = pX (x) pY (y) for all x, y,
that is, if we can factor the joint pmf into a pmf that depends only
on X and a pmf that depends only on Y .
This is the sam
Covariance
When discussing a single RV, we used the notion of variance to capture how much that RV could dier from its expected value. With
two RVs we have a similar notion called the covariance, dened as
cov(X, Y ) = E[(X E[X])(Y E[Y ])] = E[XY ] E[X] E[
Bayes rule for random variables
There are many situations where we want to know X, but can only
measure a related random variable Y or observe a related event A.
Bayes gives us a systematic way to update the pdf for X given this
observation.
We will look
Now that we have the basic tools, lets take a closer look at some of
the most common distributions for continuous random variables.
Uniform distribution
We say that X is uniform on [a, b] if
fX (x) =
1
ba
0
x [a, b]
otherwise.
We write this as
X Uniform([
III. General Random Variables
ECE 3077 Notes by M. Davenport, J. Romberg and C. Rozell
Continuous random variables
So far, we have only been concerned with random variables that take
a discrete set of values (that is, the possible outcomes are nite or
cou
ECE 3077A
Lecture 22
Functions of RV, Mapping Uniform
Random Variables
Prof. Elliot Moore II
School of Electrical & Computer Engineering
Georgia Institute of Technology
Class Announcements
HW8 DUE on Monday, March 27, 2017
Exam 2, (tentative) Wednesday
ECE 3077A
Lecture 20
Conditionals and Independence
Prof. Elliot Moore II
School of Electrical & Computer Engineering
Georgia Institute of Technology
Class Announcements
HW7 DUE on Monday, March 13, 2017
IN CLASS EXERCISES ON WEDNESDAY, March 8, 2017
An
ECE 3077A
Lecture 19
Joint PDFs
Prof. Elliot Moore II
School of Electrical & Computer Engineering
Georgia Institute of Technology
Class Announcements
HW7 DUE on Monday, March 13, 2016
IN CLASS EXERCISES ON WEDNESDAY, March 8, 2017
Any other issues?
Joi
An axiomatic approach
So far we have made some intuitive judgements about how
probability laws ought to behave in some very simple cases
In order to be able to generalize this, it is helpful to explicitly
state the properties we will need a probability la
Independence
Two events
and
are independent if
Examples of independent events:
the outcomes of consecutive rolls of a die
whether you are over 6 tall and whether the person sitting
next to you is over 6 tall
whether two randomly chosen people in the cl
Continuous random variables
So far we have focused on discrete random variables
Many things in real life are more naturally modelled as
continuous
the velocity of a car
the amount of time a router waits between packets
the location where a dart Im abou
Independence of random variables
We say that random variables
and
are independent if
for all
This is the same as saying that the events
are independent for all
Since
means that
and
, independence also
for all
For any value of
, the conditional pmf for
is
Properties of mean and variance
Below,
1.
is a random variable, and
are constants
2.
3. We can collect the two results into one statement:
So if
have
has the form
, then we actually do
.
But again, this is not true for general
Properties of mean and varia
Uniform distribution
We say that a random variable
We write this as
The cdf is
is uniform on
if
Expectation and variance
The expectation and variance can be calculated using basic
calculus
Exponential distribution
We say that
is exponential with parameter
Joint pdfs of multiple random variables
Just as before, we can describe everything there is to know
about a pair
of continuous random variables using their
joint pdf
For example, let
Another example
Let
Another example
Let
Han and Chewie at the Cantina
Ha
Joint pdfs of multiple random variables
Just as with discrete random variables, we can describe everything
there is to know about a pair of random variables X, Y associated
with the same experiment using their joint pdf fX,Y (x, y).
The joint pdf fX,Y (x,
General Random Variables
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 19:33, June 15, 2016
Continuous random variables
So far, we have only been concerned with random variables that take
a discrete set of values (that
Total probability theorem for pdfs
If A1 , . . . , An are events that partition the sample space ,
Ai Aj = ,
and
n
[
Ai = ,
i=1
then we can break apart the pdf fX (x) for a random variable X as
fX (x) =
n
X
P (Ai ) fX|Ai (x)
i=1
Exercise:
Suppose that Sub
Compendium of continuous random variables
In this set of notes, we collect the basic facts (pdf, cdf, mean, variance,
etc) for random variables that are commonly used in probabilistic
modeling and statistics. All of the distributions discussed below also
Bonus part (question c of ex #2) :
If we generate a Rayleigh distribution with the mapping function of question 2, we get the following
plot:
0.5
Estimated PDF
True PDF
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
1
2
3
y values
4
5
6
As we can see, the h
ECE 3077, Spring 2017
Homework #5
Due Monday February 20, at the beginning of class
Reading: B&T 2.52.7 (NOTE: Starred (?) problems do not have to be turned in)
1. Random variables X and Y have the joint PMF
(
cxy
pXY (x, y) =
0
x = 1, 2, 4; y = 1, 3
othe
ECE 3077, Spring 2017
Homework #4 Solutions
1
2
3 Since the restaurant price is fixed at $60, revenue between $3800 and $4100 would mean:
$3800
= 63.33 64patrons. (NOTE: We use 64 since the revenue is more than $3800)
$60
$4100
= 68.33 68patrons. (NOTE: W
ECE3077 Spring 2017
Homework #1 Solutions
1.)
ECE3077 Spring 2017
2)
3)
ECE3077 Spring 2017
4)
ECE3077 Spring 2017
5)
ECE3077 Spring 2017
6)
7)
The problem states that: = 0.7 = + () and = 0.9 = + (). Also,
from the given information we can determine that
ECE 3077, Spring 2014
Homework #7
Due Monday, March 13, at the beginning of class
Reading: B&T 3.43.5
1. Metal meter sticks are manufactured on a production line where the true length can
be modeled as a normal random variable X with mean 1 m (one meter)
ECE 3077, Spring 2017
Homework #5 Solutions
1 a
b.
c.
1
d.
e.
f.
2
g. Yes. E[XY ] = E[X]E[Y ]
h. Since X and Y are independent, pX|Y (x|y) = pX (x)f oranyvalueof y
i. P (A) = 24/28
1/3 x = 2
P (X = x X > 1)
2/3 x = 4
=
pX|A (x) =
P (A)
0
otherwise
2 a. S