: Ex Bucket Sorting (Gonnet, 4.2.3, pg. 176) (1-pass version) Assume you know the distribution of a set of keys to be sorted. That is, assume we have M buckets that partition the key space, M keys, and Pr{key belongs in bucket bm} = 1/M , 1m M . for m :=1 to M do initialize bucket bm (M) for each key do INSERT(key, appropriate bucket) (M) InsertionSort each bucket (length of bucket2) Concatenate buckets 1,2,...M (M) InsertionSort requires time quadratic in # items in bucket. Let nm, 1m M , be a r.v. denoting the number of keys in bucket bm. Total expected time to InsertionSort the M buckets:
1m M 2 E[nm ]
V(nm) = E[nm2] - (E[nm])2 ==> E[nm2] = V (nm) + (E[nm])2 E[nm]? V (nm)? A Bernoulli Trial is an experiment with two possible outcomes: success, and failure, with Pr{success}=p, Pr{failure}=1-p =q. For Bucket Sort, a trial entails picking a bucket, b*, and testing whether a key -> b*. Define rv Xm = if keym ->b* then 1 else 0. nb = Xm
0 for k <0 or k >M . M k M -k Binomial Distribution : Pr{nb =k}= p q k M k M -k Note that it's legal since Pr{nb =k}= 0 for all k and p q k M M k pk q M -k = k pk q M -k = ( p + q) M = 1 k 0k M M E[nb]= k * Pr{nb = k} = k p k q M -k k 0k M 0k M M M M - 1 Get rid of k multiplier: Absorption/extraction = k k k -1 M - 1 k M -k M - 1 k M -k E[nb]= M =M p q p q k -1 k -1 0k M 0k M (make exponent of p match lower term of binomial) M - 1 k -1 M -k = Mp p q k -1 0k M (replace k by k+1) M - 1 k M -1- k M - 1 k M -1- k =Mp = Mp p q p q 0 k +1 M k -1 k M k
What is Pr{nb =k}?
1m M
1
M - 1 = 0) -1 M - 1 k M -1- k Mp =Mp p q k 0 k M 0 k M Another approach: nb = Xm , so
= (because
M - 1 k M p q k -1
-1- k
= M p (p + q )M
-1
=Mp
(Xm) = p- p2 = p(1-p) = pq Theorem : If the M r.v.s are independent, then V Xm = V(Xm ) 1m M 1m M So for Bucket Sort, V Xm = Mpq , and the expected time to Insertion 1m M Sort bucket bm = E[nm2]=V (nm)+(E[nm])2=Mpq + Mp =Mp(2-p) 1 1 Noting that p = , this is 2 - = constant time to Insertion Sort each M M bucket, & (expected) linear time to Bucket Sort the array. Ex : Assume n bits to be transmitted across a channel, Pr{error-free transmission of a bit} = p, & errors are independent. If code capable of detecting/correcting up to e errors, then probability of correct n transmission of the n bits = k q k pn-k 0ke Ex : Choosing a leader in a ring Itai & Rodeh, Symmetry Breaking in nd FOCS, 1981 Distributive Networks , 22 Given a ring of n identical processors, in the same state, but each processor endowed with a (non-psuedo) random number generator, they must all agree on a leader. Each processor can only transmit 1-bit message each time. Initially, all processors are active. loop Each processor draws from U(1..#active processors ) independently of the others. All processors that chose 1 send this message. if 0 processors chose 1 then repeat the loop if 1 processor chose 1 then it's the leader if >1 processors chose 1 then processors that didn't choose 1 become inactive & the loop is repeated n 1 k n - 1 n-k (n, k) = Let Pr{k processors chose 1}= k n n Let (n) - expected number of passes through the loop
E[nb ] = E Xm = E[Xm ] = p = Mp 1m M 1m M 1m M 2] - (E[Xm])2 V (nm)? V (Xm) = E[Xm E[X 2] = 12p + 02q = p, (E[X ])2 = p2, V
1m M
m
m
2
(n) = 1 + (n,0)*(n) +
...Results: For all n 2, (n)<e =2.718...
2kn
(n, k) * (k) ==> (n) =
n
1+
1 - (n,0) - (n,n) lim (n) < 2.442
2kn-1
(n, k) * (k)
Ex : Counting distinct words Flajolet & Martin, "Probabilistic Counting Algorithms for Database Applications", J. Comp. Syst. Sciences , 1985 Assume there are n (not necessarily distinct) words in a text. Let there {0,1}5+lgn . If s is a string of bits, let be a hashing function h :{words} ( x,b), b {0,1}, denote the index of the leftmost bit of x equal to b signature =s1s2...s5+lgn = 00...0 for each word x do signature[(h(x),1)] := 1 return((signature,0)) 2k If algorithm returns k, estimate number of distinct words= 1.5470 If (signature,0)=0, then leftmost bits of signature =1110. If there are 16 distinct words on the tape, 16 15 =.356 Pr{none of hash codings begins 0001}= 16 Ex :2-statistic Given a sample space with K events (for dice, the K =11 events are Xi {2,...,12}) and a series of n independent experiments, we expect each event i to occur n*Pr{Xi} times. n =10000 throws of dice Event Pr{Xi} n*Pr{Xi} Observed Number Yi 2 1/36 277.777780 269 3 1/18 555.555560 520 4 1/12 833.333358 849 5 1/9 1111.111119 1088 6 5/36 1388.888955 1382 7 1/6 1666.666716 1664 8 5/36 1388.888955 1429 9 1/9 1111.111119 1112 10 1/12 833.333358 836 11 1/18 555.555560 581 12 1/36 277.777780 270 2 Statistic: (Yi - n * Pr{Xi}) For loaded dice, this statistic should be large;
1i K
Problem : If Pr{Xi}>>Pr{Xj}, then probably (Yi - n *Pr{Xi})2 >> (Yj - n *Pr{Xj})2 . (Yi - n * Pr{Xi})2 2-statistic: n * Pr{X } i 1i K fair small. 3
For above dice data, 2-statistic = 5.917396 99% 95% 75% 50% 25% 5% 1% n=1 .00016 .00393 .1015 .4549 1.323 3.841 6.635 n=9 2.088 3.325 5.899 8.343 11.39 16.92 21.67 n=10 2.558 3.940 6.737 9.342 12.55 18.31 23.21 For dice data, there are K =11 categories, hence n=10 degrees of freedom (Y11 depends upon Y1,...,Y10). The 2-statistic should be > 5.917396 95% between & 75% of the time, so our deviation is perfectly reasonable (we can not reject the hypothesis that our dice are fair). Geometric Distribution : Let r.v. X denote the number of Bernoulli trials until success . X has values in {1,2,3,...}. Pr{X =k} = qk-1p It's a legal distribution since Pr{X =k} 0 k 0 and p p k -1 k -1 k Pr{X = k} = q p = p q = p q = 1 - q = p = 1 1k 0k 1k 1k p k -1 k -1 E[X] = kq p = p kq = p kq k -1 = kq k q 0k 1k 0k 1k Tangent : kq k = q +2q2+3q3+...
0k
1 = 1+q +q2+q3+... 1- q d d f-f* g g* 1 d 1 d f dx dx 2 3 = = 2 =1+2q +3q +4q +... because 2 dq 1 - q (1 - q) dx g g q = q +2q2+3q3+4q4+... (end of tangent ) (1 - q)2 p q q V (X )= 2 E[X] = 2 = 1/p , q (1 - q) p Geometric waiting times are memoryless (& the geometric is the only memoryless distribution). If equipment's life span is geometric, then it doesn't age: -waiting time in flipping a coin until heads -duration of phone conversations within a city -radioactive decay -life expectancy of an adult fish -aging of glass labware Theorem : If a (discrete) process is memoryless, then it's geometric. Proof : If memoryless, Pr{X =k |X >k -1} = Pr{X =1}, 1k Pr{X = k} By Bayes's Rule, Pr{X =k |X >k -1} = = Pr{X =1} Pr{X > k - 1} Pr{X =k} = Pr{X >k-1} - Pr{X >k} (Pr{X >k-1} - Pr{X >k})/Pr{X >k-1} = Pr{X=1} 1-Pr{X=1} = Pr{X >k}/Pr{X >k-1} = Pr{X >k|X >k -1} = q for all k .
4
Ex : In hashing (to b buckets) with collision resolution by chaining, how many keys must be inserted until every bucket contains a key? A hit is th stage is the sequence of when a bucket gets its first key, and the k st hit until the k th hit. The first stage consists inserts between the (k-1) th stage, Pr{hit}= b - k + 1 . Let nk be a r.v. of the first insert. For the k b th stage. The number of inserts denoting the number of inserts during the k for every bucket to get a hit is n = nk . Because each nk is geometrically
b - k +1 b distributed (with Pr{success}= ), E[nk]= . By linearity of b b - k +1 b 1 expectation, E[n] = E nk = E[nk ] = =b = bHb b - k +1 k 1kb 1kb 1kb 1kb
1kb
Poisson Distribution Let X be an r.v. denoting the number of events in a given time interval (or region). X can assume values {0,1,2,...}. Let pk(t)=Pr{X =k during time interval t}. Assume: -# events in non-overlapping intervals independent. -For small dt : p1(dt) = dt + o(dt) pk (dt) = o(dt)
2k
Given independence, p0(t +dt) = p0(t)*p0(dt) p0(dt) = 1 pk (dt) = 1 - ( dt +o (dt)+o (dt))=1-
dt -2o(dt)
p0(t +dt) = p0(t)*[1-
1k
dt -2o(dt)]
p0 (t + dt) - p0 (t) 2o(dt) p0 (t) = - p0(t) - p0(t) dt dt Combined with p0(0)=1 yields p0(t)=e -t
p1(t +dt) = p1(t)p0(dt) + p0(t )p1(dt ) = p1(t)[1-p1(dt)] + p0(t)p1(dt) p1(t +dt) - p1(t) = - p1(t)p1(dt) + p0(t)p1(dt) = - p1(t) dt + p0(t) dt
p1 (t + dt) - p1 (t) = - p1 + p0(t) dt ( t)n - t e (Poisson Distribution) n!
p1(t) = ( t)e -(t) ... pn(t) =
Legal distribution since: - pn(t)0 for all n 0 ( t)n - t - t ( t)n - pn (t) = e =e = e -t e t = 1 n! n! 0n 0n 0n
5
Ex :Flying bomb hits in London during WW2 in regions of area 1/4 km2 t = 0.9323 km2 n 0 1 2 3 4 5 Yn 229 211 93 35 7 1 npn 226.74 211.39 98.54 3...