# Register now to access 7 million high quality study materials (What's Course Hero?) Course Hero is the premier provider of high quality online educational resources. With millions of study documents, online tutors, digital flashcards and free courseware, Course Hero is helping students learn more efficiently and effectively. Whether you're interested in exploring new subjects or mastering key topics for your next exam, Course Hero has the tools you need to achieve your goals.

100 Pages

### note6

Course: ECE 111, Spring 2012
School: Waterloo
Rating:

Word Count: 841

#### Document Preview

6: Lecture Connecting Entropy to Uniquely Decodable Codes En-hui Yang University of Waterloo En-hui Yang ECE 415: Multimedia Communications Kraft Inequality C (X1 )C (X2 ) X = X1 X2 Memoryless coder C DMS Rate R Kraft inequality Entropy H (X ) Figure: 6.1 Connecting entropy to uniquely decodable codes Theorem (6.1 Kraft inequality) Let X = {a0 , a1 , , aJ 1 }. Then any uniquely decodable code C...

Register Now

#### Unformatted Document Excerpt

Coursehero >> Canada >> Waterloo >> ECE 111

Course Hero has millions of student submitted documents similar to the one
below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.

Course Hero has millions of student submitted documents similar to the one below including study guides, practice problems, reference materials, practice exams, textbook help and tutor support.
6: Lecture Connecting Entropy to Uniquely Decodable Codes En-hui Yang University of Waterloo En-hui Yang ECE 415: Multimedia Communications Kraft Inequality C (X1 )C (X2 ) X = X1 X2 Memoryless coder C DMS Rate R Kraft inequality Entropy H (X ) Figure: 6.1 Connecting entropy to uniquely decodable codes Theorem (6.1 Kraft inequality) Let X = {a0 , a1 , , aJ 1 }. Then any uniquely decodable code C over X satises the following Kraft inequality J 1 2|C (x )| = x X 2nj 1 j =0 where nj = the length of the codeword C (aj ). En-hui Yang ECE 415: Multimedia Communications (6.1) Theorem (6.2) Let X = {a0 , a1 , , aJ 1 }. Given a set of codeword lengths nj , 0 j J 1, that satisfy the Kraft inequality (6.1), there exists a prex code C over X such that |C (aj )| = nj for all 0 j J 1. Example 6.1 (Illustration of Theorems 6.1 and 6.2): Let us revisit Example 4.1. The code C2 corresponding to f2 (x ) is uniquely decodable; it has the following codeword lengths: n0 = 1, n1 = 2, n2 = 3, and n3 = 3 It is easy to verify that in this case 3 2nj = 1 j =0 En-hui Yang ECE 415: Multimedia Communications i.e., the Kraft inequality is satised with equality. On the other hand, for the same set of codeword lengths, a prex is given by f3 (x ). Implication As long as R is concerned, it sufces to consider prex codes only! Example 6.2: Let n0 = n1 = 2, and n2 = n3 = 3 The above values of nj satisfy the Kraft inequality. A prex code is given below: x a0 a1 a2 a3 En-hui Yang C (x ) 00 01 100 101 ECE 415: Multimedia Communications Source Coding Theorem Theorem (6.3 Memoryless case) Let X = {Xi } 1 be a DMS over X with a common pmf i= p(x ), x X . Then the following hold: (a) For any uniquely decodable code C over X , its average codeword length R in bits per symobl is lower bounded by H (X ): R H (X ) = H (X1 ) (6.2) (b) There is a prex code C over X such that R = p(x )|C (x )| < H (X ) + 1 x X En-hui Yang ECE 415: Multimedia Communications (6.3) Proof of Theorem 6.3: Equation (6.2) follows directly from the log sum inequality and Kraft inequality. To see this is the case, let nj = |C (aj )|, 0 j J 1 Then J 1 R= pj nj j =0 where pj = p(aj ). Applying the log sum inequality, one gets J 1 j =0 J 1 pj pj log nj 2 2nj log j =0 0 where (6.4) is due to the Kraft inequality J 1 2nj 1 j =0 En-hui Yang ECE 415: Multimedia Communications (6.4) Moving pj log pj to the right side of (6.4) yields R H ) To (X show Part (b) of Theorem 6.3, suppose without loss of generality that pj > 0 for all 0 j J1 . Let nj = log pj , 0 j J 1 where for any real number y , y = the least integer y It is easy to verify that y y <y +1 Therefore log pj nj < log pj + 1 and J 1 J 1 nj 2 j =0 En-hui Yang pj = 1 j =0 ECE 415: Multimedia Communications (6.5) In view of Theorem 6.2, there is a prex code C such that |C (aj ) = nj . This, together with (6.5), implies J 1 R = pj nj < H (X ) + 1 j =0 This completes the proof of Theorem 6.3. Example 6.3: Let us revisit Example 5.2: a with probability 1/2 b with probability 1/4 X= c with probability1/8 d with probability 1/8 n0 = log p(a) = 1, n1 = log p(b) = 2 n2 = log p(c ) = 3, and n3 = log p(d ) = 3 The code C = C3 corresponding to f3 (x ) in Example 4.1 is a prex code with the above nj as its codeword length, and R = 1/2 + 1/2 + 3/8 + 3/8 = 7/4 = H (X ) En-hui Yang ECE 415: Multimedia Communications One Bit Reduction Block Memoryless Codes x1 x2 xn xn+1 x2n DMS x1 x2 xn Block mem- C (x1 xn )C (xn+1 x2n ) oryless Rate R coder C C (x1 x2 xn ) Figure: 6.2 Block memoryless code Denition A block memoryless code with block length n is a lossless source code which encodes source sequences in an independent, n-block by n-block manner. It is thus uniquely described by a mapping C from the extended alphabet X n to the set of all codewords corresponding to all n-blocks: x1 x2 xn C (x1 x2 xn ) En-hui Yang ECE 415: Multimedia Communications All properties of memoryless codes can be carried over to the case of block memoryless codes. In particular, we have the following result. Theorem (6.5 Block memoryless case) Let X = {Xi } 1 be a DMS over X with a common pmf i= p(x ), x X . Then the following hold: (a) For any uniquely decodable code C with block length n, its average codeword length in bits per symbol is lower bounded by H (X ): 1 1 R= n n p(x1 xn )|C (x1 xn )| H (X ) x1 xn (6.6) X n (b) There is a prex code C with block length n such that 1 1 R= n n p(x1 xn )|C (x1 xn )| < H (X ) + x1 xn X n 1 n (6.7) En-hui Yang ECE 415: Multimedia Communications Remark The gap between the lower bound in (6.6) and the upper 1 bound in (6.7) is now n . Letting n , we see that H (X ) = the smallest compression rate in bits per symbol of the DMS X = {Xi } 1 i= En-hui Yang ECE 415: Multimedia Communications
Find millions of documents on Course Hero - Study Guides, Lecture Notes, Reference Materials, Practice Exams and more. Course Hero has millions of course specific materials providing students with the best way to expand their education.

Below is a small sample set of documents:

Waterloo - ECE - 111
Lecture 7: Huffman CodingEn-hui YangUniversity of WaterlooEn-hui YangECE 415: Multimedia CommunicationsGiven a pmf pj = p(aj ), 0 j J 1, overX = cfw_a0 , a1 , , aJ 1 we now look at how to design an optimal prex code C such thatJ 1R=pj |C (aj )|
Waterloo - ECE - 111
Lecture 8: Arithmetic CodingBasic IdeaEn-hui YangUniversity of WaterlooEn-hui YangECE 415: Multimedia CommunicationsDrawbacks of Huffman CodingIn principle, the Huffman coding algorithm can also be appliedto design optimal prex codes with block len
Waterloo - ECE - 111
Lecture 9: Adaptive Arithmetic CodingEn-hui YangUniversity of WaterlooEn-hui YangECE 415: Multimedia CommunicationsUniversal Source CodingIn Huffman coding and arithmetic coding discussed so far, boththe encoder and decoder are assumed to know the
Waterloo - ECE - 111
Lecture 10: Run Length CodingEn-hui YangUniversity of WaterlooEn-hui YangECE 415: Multimedia CommunicationsBasic IdeaRun length coding is efcient for data sequences where longsegments of repeated symbols (runs) appear. Consider, forexample, a bina
Waterloo - ECE - 111
Lecture 11: Lempel-Ziv CodingEn-hui YangUniversity of WaterlooEn-hui YangECE 415: Multimedia CommunicationsLempel-Ziv codes are universal codes which are based onstring matching. Since the original work of Ziv and Lempel inlater 1970s, many variant
Waterloo - ECE - 111
ECE 415: Multimedia CommunicationsHomework Set 11Due Monday, Jan. 23, 2012 (Hand in to your TA)The following problems are from Section 5 (Exercises) of Chapter 3 of Reference Book [1]:1. Problem 22. Problem 43. Problem 54. Problem 65. Problem 8
Waterloo - ECE - 111
ECE 415: Multimedia CommunicationsHomework Set 21Due Monday, Jan. 23, 2012 (Hand in to your TA)Problem 1 Determine which of the following codes is uniquely decodable.(a) cfw_0, 10, 11.(b) cfw_0, 01, 11.(c) cfw_0, 01, 10.(d) cfw_00, 01, 11, 001, 01
Waterloo - ECE - 111
ECE 415: Multimedia CommunicationsHomework Set 31Due Monday, Jan. 30, 2012 (Hand in to your TA)Problem 1 The probability mass function of X is given in (1):x1234p(x) 0.124 0.187 0.3 0.389(1)(a) Letnj = log p(X = j ) , 1 j 4.Design a prex cod
Waterloo - ECE - 111
Tutorial 9 ECE 415 Jin Meng HW 7 Problem 1 Consider a random vector U=(U1U2U3U4)^T with zero mean and covariance matrix &quot;\$\$\$\$!=\$\$\$\$\$#13 %872''131387''22'1313 '7822''131378'22&amp;132 Comput
Waterloo - ECE - 111
ECE428 Winter 2012Part II: CryptographyAssignment 11. The following is the result of a Vigenere cipher of unknown period Explain how you would go aboutdeciphering the cipher? You may use any of the many tools available on the internet to help you actu
Waterloo - ECE - 111
Tutorial on Assignment 2ECE 428University of WaterlooWinter 2012Question 1 (6.1)Public-key cryptography can be used for encryptionand key exchange. Furthermore, it has someproperties (such as nonrepudiation) which are notoffered by secret key cryp
Waterloo - PSYCH - 101
Introduction to PsychologyPsychology 101 (Section 001)University of WaterlooWinter 2012COURSE SYLLABUSInstructor:Class Meeting:Office:Office Hours:Phone:E-mail:Course Website:Richard EnnisTuesday, 6:30 - 9:20 p.m., HH159PAS 3017Thursday, 10
Waterloo - PSYCH - 101
lecture1.txt2012-02-08*The Basic ModelEnvironment -| Person -&gt; Behaviour -&gt; Outcome-Person- Innate tendencies of person determining how a person behavesPerson interacts with environment (denoted as P x E)- Environment affects person,e.g. social
Waterloo - PSYCH - 101
lecture2.txt2012-02-12# Background: The Industrial RevolutionEarly employment for children in the case of civilians- Mines, Cleaning chimney, factory maintenance- no child labor laws# Emergence of different point of views on childrenSapling- Just
Waterloo - PSYCH - 101
lecture3.txt2012-02-12Jan 17, 2012Lecture 3* Review of Cog. Dev: Jean Piaget- Children are genuinely different from the way they think and behave in comparison toadults- public education- child labour laws- SchemaP x E (child interacting with wo
Waterloo - PSYCH - 101
lecture4.txt2012-02-12Lecture 4Sensation and PerceptionPXEPerson perceives the world.-Two Inseparable ProcessesSensation- Physical sensing of environment- Physiological processes- Relatively objective- Learning and experience not required- eg
Waterloo - PSYCH - 101
lecture6.txt2012-02-07Lecture 6Feb 7, 2012Midterm: 6:30 to 7:45Comprehension from study guide: have a look=Memory*Sensation and Perception Across Time- Memory: Capacity to store info that's been previously processed*3 phases of memory-Encoding
Waterloo - PSYCH - 101
Lecture'5'(January'31,'2012)'Lecture'Outline'(midterm'upto'end'of'this'lecture)''Associations'! Volkswagen+girl+=+aroused.next+time,+when+you+see+a+Volkswagen,+u+get+aroused?!+! Associating+something+with+something+positive,+may+actually+have+an+imp
Waterloo - PSYCH - 101
Page 1 of 71TipsChapter 3: The Nature and Nurture of Behavior - Exam 1Exam BuilderToolboxHelpTest BanksEdit Exam 1Title &amp; Introductory Text : Add/Edit the assessment's title and introductory text by clicking the link below. The titleand introduct
UPR Humacao - ECON - 101
Captulo IIIFRDRIC CHOPIN (1810-49)III. 1. Obras para pianoBenedetto seala que la obra para piano de Chopin se puede clasificar de acuerdo a losmodos, tiempos y lugares de su actividad pianstica en diversos gneros y formas. (104)Salvo algunas cancione
Minnesota - CSCI - 5512
Course Overview, Probability BasicsCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeJanuary 18, 2012Instructor: Arindam BanerjeeCourse Overview, Probability BasicsGeneral InformationCourse Number: CSci 5512 Class: Mon Wed 12:45-02:0
Minnesota - CSCI - 5512
Exact InferenceCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeJanuary 21, 2012Instructor: Arindam BanerjeeExact InferenceOverview: Inference TasksSimple Queries: Compute posterior marginals P(b|j, m)Instructor: Arindam BanerjeeE
Minnesota - CSCI - 5512
The Sum-Product AlgorithmCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeJanuary 30, 2012Instructor: Arindam BanerjeeThe Sum-Product AlgorithmFactor GraphsMany problems deal with global function of many variablesInstructor: Arinda
Minnesota - CSCI - 5512
Approximate Inference: StochasticCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeFebruary 1, 2012Instructor: Arindam BanerjeeApproximate Inference: StochasticBayesian Networks with LoopsP(C) .50CloudyC P(S|C) T .10 F .50Sprinkle
Minnesota - CSCI - 5512
Approximate Inference: MCMCCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeFebruary 6, 2012Instructor: Arindam BanerjeeApproximate Inference: MCMCProblemsPrimarily of two types: Integration and OptimizationInstructor: Arindam Bane
Minnesota - CSCI - 5512
Junction TreesCSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeFebruary 13, 2012Instructor: Arindam BanerjeeJunction TreesReparameterizationConsider a Bayesian network p(a, b, c, d) = p(a|b)p(b|c)p(c|d)p(d)Instructor: Arindam Baner
Minnesota - CSCI - 5512
Probabilistic Reasoning over Time: Part ICSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeFebruary 15, 2012Instructor: Arindam BanerjeeProbabilistic Reasoning over Time: Part IOutlineTime and uncertaintyInstructor: Arindam Banerjee
Minnesota - CSCI - 5512
Probabilistic Reasoning over Time: Part IICSci 5512: Artificial Intelligence II Instructor: Arindam BanerjeeFebruary 22, 2012Instructor: Arindam BanerjeeProbabilistic Reasoning over Time: Part IIHidden Markov ModelsXt is a single, discrete variable
Minnesota - CSCI - 5512
Making Simple DecisionsCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeFebruary 27, 2012Instructor: Arindam BanerjeeMaking Simple DecisionsPreferencesApL1pBA lottery is a situation with uncertain prizesLottery L = [p , A; (1 p
Minnesota - CSCI - 5512
Markov Decision ProcessesCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeFebruary 29, 2012Instructor: Arindam BanerjeeMarkov Decision ProcessesSequential Decision ProblemsSearchexplicit actionsand subgoalsPlanninguncertaintyand
Minnesota - CSCI - 5512
Game TheoryMechanism DesignCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeMarch 7, 2012Instructor: Arindam BanerjeeGame TheoryMechanism DesignOutlinePayos and StrategiesDominant Strategy EquilibriumNash EquilibriumMaximin Strate
Minnesota - CSCI - 5512
Learning From ObservationsCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeMarch 19, 2012Instructor: Arindam BanerjeeLearning From ObservationsOutlineLearning AgentsInductive LearningDecision Tree LearningMeasuring Learning Perform
Minnesota - CSCI - 5512
Learning TheoryCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeMarch 21, 2012Instructor: Arindam BanerjeeLearning TheoryPAC LearningLearning from a Hypothesis Space HInstructor: Arindam BanerjeeLearning TheoryPAC LearningLearning
Minnesota - CSCI - 5512
Learning with Hidden VariablesCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 18, 2012Instructor: Arindam BanerjeeLearning with Hidden VariablesHidden VariablesReal world problem have hidden variablesInstructor: Arindam Banerj
Minnesota - CSCI - 5512
Neural NetworksCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeMarch 28, 2012Instructor: Arindam BanerjeeNeural NetworksBrain1011 neurons of &gt; 20 types, 1014 synapses, 1ms10ms cycle timeSignals are noisy spike trains of electrical p
Minnesota - CSCI - 5512
Linear ModelsCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 2, 2012Instructor: Arindam BanerjeeLinear ModelsUnivariate Linear Regression(a)(b)hw (x ) = w1 x + w0nnL2 (yi , hw (xi ) =2Loss (hw ) =i =1Instructor: Arinda
Minnesota - CSCI - 5512
Convex FunctionsA function f is convex if dom(f ) is a convex set and [0, 1]f (x1 + (1 )x2 ) f (x1 ) + (1 )f (x2 )A function f is concave if f is convexInstructor: Arindam BanerjeeConvex Analysis and OptimizationFirst Order Conditionsf is convex i
Minnesota - CSCI - 5512
Nonparametric ModelsCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 4, 2012Instructor: Arindam BanerjeeNonparametric ModelsParametric Vs NonparametricParametric modelsInstructor: Arindam BanerjeeNonparametric ModelsParametri
Minnesota - CSCI - 5512
Support Vector MachinesCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 8, 2012Instructor: Arindam BanerjeeSupport Vector MachinesLinear SeparatorsInstructor: Arindam BanerjeeSupport Vector MachinesLinear SVMs: Separable Case
Minnesota - CSCI - 5512
BoostingCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 11, 2012Instructor: Arindam BanerjeeBoostingEnsemble LearningUse a collection of hypothesis from the hypothesis spaceInstructor: Arindam BanerjeeBoostingEnsemble Learni
Minnesota - CSCI - 5512
Statistical LearningCSci 5512: Articial Intelligence IIInstructor: Arindam BanerjeeApril 16, 2012Instructor: Arindam BanerjeeStatistical LearningFull Bayesian learningThe Bayesian view of learningInstructor: Arindam BanerjeeStatistical LearningF
Minnesota - CSCI - 5525
PCA vs FA! PCA! FAProject x to zCombine z to xz = WT(x !)x ! = Vz + !xzzxE. Alpaydin, Introduction to Machine LearningFactor Analysis! Finda small number of factors z, which whencombined generate x :xi !i = vi1z1 + vi2z2 + . + vikzk + !iw
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Dimension ReductionRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaFeature SelectionNP-hard to search through all the combinations Needheuristic solutionsThe assumption is bas
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Dimension ReductionRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaLinear Discriminant Analysis Finda low-dimensionalspace such that when xis projected, classes arewell-separ
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Course OverviewRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaWelcome to CSci 5525Course: Machine LearningInstructor: Rui Kuang (Ray), Assistant Professor (CS&amp;E) Contact:Offi
Minnesota - CSCI - 5525
CHAPTER 5:Multivariate MethodsE. Alpaydin, Introduction to Machine LearningMultivariate Data Multiplemeasurements (sensors) d inputs/features/attributes: d-variate N instances/observations/examples111X X X 12d 22212dX X X X= NN
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Bayes DecisionTheory andParametric ModelsRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaRegression exampleCoefficients increase inmagnitude as orderincreases:1: [-0.0769, 0
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Bayes DecisionTheory andParametric ModelsRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaParametric vs NonparametricParametric methods: Amodel (usually a type of simple distr
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Bayes DecisionTheory andParametric ModelsRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaParametric Classification Discriminantfunctiongi ( x ) = p( x | Ci ) P (Ci )orgi (
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Supervised LearningRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaSupervised Learning Classification RegressionInput Feature Space&quot; x1 %\$'\$ x2 'x = \$ . '\$'\$ . '\$xD '#&amp;
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Supervised LearningRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaSupervised Learning ClassificationData: RegressionX = cfw_x t,r t N=1tX = cfw_x t,r t N=1trt &quot; #(Clas
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Supervised LearningRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaNoise and Model ComplexityGiven similar training error,use the simpler oneSimpler to use (lowercomputational
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)NonparametricMethodsRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaDensity EstimationGiven the training set X=cfw_xtt drawn iid from p(x)Divide data into bins of size h Histo
Minnesota - CSCI - 5525
CSCI5525: Machine Learning (Spring 2012)ClusteringRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaExpectation-Maximization (EM)Complete likelihood, Lc( |X,Z), in terms of x and zLc (&quot; | X ) = log # p(x t , zt | &quot;) = \$t
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Linear DiscriminationRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaLikelihood- vs. Discriminantbased ClassificationLikelihood-based: Assume a model for p(x|Ci),use Bayes rule
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Linear DiscriminationRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaLogistic DiscriminationlogTwo classes: Assume log likelihood ratio is linearp (C1 | x )p (C1 | x )= log=
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Local ModelsRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaOnline k-meansWinner-take-all networkWeight decay term:!mij = !bit ( x tj &quot; mij ) = !bit x tj &quot; !bit mijE. Alpaydin
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Multilayer PerceptronRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaBiological Neural Nets Pigeonsas artexperts (Watanabe et al. 1995)Experiment: Pigeonin Skinner box Pres
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Multilayer PerceptronRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaBackpropagationHy i = vT z = &quot; v ih zh + v i 0ih =1zh = sigmoid ( wT x )h=1*\$ d&amp;&quot; w hj x j + w h 0 '
Minnesota - CSCI - 5525
CSCI 5525: Machine Learning (Spring 2012)Multilayer PerceptronRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaTuning the Network SizeDestructiveWeight decay:penalizing non-zeroparameters The same as addingadditiona
Minnesota - CSCI - 5525
CSCI5525: Machine Learning (Spring 2012)ClusteringRui KuangDepartment of Computer Science and EngineeringUniversity of MinnesotaMixture of GaussiansKp( x | &quot;) = % # iN ( x | i , \$i ),i =1Kwith 0 &amp; # i &amp; 1 and % # i = 1.i =1Mixture of Gaussians