1 log 1 1 1 Note 1 1 11 2 1 1 2 89 Alternative Estimating parameters \u2113 log 1

1 log 1 1 1 note 1 1 11 2 1 1 2 89 alternative

This preview shows page 89 - 101 out of 139 pages.

𝑖 ) 𝑥 𝑖 ] 𝑛 𝑖=1 = log 1 1 + 𝑒 −𝜷𝑦 ( 𝑖 ) 𝑥 ( 𝑖 ) 𝑛 𝑖=1 Note 𝑝 𝑡 = 1 1+𝑒 −𝑡 𝑑𝑝 ( 𝑡 ) 𝜕𝑡 = ( 1)(1 + 𝑒 −𝑡 ) −2 𝑒 −𝑡 1 = 𝑒 −𝑡 1+𝑒 −𝑡 2 = 𝑝 𝑡 𝑝 ( −𝑡 ) 89
Image of page 89
Alternative: Estimating parameters 𝜷 = log[ 𝑝 𝑦 ( 𝑖 ) 𝑥 𝑖 ] 𝑛 𝑖=1 = log 1 1+𝑒 −𝜷𝑦 ( 𝑖 ) 𝑥 ( 𝑖 ) 𝑛 𝑖=1 𝜕ℓ 𝜕𝛽 𝑗 = 1 𝑝 𝑦 ( 𝑖 ) 𝑥 𝑖 𝑛 𝑖=1 𝑝 𝑦 ( 𝑖 ) 𝑥 𝑖 𝑝 −𝑦 ( 𝑖 ) 𝑥 𝑖 𝑦 ( 𝑖 ) 𝑥 𝑗 ( 𝑖 ) = � 𝑝 −𝑦 ( 𝑖 ) 𝑥 𝑖 𝑦 ( 𝑖 ) 𝑥 𝑗 ( 𝑖 ) 𝑛 𝑖=1
Image of page 90
Alternative: Gradient Ascent Updates to all parameters done in one shot 𝜷′ = 𝜷 + 𝛼 𝜕𝑓 𝜷 𝜕𝜷 = 𝜷 + 𝛼 � 𝑝 −𝑦 ( 𝑖 ) 𝑥 𝑖 𝑦 ( 𝑖 ) 𝒙 ( 𝒊 ) 𝑛 𝑖=1 91 m-dimensional row vector
Image of page 91
Gradient Ascent: Vectorize 𝜷 = 𝜷 + 𝛼 𝜕𝑓 𝜷 𝜕𝜷 = 𝜷 + 𝛼 � 𝑝 −𝑦 ( 𝑖 ) 𝑥 𝑖 𝑦 ( 𝑖 ) 𝒙 ( 𝒊 ) 𝑛 𝑖=1 𝛽 1 𝛽 𝑚 = 𝛽 1 𝛽 𝑚 + 𝛼 𝑦 ( 1 ) 𝑝 ( −𝑦 ( 1 ) 𝑥 1 ) 𝑦 ( 𝑛 ) 𝑝 ( −𝑦 ( 𝑛 ) 𝑥 𝑛 ) 𝑥 1 ( 1 ) 𝑥 𝑚 ( 1 ) 𝑥 1 ( 𝑛 ) 𝑥 𝑚 ( 𝑛 ) 92
Image of page 92
Alternative: Gradient Descent 𝜷 = 𝜷 ( 1 ∗ 𝛼 𝜕𝑓 𝜷 𝜕𝜷 ) = 𝜷 𝛼 � 𝑝 −𝑦 𝑖 𝑥 𝑖 𝑦 𝑖 𝒙 𝒊 𝑛 𝑖=1 = 𝜷 − 𝛼 � − 1 − 𝑝 𝑦 𝑖 𝑥 𝑖 𝑦 𝑖 𝒙 𝒊 𝑛 𝑖=1 = 𝜷 − 𝛼 � 𝑝 𝑦 𝑖 𝑥 𝑖 1 𝑦 𝑖 𝒙 𝒊 𝑛 𝑖=1 = 𝜷 − 𝛼 � 1 1 + 𝑒 −𝑦 ( 𝑖 ) 𝛽𝑥 ( 𝑖 ) 1 𝑦 𝑖 𝒙 𝒊 𝑛 𝑖=1 93 Used in Spark example code
Image of page 93
Example codes logistic regression ./bin/spark-submit examples/src/main/python/logistic_regression.py data/mllib/lr_data.txt 10 94
Image of page 94
Training Data for Logistic Regression data/mllib/lr_data.txt 10-dimensional data points with class label Format: class label (1/-1), x1, x2, … For example: 1 2.1419053154730548 1.919407948982788 … -1 0.3529201223821201 -2.33440404253745 … 95
Image of page 95
Logistic regression with Spark from operator import add import numpy as np D = 10 points = sc.textFile('data/mllib/lr_20_data.txt').mapPar titions(readPointBatch).cache() # Note that Spark divides data into 2 partitions 96
Image of page 96
Logistic regression iterations = 10 # Initialize w to a random value w = 2 * np.random.ranf(size=D) - 1 print("Initial w: " + str(w)) for i in range(iterations): print("On iteration %i" % (i + 1)) w -= points.map(lambda m : gradient(m, w)).reduce(add) print("Final w: " + str(w)) 97 Matrix
Image of page 97
Logistic regression def readPointBatch(iterator): strs = list(iterator) matrix = np.zeros((len(strs), D + 1)) for i, s in enumerate(strs): matrix[i] = np.fromstring(s.replace(',', ' '), dtype=np.float32, sep=' ') return [matrix] 98
Image of page 98
Logistic regression def gradient(matrix, w): Y = matrix[:, 0] # point labels X = matrix[:, 1:] # point coordinates # For each point (x, y), compute gradient function, then sum these up return ((1.0 / (1.0 + np.exp(-Y * X.dot(w))) - 1.0) * Y * X.T).sum(1) 99
Image of page 99
Logistic regression Consider 20 data points, 10 dimensions/each point X is a 20-by-10 array w is a 1-d array (size 10) X.dot(w) computes dot product of w with each
Image of page 100
Image of page 101

You've reached the end of your free preview.

Want to read all 139 pages?

  • Fall '14
  • Maximum likelihood, Likelihood function, Logistic function, RDD

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture

  • Left Quote Icon

    Student Picture