{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Chapter1-NLP-Basics

# Chapter1-NLP-Basics - One-dimensional Taylor’s Series...

This preview shows pages 1–10. Sign up to view the full content.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: One-dimensional Taylor’s Series Approximation (Taylor1.doc) Let f be a function which has continuous derivatives to order (n+1) in an interval [51, b]. The Taylor’s series polynomial expansion, 1;, off at some point x0 in [a, b] is given by _ 2 — n Tl(x;xo)=f(xo)+f’(Iu)(I—Io)+f”(xo)(x 2x0) +-.-+f(ﬂ)(xo)(x n—TO) where f’, f ”, and f m are ﬁrst, second, and nEh derivatives of the function f with respect to x. The error between the true function f at x and the approximation I], at x is given by n+1 R x:x = (n+1) f Em.- n( o) f ( ) (n + I)! where 5c is a point between x and x0. In general we don’t know the exact value of the point i but this remainder or error term frequently allows for an upper bound on the error to be computed. Example: Consider approximating the function 2‘ about the point IO = 0. That is, we want a polynomial approximation for e‘. (The special case of Taylor’s series using x0 = 0 as the expansion point is called a lVlaclautin2 expansion.) The approximation is developed as (ex) :33, (ex) = 1"”,‘1 (en)_ez' (dX) yielding __ 2 L n _ __ n+1 e‘=e°+e°(x—0)+e°(x 0) +---+e°(x 0) -i—ei‘——(I 0) , n! (n+1)! which reduces to the well known series 1 x2 x3 Id In 2 xn+l e 21+x+-—-+—+—+-~-+—+e . 2 3! 4! n! (n+1)! The error of the approximation, that is the error if we omit the remainder term, for x e [-1, l] is bounded by ‘ 184-1 1 e e = (n+1)! '(n+1)1‘ ZWBEC-anuse this ideaito develop -inte‘grlation*~forrnul‘a§ for. difﬁculti'funcloris; {Consider H integrating e'x over [0, ":3. Then substituting -x for x and integrating the expansion term by term we obtain Note: If we apply Taylor’s expansion using only the constant term with the error term being the linear term, we have f(X)= f(xo)+f’(i)(x—xo)i with it being some point between x and 1:0. This yields the standard derivative form of the well known mean value theorem (Buck, page 43), where i is a point between x and x0, f(x)-f(xo) (it—10) I f '(i) = Homework: 1. Use the methods of Taylor’s series expansion, substitution and term by term integration to integrate Answer: te"’1dx=t-t—+—-—u-+---. ° 3 10 42 2. Expand a’ in a Taylor’s series about the point 0. 3. Expand sin(x) (x in degrees) about the point 0. Footnotes: 1 Brook Taylor (1685-1731), English mathematican. 2 Colin Maclaurin (1698—1746), Scotch mathematican. References: Kreyszig, Erwin. 1962. Advanced Engineering Mathematics. John Wiley and Sons, Inc., New York. For short biosketches of famous contributors to mathematics. Stationary Points: maximum, minimum, and saddle points off (StanionaryPointsldoc) The deﬁnition of a derivative of a function f at a point x° is I _ - f(xo+h)—f(xo) f (x°)—}‘1_gg—-——h——-— when this limit exists. Note that the direction of approaching the limit point is immaterial (here it approaches zero from above or below) and the limit must be the same from any direction for the derivative to exist. (In multiple dimensions the same property must hold, but there are an inﬁnite number of directions from which to approach the limit point.) We use the limit concept to show that if a function with continuous ﬁrst derivative, has a local minimum at a point, say x°, then f ’(x0) = 0. (The following proof follows that of Protter and Morrey, 1964.) A point x° is a local minimum of a function f with continuous ﬁrst derivative, if for all x in a neighborhood of x0, denoted N,3 (x0) = {xlix —- x°|| < E} , the following is true f(x) 2 for“), for allx e N£(x°). Assuming that f ’ exists and is continuous on N€(x°) , we have that f(x° +h) 2 ﬁx”), for allx = x° Ht 6 N,(x°) and, thus, o .. 0 W20, h>0andforallx=xo+h€Ne(xo)- The limit must also be bounded and thus, f ’(xo) 2 0. Now considering I: < 0, we have due to the division by a negative number causes the inequality sign to be reversed , _ 5 W50, h<0andforallx=xo+hENe(xo) and this implies that f ’(xo) 5 0. Hence, since the limits must be the same, the only possibility is that f ’(x0) = 0. Therefore, if x° is a minimum of a function f with a continuous ﬁrst derivative on the interior of any interval for x, then f ’(x0) = 0. The derivative equal to zero property is a necessary but not sufﬁcient condition for a point to be a minimum. It turns out that the same condition, f ’(x0) = 0, applies to local maximum points as well. This is also true for inﬂection points. Hence, we have a class of points, S f, called stationary points which satisfy s, = {arm = 0}. Figure I is a graph of a function showing the tangent planes to the curve with zero slope. The points of tangency are the stationary points of the function. That is, points where the ﬁrst derivative is zero. Figure 1. Graph of a function with tangent planes with zero slepes identiﬁed The stationary point condition is a method for isolating maximum, minimum and inﬂection points of a function. But we need to be able to distinguish between the stationary points according to type (max, min, inﬂection). A multiple-dimensional inﬂection point is called a saddle point and is the term most commonly applied to this category of stationary points. This discrimination process is based on the Taylor’s series expansion previously developed. Recall the Taylor’s series expansion of the function f with a second order error term is (x x) f(X)= f(x°’°—)+f(X)(x -X°)+f”(x)-—---— 2 wherej? is a point between x and x". When x" is a stationary point, we have, since flcrﬂ) = 0’ (x— X)2 f(X)= f(x°)+f”(X)-———— 2 and if x0 is a strict—local minimum point then f (x) > f (x0). This implies that (x- I")2 2 f ”(x) > 0. This condition holds if f "(2) > 0. Note here that we do not know the exact value of 5:; only that we can make it as near to x° as needed since the local minimum condition holds for all x in an E neighborhood about x°, that is, for all x E (x° -€,x° + 5). However, since f ” is assumed to be a continuous function, then the sign of f " at x0 and at i will be the same. This continuity condition allows us to test the sign of f ” at the known stationary point x0 rather than the unknown point 1?. This leads us to the discriminate test for stationary points, where x° is a point such that f ’(x°)= 0 and if _ f ”(x°) > 0, then x° is a local minimum, f ”(x°) < 0, then x0 is a local maximum, f ”(x°) = 0, need to test higher order approximations. When the second derivative at the stationary point is zero, then we must go to the third derivative term. If the third derivative term is zero, f ”’(x°) = 0, then we continue with the fourth derivative term, etc., until we obtain a non-zero term at the stationary point. Ifthe ﬁrst non-zero tennis an odd power, f w (x0) at Owith n-odd, then the point is an inﬂection point (or saddle point). If the power it is even, then the cases are similar to that of f”(x°). That is, if n is even and f“’(x°) > 0 then x0 is a local minimum and if n is even and f‘")(x°) < 0 then x° is a local maximum. Example 1: 4 3 2 Find the stationary point(s) of the function f(x) = x? — 3’3“" —- 925- + 6x + 5 and determine their type(s). This function is graphed in Figure 2. The derivative function f’, graphed in Figure 3, is f’(x)=x3—2x2-5x+6. This function can be factored as f'(x) = x3 —2x2 -5x+ 6 ._. (x—l)(x+2)(x—3) and the stationary points are x, = 1, x2 = —2, and x3 = 3. To determine what type they are, we evaluate f ”(x,) for each stationary point x, and classify the points according to the sign. The function f”(x) = 3x2 - 4x + 5 and thus, (1') at the point x =1, f”(l) = 3— 4—- 5 = —6 < 0, and is a local maximum point, (ii) at the point x = -2, f”(—-2) = 12+ 8- 5 =15 > O, and is a local minimum point, and (iii) at the point x = 3, f”(3) z 27*12— 5 =10 > 0, and is a local minimum point. Figure 2. Plot of the function f over [-5, 5]. f'tx) Figure 3. Plot of f’ over [-5, 5]. Notice that the zero’s of the f ' function coincide with maximum and minimum points of the function. Thus, our solution method is to find the roots of the function f’ which, hopefully, can be obtained with less work than directly searching for the maximum and - minimum points. Example 2: Find the stationary point(s) of the function f (x) = (x — 1)5 and determine their type(s). The derivative function f’ is f’(x) = 5(x — 1)4 = 0, at the point x =1. The derivatives yield f ”(x) = 20(x — D3, which is zero at the point x = 1, f ”'(x) = 600: -1)2, which is zero at the point x = 1, f (4) (x) = 120(x - 1), which is zero at the point x =1, and f ‘5’ (x) = 120, which is non - zero at the point x = 1. Thus, since the 5‘h derivative is non-zero, then the stationary pointx = 1 is an inflection point. Homework: Find the stationary points, if they exist, and determine their type. 1- f (x) = x3 + x 2. f(x) = x4 +Jt2 3. f(x) = 615 —-4x+lO References: Protter, Murray H. and Charles B. Morrey, Jr. 1964. College Calculus with Analytic Geomegg. Addison-Wesley Publishing Company, Inc., Reading, Mass. Generalization to n Dimensions: The necessary conditions for a stationary point for a function of n variables is that all the ﬁrst—order partial derivative are zero. This means that Wzo, foralli='1,2i"'=n‘ (2) L To see why this result is true, consider the n-dimensional function f(x1,x2, - -- ,x_) which has a local minimum at the point (xﬂxg, ---,x,?) if 0 0 O 0 0 0 f(x1.x2,-~,x,.)2f(x,,x2,---,x,), forall(x1,x2,'~-,x,)eNE(x1Jew-ax.) Again, a point is in ane neighborhood of another point if the distance between the two points is less than E, or 2(xi—xf’r <6. {:1 Censider a variation in the 1"“ coordinate about the point (xf,x§ , ,x? , - -- ,xf). This new point is of the form (xlogé', ~-,x? +h, ,xS) where his small; technically «I? < G. Since this point is within thee neighborhood of the test point, then 0 D 0 G f(x,°,x2,---,x?+h,~-,x3)2f(x1.x2.---, f,‘--,x,). This implies for h > 0, that. 0 U 0 0 0 0 0 0 f(x12xzs”'sx; +h:"'axn)—f(x1:xzan'axl's'uvxﬂ) >0 Wk ._ . Taking the limit as h approaches zero as a positive number this becomes 0 0 D 0 Qf(xl,x2,'--,xi,---,x,,) 8x. I 20. Taking the limit as h approaches zero as a negative number yields the following, note that dividing by a negative number changes the direction of the inequality, 0 U 0 0 J(x1,12,...,xi,.n’xn) 3x- I 50. These two conditions holding simultaneously imply that the ﬁrst partial of f with respect to xi must be zero. Thus, we have the general multiple dimensional local minimum point condition stated above. By the same argument, with inequalities ﬂipped, we ﬁnd the same condition must hold at a maximum point. So the general multiple-dimension stationary point condition is that of equation (2). One-Dimensional Newton-Raphson Method Our problem is to ﬁnd an 1? such that go?) = 0. We approach this problem by making a guess for the root, call it x1, and then use Taylor’s Series to deve10p a linear approximation to 3 about this point: 30:) = g(x‘) + g’(x1)(x - x1). The idea is to ﬁnd g(x) = 0, so we set the linear appr0ximation to zero, the right-hand side of the above equation, and then solve for the value of x, call it x2, which satisﬁes this condition 3(x‘)+ g’(X‘)(x-x1) = 0, which yields the new estimate of the root, x2, as x2 =x1_ 30511)- g’(x ) If we continue this process, we have the general iteration formula xk+l =xk _ 3(1") 3’06") e The sequence of points, {xk};1 frequently converges to a point, call it )2, and this point has the property that go?) = 0. That is, the limit point it is a root of the function 3. Note that convergence is not guaranteed. When this sequence converges, it does so very fast, so that we can truncate the sequence when we get within an acceptable tolerance of the root. This process is illustrated below. Figure 2. The N ewton-Raphson root-finding process. The function gl is the linear approximation to g at the point x1 = 3.5. The function gll is the linear approximation to g at the second iteration point x2 = 4.75 . Example. Find a root of the function g(x) = x3 —11x2 + 39x: 45. This function has two distinct roots: 3 and 5. The following table displays two sets of iterations of the process using the starting points 2 and 6. From the initial point x1 = 2, the process converges to the root SE = 3 . From the initial point x1 = 6, the process converges to the root )2 = 5. The initial and undated points as well as the function value are displayed for each of 10 iterations. k+1 ) 30: 6.000 5.400 2.3040 5.400 5.100 0.4410 5.100 5.009 0.0351 5.009 5.000 0.0003 5.000 5.000 0.0000 5.000 5.000 0.0000 5.000 5.000 0.0000 5.000 5.000 0.0000 5.000 5.000 0.0000 5.000 5.000 0.0000 \DOOHJONUl-h-UJNHR‘ l—' 0 Problems 1. Find a root of the function x2 - 2x—8 using the Newton-Raphson method with starting - 1 pomt x = 3. 2. For Problem 1, can you find a starting point which will cause the Newton—Raphson method to fail? ...
View Full Document

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern