# sga.pdf - Optimization Methods in Machine Learning, Higher...

• 5

Course Hero uses AI to attempt to automatically extract content from documents to surface to you and others so you can study better, e.g., in search results, to enrich docs, and more. This preview shows page 1 - 3 out of 5 pages.

Optimization Methods in Machine Learning, Higher School of EconomicsStaff Graded Assignment: ExperimentsReport in PDF format about the conducted research.Each conducted experiment should be arranged as a separate section in the PDF document (sectionnamename of the corresponding experiment). For each experiment, you must first write a description ofit: what functions are optimized, how data is generated, what methods and with what parameters are used.Next, the results of the corresponding experiment should be presentedgraphs, tables, etc. Finally, afterthe results of the experiment, your conclusions should be writtenwhat dependence is observed and why.Important:The report should not contain any code. Each graph should be commented onwhat itshows, what conclusions can be drawn from this experiment. The axes must be signed. If several curves aredrawn on the chart, then there should be a legend. The lines themselves should be drawn thick enough sothat they are clearly visible. The following figures (1, 2) are examples of required graphs.Figure 1: Investigation of the number of iterations to convergence of the method for different dimensionsand the condition number. The set of colored lines corresponds to running algorithms from random data.Figure 2: An example of how the algorithm works for different starting points.1
1Practical task 1: Gradient descent and Newton methods.1.1Experiment: Gradient descent trajectory on a quadratic functionAnalyze the gradient descent trajectory for several quadratic functions:come up with two or threequadratictwo-dimensionalfunctions on which the method will work differently, draw graphs with function-level lines and method trajectories.Try to answer the following question:How does the behavior of the method differ depending on the numberof conditionality of the function, the choice of the starting point, and the step selection strategy (constantstrategy, Armijo, Wolfe)?To draw level lines, you can use theplot_levelsfunction, and to draw trajectoriesplot_trajectoryfrom the fileplot_trajectory_2d.py, attached to the task.Also note that the oracle of the quadratic functionQuadraticOracleis already implemented in theoraclesmodule. It implements the functionf(x) = (1/2)hAx, xi - hb, xi, whereASn++,bRn.1.2Experiment: Dependence of the number of iterations of gradient descenton the number of conditionality and dimension of the spaceInvestigate how the number of iterations required for gradient descent to converge depends on the fol-lowing two parameters: 1) conditionality numbersκ1of the optimized function and 2) the dimension ofthe spacenof the optimized variables.To do this, for the given parametersnandκ, generate a random quadratic problem of sizenwiththe condition numberκand run a gradient descent on it with some fixed required accuracy. Measure thenumber of iterations ofT(n, κ)that the method needed to make before convergence (successful exit by thestop criterion).

Course Hero member to access this document

Course Hero member to access this document

End of preview. Want to read all 5 pages?