sga.pdf - Optimization Methods in Machine Learning, Higher...

Optimization Methods in Machine Learning, Higher School of Economics
Staff Graded Assignment: Experiments
Report in PDF format about the conducted research.
Each conducted experiment should be arranged as a separate section in the PDF document (section
name name of the corresponding experiment). For each experiment, you must first write a description of
it: what functions are optimized, how data is generated, what methods and with what parameters are used.
Next, the results of the corresponding experiment should be presented graphs, tables, etc. Finally, after
the results of the experiment, your conclusions should be written what dependence is observed and why.
Important: The report should not contain any code. Each graph should be commented on what it
shows, what conclusions can be drawn from this experiment. The axes must be signed. If several curves are
drawn on the chart, then there should be a legend. The lines themselves should be drawn thick enough so
that they are clearly visible. The following figures (1, 2) are examples of required graphs.
Figure 1: Investigation of the number of iterations to convergence of the method for different dimensions
and the condition number. The set of colored lines corresponds to running algorithms from random data.
Figure 2: An example of how the algorithm works for different starting points.
1

1 Practical task 1: Gradient descent and Newton methods.
1.1 Experiment: Gradient descent trajectory on a quadratic function
Analyze the gradient descent trajectory for several quadratic functions: come up with two or three
quadratic two-dimensional functions on which the method will work differently, draw graphs with function-
level lines and method trajectories.
Try to answer the following question: How does the behavior of the method differ depending on the number
of conditionality of the function, the choice of the starting point, and the step selection strategy (constant
strategy, Armijo, Wolfe)?
To draw level lines, you can use the plot_levels function, and to draw trajectories plot_trajectory
from the file plot_trajectory_2d.py , attached to the task.
Also note that the oracle of the quadratic function QuadraticOracle is already implemented in the
oracles module. It implements the function f ( x ) = (1 / 2) h Ax, x i - h b, x i , where A ∈ S n
++
, b ∈ R n .
1.2 Experiment: Dependence of the number of iterations of gradient descent
on the number of conditionality and dimension of the space
Investigate how the number of iterations required for gradient descent to converge depends on the fol-
lowing two parameters: 1) conditionality numbers κ ≥ 1 of the optimized function and 2) the dimension of
the space n of the optimized variables.
To do this, for the given parameters n and κ , generate a random quadratic problem of size n with
the condition number κ and run a gradient descent on it with some fixed required accuracy. Measure the
number of iterations of T ( n, κ ) that the method needed to make before convergence (successful exit by the
stop criterion).