Searched Step Methods

“Work smarter, not harder.”

If we don’t want to take so many steps, we need to rely on additional information to find out what’s the largest step to take in each iteration:

Backtracking Line Search: in each step, reduce the stepsize until a valid stepsize (one that actually results in a descent of objective) is found.

Steepest Descent: the most specific definition of this method is a.k.a. exact line search, we can calculate gradient of objective function against stepsize as well, when such gradient = 0, a steepest descent stepsize is found.

Conjugate Gradient: The greediest method may become myobic sometimes, resulting in zigzagging in the optimization path. Adding a bit residue step from all the previous steps will speedup the convergence.

References:

https://www.youtube.com/watch?v=pQvxxKkceRw&list=LL&index=10

https://www.youtube.com/watch?v=h4cG8jLGmKg&list=LL&index=17&t=439s