Introduction

“Ask a man to improve and he resents you for a day; show a man to improve and he is grateful for a lifetime.” – Yuli

fishing

Personal Perspective on Optimization

  1. Why this website?
    • In my studies of Optimization, there is an abundance of academic resources available in the form of sophisticated mathematical derivations. These resources are often organized into specific methods or algorithms, providing a detailed account of their discovery and implementation methods, making them a useful starting point.
    • However, for end-users like myself seeking a suitable optimization method for a specific field of AI or Data Science, a more concise description of each method, interactive visualizations to quickly understand their behavior, and a comparison of methods to evaluate their performance would be beneficial.
    • Two works that greatly influenced my work include:
      • a. Professor Andrew Ning’s YouTube channel and his book on Optimization, which provided me with a comprehensive overview of Optimization and valuable insights into key methods.
      • b. Emilien Dupont’s GitHub page, which features an incredible 2D visualization comparing fixed-step gradient-based optimization algorithms. I have adopted a similar format and added additional functionalities and algorithms.
  2. What is optimization?
    • Optimization is to find the best solution among an infinite sea of possibilities. It is a dance between the mind and reality, where we seek to balance our aspirations with the limitations of the world.
  3. Cool things done by optimization.
    • Artificial Intelligence: from the very fundamental of gradient-based neural network training, to hyperparameter tuning, to the “structure from motion” in Computer Vision and the state-of-the-art Reinforcement Learning algorithms like TRPO and PPO.
    • self-driving cars: plan the routes and make realtime driving decisions.
    • design fighter-jets: improve the lift and maneuverability, reduce the weight and drag.
    • discover new drugs: balance the efficacy with toxicity and ease of synthesis.
    • manage power grid: schedule the generation and consumption of power, taking into account cost, reliability, and environmental impact.
  4. Less cool (in subjective sense) things done by optimization?
    • find the route to deliver groceries to your door.
    • place and schedule the advertisements on websites.
    • design slot machines to make them more addictive.
  5. Why optimization is more often minimizing than maximizing?
    • “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. – Antoine de Saint-Exupéry”
  6. Is the word ‘optimization’ related to optimism?
    • Both stem from the Latin word for ‘best’, but optimization is the yin to optimism’s yang - while optimism is about seeing the best in things, optimization is about making the best of things.

References:

Martins, J. R. R. A. and Ning, A., “Engineering Design Optimization”, Cambridge University Press, 2022. Retrieved from https://mdobook.github.io/

Nocedal, J., & Wright, S. (2006). “Numerical Optimization” (2nd ed.). Springer. Retrieved from https://www.numerical.rl.ac.uk/people/nimg/course/lectures/raphael/lectures/lec7slides.pdf

https://emiliendupont.github.io/2018/01/24/optimization-visualization/

https://optimization.cbe.cornell.edu/index.php?title=Main_Page

https://en.wikipedia.org/wiki/Mathematical_optimization