Meili Zhang (@mlm404)【进行时】精读paper: Symbolic Discovery of Optimization Algorithms (2024 NeurIPS) 中发帖

摘要: 
Topic:

Algorithm discovery as program search.
The discovered algorithm used for Neural Network training.

Problem:

infinite and sparse program space → efficient search techniques
gap between proxy and target space → selection and simplification strategy

Advantages:

memory efficient
better performance compared with Adam (applied to many model training tasks)

Analysis (better performanc...