Meili Zhang (@mlm404) 在 【进行时】精读paper: Symbolic Discovery of Optimization Algorithms (2024 NeurIPS) 中发帖
摘要:
Topic:
Algorithm discovery as program search.
The discovered algorithm used for Neural Network training.
Problem:
infinite and sparse program space → efficient search techniques
gap between proxy and target space → selection and simplification strategy
Advantages:
memory efficient
better performance compared with Adam (applied to many model training tasks)
Analysis (better performanc...