2 papers
cs.LG2025
Scaling Recurrent Neural Networks to a Billion Parameters with Zero-Order Optimization
Francois Chaubard, Mykel Kochenderfer
During inference, Recurrent Neural Networks (RNNs) scale constant in both FLOPs and GPU memory with increasing context length, as they compress all prior tokens into a fixed-size m…
cs.LG2024
Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering
Francois Chaubard, Duncan Eddy, Mykel J. Kochenderfer
We introduce Gradient Agreement Filtering (GAF) to improve on gradient averaging in distributed deep learning optimization. Traditional distributed data-parallel stochastic gradien…