124 citations · 231 across the 10 of their papers we have counts for
10 papers
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
Steven Kolawole, Lucio Dery, Jean-François Kagy +3
Structured pruning is a promising approach to create smaller, faster large language models. However, existing methods typically rely on computing the gradient via backward passes,…
Multitask Learning Can Improve Worst-Group Outcomes
Atharva Kulkarni, Lucio Dery, Amrith Setlur +3
In order to create machine learning systems that serve a variety of users well, it is vital to not only achieve high average performance but also ensure equitable outcomes across d…
DeMuX: Data-efficient Multilingual Learning
Simran Khanuja, Srinivas Gowriraj, Lucio Dery +1
We consider the task of optimally fine-tuning pre-trained multilingual models, given small amounts of unlabelled target data and an annotation budget. In this paper, we introduce D…
Cross-Modal Fine-Tuning: Align then Refine
Junhong Shen, Liam Li, Lucio M. Dery +4
Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many ot…
Multi-step Planning for Automated Hyperparameter Optimization with OptFormer
Lucio M. Dery, Abram L. Friesen, Nando De Freitas +2
As machine learning permeates more industries and models become more expensive and time consuming to train, the need for efficient automated hyperparameter optimization (HPO) has n…
AANG: Automating Auxiliary Learning
Lucio M. Dery, Paul Michel, Mikhail Khodak +2
Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whil…
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
Lucio M. Dery, Paul Michel, Ameet Talwalkar +1
In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for levera…
Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
Lucio M. Dery, Yann Dauphin, David Grangier
While deep learning has been very beneficial in data-rich settings, tasks with smaller training set often resort to pre-training or multitask learning to leverage data from other t…
Audio to Body Dynamics
Eli Shlizerman, Lucio M. Dery, Hayden Schoen +1
We present a method that gets as input an audio of violin or piano playing, and outputs a video of skeleton predictions which are further used to animate an avatar. The key idea is…
Weakly Supervised Classification in High Energy Physics
Lucio Mwinmaarong Dery, Benjamin Nachman, Francesco Rubbo +1
As machine learning algorithms become increasingly sophisticated to exploit subtle features of the data, they often become more dependent on simulations. This paper presents a new…