144 citations · 471 across the 20 of their papers we have counts for
20 papers
Composable Function-preserving Expansions for Transformer Architectures
Andrea Gesmundo, Kaitlin Maile
Training state-of-the-art neural networks requires a high cost in terms of compute and time. Model scale is recognized to be a critical factor to achieve and improve the state-of-t…
Multipath agents for modular multitask ML systems
Andrea Gesmundo
A standard ML model is commonly generated by a single method that specifies aspects such as architecture, initialization, training data and hyperparameters configuration. The prese…
A Multiagent Framework for the Asynchronous and Collaborative Extension of Multitask ML Systems
Andrea Gesmundo
The traditional ML development methodology does not enable a large number of contributors, each with distinct objectives, to work collectively on the creation and extension of a sh…
A Continual Development Methodology for Large-scale Multitask Dynamic ML Systems
Andrea Gesmundo
The traditional Machine Learning (ML) methodology requires to fragment the development and experimental process into disconnected iterations whose feedback is used to guide design…
muNet: Evolving Pretrained Deep Neural Networks into Scalable Auto-tuning Multitask Systems
Andrea Gesmundo, Jeff Dean
Most uses of machine learning today involve training a model from scratch for a particular task, or sometimes starting with a model pretrained on a related task and then fine-tunin…
An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systems
Andrea Gesmundo, Jeff Dean
Multitask learning assumes that models capable of learning from multiple tasks can achieve better quality and efficiency via knowledge transfer, a key feature of human learning. Th…
Scaling Up Models and Data with and
Adam Roberts, Hyung Won Chung, Anselm Levskaya +40
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can…
Routing Networks with Co-training for Continual Learning
Mark Collier, Efi Kokiopoulou, Andrea Gesmundo +1
The core challenge with continual learning is catastrophic forgetting, the phenomenon that when neural networks are trained on a sequence of tasks they rapidly forget previously le…
Ranking architectures using meta-learning
Alina Dubatovka, Efi Kokiopoulou, Luciano Sbaiz +3
Neural architecture search has recently attracted lots of research efforts as it promises to automate the manual design of neural networks. However, it requires a large amount of c…
Flexible Multi-task Networks by Learning Parameter Allocation
Krzysztof Maziarz, Efi Kokiopoulou, Andrea Gesmundo +3
This paper proposes a novel learning method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharin…
Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation
Iulia M. Comsa, Krzysztof Potempa, Luca Versari +3
The timing of individual neuronal spikes is essential for biological brains to make fast responses to sensory stimuli. However, conventional artificial neural networks lack the int…
Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
Zalán Borsos, Andrey Khorlin, Andrea Gesmundo
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel arc…
Fast Task-Aware Architecture Inference
Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz +3
Neural architecture search has been shown to hold great promise towards the automation of deep learning. However in spite of its potential, neural architecture search remains quite…
Parameter-Efficient Transfer Learning for NLP
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski +5
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire n…
Neural Architecture Search Over a Graph Search Space
Stanisław Jastrzębski, Quentin de Laroussilhe, Mingxing Tan +3
Neural Architecture Search (NAS) enabled the discovery of state-of-the-art architectures in many domains. However, the success of NAS depends on the definition of the search space.…
Evolutionary-Neural Hybrid Agents for Architecture Search
Krzysztof Maziarz, Mingxing Tan, Andrey Khorlin +2
Neural Architecture Search has shown potential to automate the design of neural networks. Deep Reinforcement Learning based agents can learn complex architectural patterns, as well…
Transfer Learning with Neural AutoML
Catherine Wong, Neil Houlsby, Yifeng Lu +1
We reduce the computational cost of Neural AutoML with transfer learning. AutoML relieves human effort by automating the design of ML algorithms. Neural AutoML has become popular f…
Analyzing Language Learned by an Active Question Answering Agent
Christian Buck, Jannis Bulian, Massimiliano Ciaramita +4
We analyze the language learned by an agent trained with reinforcement learning as a component of the ActiveQA system [Buck et al., 2017]. In ActiveQA, question answering is framed…
Transfer Learning to Learn with Multitask Neural Model Search
Catherine Wong, Andrea Gesmundo
Deep learning models require extensive architecture design exploration and hyperparameter optimization to perform well on a given task. The exploration of the model design space is…
Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
Christian Buck, Jannis Bulian, Massimiliano Ciaramita +4
We frame Question Answering (QA) as a Reinforcement Learning task, an approach that we call Active Question Answering. We propose an agent that sits between the user and a black bo…