75 citations · 111 across the 11 of their papers we have counts for
13 papers
Sliding-window beats linear attention
Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron +1
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy: every new token costs more than the previous one, and its keys and values…
Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
Arjun Krishnakumar, Rhea Sanjay Sukthanker, Hannan Javed Mahadik +5
Small Language models (SLMs) offer an efficient and accessible alternative to Large Language Models (LLMs), delivering strong performance while using far fewer resources. We introd…
Compressing Large Language Models with Automated Sub-Network Search
Rhea Sanjay Sukthanker, Benedikt Staffler, Frank Hutter +1
Large Language Models (LLMs) demonstrate exceptional reasoning abilities, enabling strong generalization across diverse tasks such as commonsense reasoning and instruction followin…
HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +4
The increasing size of language models necessitates a thorough analysis across multiple dimensions to assess trade-offs among crucial hardware metrics such as latency, energy consu…
Multi-objective Differentiable Neural Architecture Search
Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler +3
Pareto front profiling in multi-objective optimization (MOO), i.e., finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that req…
Weight-Entanglement Meets Gradient-Based Neural Architecture Search
Rhea Sanjay Sukthanker, Arjun Krishnakumar, Mahmoud Safari +1
Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than tr…
Neural Architecture Search: Insights from 1000 Papers
Colin White, Mahmoud Safari, Rhea Sukthanker +5
In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, a…
Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars
Simon Schrodi, Danny Stoll, Binxin Ru +3
The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towar…
Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
Samuel Dooley, Rhea Sanjay Sukthanker, John P. Dickerson +3
Face recognition systems are widely deployed in safety-critical applications, including law enforcement, yet they exhibit bias across a range of socio-demographic dimensions, such…
Generative Flows with Invertible Attentions
Rhea Sanjay Sukthanker, Zhiwu Huang, Suryansh Kumar +2
Flow-based generative models have shown an excellent ability to explicitly learn the probability density function of data via a sequence of invertible transformations. Yet, learnin…
Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution
Yan Wu, Zhiwu Huang, Suryansh Kumar +3
Modern solutions to the single image super-resolution (SISR) problem using deep neural networks aim not only at better performance accuracy but also at a lighter and computationall…
Neural Architecture Search of SPD Manifold Networks
Rhea Sanjay Sukthanker, Zhiwu Huang, Suryansh Kumar +3
In this paper, we propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks, aiming to automate the design of SPD neural archite…
Anaphora and Coreference Resolution: A Review
Rhea Sukthanker, Soujanya Poria, Erik Cambria +1
Entity resolution aims at resolving repeated references to an entity in a document and forms a core component of natural language processing (NLP) research. This field possesses im…