1 citations · 1 across the 16 of their papers we have counts for
6 papers · 1 filter
From Attention to Activation: Unravelling the Enigmas of Large Language Models
Prannay Kaul, Chengcheng Ma, Ismail Elezi +1
We study two strange phenomena in auto-regressive Transformers: (1) the dominance of the first token in attention heads; (2) the occurrence of large outlier activations in the hidd…
Fractal Calibration for long-tailed object detection
Konstantinos Panagiotis Alexandridis, Ismail Elezi, Jiankang Deng +2
Real-world datasets follow an imbalanced distribution, which poses significant challenges in rare-category object detection. Recent studies tackle this problem by developing re-wei…
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Roy Miles, Pradyumna Reddy, Ismail Elezi +1
Large language models (LLMs) have recently emerged as powerful tools for tackling many language-processing tasks. Despite their success, training and fine-tuning these models is st…
G3DR: Generative 3D Reconstruction in ImageNet
Pradyumna Reddy, Ismail Elezi, Jiankang Deng
We introduce a novel 3D generative method, Generative 3D Reconstruction (G3DR) in ImageNet, capable of generating diverse and high-quality 3D objects from single images, addressing…
Deep Active Learning: A Reality Check
Edrina Gashi, Jiankang Deng, Ismail Elezi
We conduct a comprehensive evaluation of state-of-the-art deep active learning methods. Surprisingly, under general settings, no single-model method decisively outperforms entropy-…
Improving Knowledge Distillation using Orthogonal Projections
Roy Miles, Ismail Elezi, Jiankang Deng
Knowledge distillation is an effective method for training small and efficient deep learning models. However, the efficacy of a single method can degenerate when transferring to ot…