72 citations · 109 across the 11 of their papers we have counts for
12 papers
Short Data, Long Context: Distilling Positional Knowledge in Transformers
Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4
Extending the context window of language models typically requires expensive long-context pre-training, posing significant challenges for both training efficiency and data collecti…
Layer by Layer: Uncovering Hidden Representations in Language Models
Oscar Skean, Md Rifat Arefin, Dan Zhao +4
From extracting features to generating text, the outputs of large language models (LLMs) typically rely on the final layers, following the conventional wisdom that earlier layers c…
Does Representation Matter? Exploring Intermediate Layers in Large Language Models
Oscar Skean, Md Rifat Arefin, Yann LeCun +1
Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we inve…
Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier +4
Decoder-only Transformers often struggle with complex reasoning tasks, particularly arithmetic reasoning requiring multiple sequential operations. In this work, we identify represe…
VFA: Vision Frequency Analysis of Foundation Models and Human
Mohammad-Javad Darvishi-Bayazi, Md Rifat Arefin, Jocelyn Faubert +1
Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception ma…
Unsupervised Concept Discovery Mitigates Spurious Correlations
Md Rifat Arefin, Yan Zhang, Aristide Baratin +4
Models prone to spurious correlations in training data often produce brittle predictions and introduce unintended biases. Addressing this challenge typically involves methods relyi…
Amplifying Pathological Detection in EEG Signaling Pathways through Cross-Dataset Transfer Learning
Mohammad-Javad Darvishi-Bayazi, Mohammad Sajjad Ghaemi, Timothee Lesort +3
Pathology diagnosis based on EEG signals and decoding brain activity holds immense importance in understanding neurological disorders. With the advancement of artificial intelligen…
Continual Learning with Foundation Models: An Empirical Study of Latent Replay
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez +4
Rapid development of large-scale pre-training has resulted in foundation models that can act as effective feature extractors on a variety of downstream tasks and domains. Motivated…
Challenging Common Assumptions about Catastrophic Forgetting
Timothée Lesort, Oleksiy Ostapenko, Diganta Misra +4
Building learning agents that can progressively learn and accumulate knowledge is the core goal of the continual learning (CL) research field. Unfortunately, training a model on ne…
AvaTr: One-Shot Speaker Extraction with Transformers
Shell Xu Hu, Md Rifat Arefin, Viet-Nhat Nguyen +3
To extract the voice of a target speaker when mixed with a variety of other sounds, such as white and ambient noises or the voices of interfering speakers, we extend the Transforme…
A Statistical Real-Time Prediction Model for Recommender System
Md Rifat Arefin, Minhas Kamal, Kishan Kumar Ganguly +1
Recommender system has become an inseparable part of online shopping and its usability is increasing with the advancement of these e-commerce sites. An effective and efficient reco…
HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery
Michel Deudon, Alfredo Kalaitzis, Israel Goytom +7
Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Mul…