4 papers
When Correct Demonstrations Hurt: Rethinking the Role of Exemplars in In-Context Learning
Chenghao Qiu, Chunli Peng, Yufeng Yang +2
In-context learning (ICL) is often motivated by the intuition that demonstrations help because they provide correct input-output examples. However, we reveal a counterintuitive phe…
Attention Sinks and Outliers in Attention Residuals
Haozheng Luo, Haoran Dai, Shaoyang Zhang +10
We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel,…
GenoArmory: A Unified Evaluation Framework for Adversarial Attacks on Genomic Foundation Models
Haozheng Luo, Chenghao Qiu, Yimin Wang +9
We propose the first unified adversarial attack benchmark for Genomic Foundation Models (GFMs), named GenoArmory. Unlike existing GFM benchmarks, GenoArmory offers the first compre…
Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Haozheng Luo, Chenghao Qiu, Maojiang Su +5
To address the challenge of scarce computational resources in genomic modeling, we introduce GERM, a genomic foundation model with strong compression performance and fast adaptabil…