Understanding and Improving Noisy Embedding Techniques in Instruction Finetuning

Abhay Yadav

Recent advancements in instructional fine-tuning have injected noise into embeddings, with NEFTune (Jain et al., 2024) setting benchmarks using uniform noise. Despite NEFTune's emp…