Memorization Sinks: Isolating Memorization during LLM Training

Gaurav R. Ghosal, Pratyush Maini, Aditi Raghunathan

Large language models are susceptible to memorizing repeated sequences, posing privacy and copyright concerns. A popular mitigation strategy is to remove memorized information from…