6 papers
Insights from the ICLR Peer Review and Rebuttal Process
Amir Hossein Kargaran, Nafiseh Nikeghbal, Jing Yang +1
Peer review is a cornerstone of scientific publishing, including at premier machine learning conferences such as ICLR. As submission volumes increase, understanding the nature and…
CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs
Nafiseh Nikeghbal, Amir Hossein Kargaran, Jana Diesner
Improvements in model construction, including fortified safety guardrails, allow Large language models (LLMs) to increasingly pass standard safety checks. However, LLMs sometimes s…
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec +7
Pre-training state-of-the-art large language models (LLMs) requires vast amounts of clean and diverse text data. While the open development of large high-quality English pre-traini…
How Programming Concepts and Neurons Are Shared in Code Language Models
Amir Hossein Kargaran, Yihong Liu, François Yvon +1
Several studies have explored the mechanisms of large language models (LLMs) in coding tasks, but most have focused on programming languages (PLs) in a monolingual setting. In this…
Tracing Multilingual Factual Knowledge Acquisition in Pretraining
Yihong Liu, Mingyang Wang, Amir Hossein Kargaran +5
Large Language Models (LLMs) are capable of recalling multilingual factual knowledge present in their pretraining data. However, most studies evaluate only the final model, leaving…
On Relation-Specific Neurons in Large Language Models
Yihong Liu, Runsheng Chen, Lea Hirlimann +6
In large language models (LLMs), certain \emph{neurons} can store distinct pieces of knowledge learned during pretraining. While factual knowledge typically appears as a combinatio…