7 papers
Just-in-Time Catching Test Generation at Meta
Matthew Becker, Yifei Chen, Nicholas Cochran +12
We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hard…
OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
Liujianfu Wang, Yuyang Du, Yuchen Pan +3
Mixture-of-Experts (MoE), while offering significant advantages as a Large Language Model (LLM) architecture, faces substantial challenges when deployed on low-cost edge devices wi…
Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data
Jiacheng Liu, Mayi Xu, Qiankun Pi +5
Large Language Models (LLMs) are increasingly employed in applications that require processing information from heterogeneous formats, including texts, tables, infoboxes, and knowl…
DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
Tianteng Gu, Bei Liu, Bo Xiao +3
Pruning is a widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation - especia…
ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
Tong Chen, Faeze Brahman, Jiacheng Liu +5
Language models (LMs) can memorize and reproduce segments from their pretraining data verbatim even in non-adversarial settings, raising concerns about copyright, plagiarism, priva…
DataDecide: How to Predict Best Pretraining Data with Small Experiments
Ian Magnusson, Nguyen Tai, Ben Bogin +10
Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and…