5 papers
CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models
Jeffrey G. Wang, Jason Wang, Marvin Li +1
Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on langua…
ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models
Jason Z Wang
At matched accuracy, open-weight LLMs differ substantially in the shape of their error severity distribution -- a difference invisible to the scalar error rate. Hallucination bench…
The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models
Jason Z Wang
We give a stereological theory of LLM benchmark coverage. For any suite with effective dimensionality d_eff, the visible Hausdorff distance between two convex capability profiles c…
The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime
Jason Z Wang
The most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 (Guo et al., 2017) -- is below the statistical noise floor. We prove this i…
Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM
YuanLab. ai, :, Shawn Wu +25
We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enha…