11 papers
Epiplexity Guided Data Selection and Generation for Out-of-Distribution Generalization
Ellen Su, Andres Potapczynski, Shikai Qiu +2
Modern systems are increasingly expected to transfer across tasks not specified during training. What data facilitates generalization in these new, unanticipated settings? One hypo…
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Shikai Qiu, Marc Finzi, Yujia Zheng +2
The paper proposes requential coding, a method where a teacher model selects training samples from the student’s own distribution so that only disagreements need to be encoded, yie…
Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety
Ting Ma, Xiufeng Huang, Benlei Cui +43
As large language models are increasingly deployed in real-world systems, safety failures can still lead to harmful outputs and dangerous misuse. We argue that the essence of safet…
Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety
Shikai Qiu, Xiaowen Xu, Benlei Cui +55
General-purpose models often struggle to reliably identify and understand real-world multimodal risks, largely due to the inherent multimodal adversarial nature of content and AI s…
Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
Vatsal Baherwani, Zixi Chen, Shikai Qiu +2
Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learn…
Customizing the Inductive Biases of Softmax Attention using Structured Matrices
Yilun Kuang, Noah Amsel, Sanae Lotfi +3
The core component of attention is the scoring function, which transforms the inputs into low-dimensional queries and keys and takes the dot product of each pair. While the low-dim…