Publications (38)
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Yihe Deng, Hritik Bansal, Fan Yin +3
We introduce OpenVLThinker, one of the first open-source large vision-language models (LVLMs) to exhibit sophisticated chain-of-thought reasoning, achieving notable performance gai…
Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation
Da Yin, Xiao Liu, Fan Yin +4
Instruction tuning has emerged to enhance the capabilities of large language models (LLMs) to comprehend instructions and generate appropriate responses. Existing methods either ma…
ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation
Fan Yin, Yao Li, Cho-Jui Hsieh +1
Adversarial Examples Detection (AED) is a crucial defense technique against adversarial attacks and has drawn increasing attention from the Natural Language Processing (NLP) commun…
Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation
Di Wu, Jia-Chen Gu, Fan Yin +2
Retrieval-augmented language models (RALMs) have shown strong performance and wide applicability in knowledge-intensive tasks. However, there are significant trustworthiness concer…
Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
Min Cai, Yuchen Zhang, Shichang Zhang +5
We propose SelfControl, an inference-time model control method utilizing gradients to control the behavior of large language models (LLMs) without explicit human annotations. Given…
Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning
Fan Yin, Jesse Vig, Philippe Laban +3
Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks. However, it remains unclear whether models truly un…
Comparisons of penalized least squares methods by simulations
Ke Zhang, Fan Yin, Shifeng Xiong
Penalized least squares methods are commonly used for simultaneous estimation and variable selection in high-dimensional linear models. In this paper we compare several prevailing…
On the Robustness of Language Encoders against Grammatical Errors
Fan Yin, Quanyu Long, Tao Meng +1
We conduct a thorough study to diagnose the behaviors of pre-trained language encoders (ELMo, BERT, and RoBERTa) when confronted with natural grammatical errors. Specifically, we c…
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Shenggui Li, Chao Wang, Yikai Zhu +14
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to…
Evaluating Human Alignment and Model Faithfulness of LLM Rationale
Mohsen Fayyaz, Fan Yin, Jiao Sun +1
We study how well large language models (LLMs) explain their generations through rationales -- a set of tokens extracted from the input text that reflect the decision-making proces…
Bayesian Analysis of Static Light Scattering Data for Globular Proteins
Fan Yin, Domarin Khago, Rachel W. Martin +1
Static light scattering is a popular physical chemistry technique that enables calculation of physical attributes such as the radius of gyration and the second virial coefficient f…
Contrastive Instruction Tuning
Tianyi Lorena Yan, Fei Wang, James Y. Huang +5
Instruction tuning has been used as a promising approach to improve the performance of large language models (LLMs) on unseen tasks. However, current LLMs exhibit limited robustnes…
Analysis of professional basketball field goal attempts via a Bayesian matrix clustering approach
Fan Yin, Guanyu Hu, Weining Shen
We propose a Bayesian nonparametric matrix clustering approach to analyze the latent heterogeneity structure in the shot selection data collected from professional basketball playe…
Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension
Fan Yin, Jayanth Srinivasa, Kai-Wei Chang
We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs), which serves as a crucial step in building trust between humans and…
On Prompt-Driven Safeguarding for Large Language Models
Chujie Zheng, Fan Yin, Hao Zhou +5
Prepending model inputs with safety prompts is a common practice for safeguarding large language models (LLMs) against queries with harmful intents. However, the underlying working…
Achieve Efficient Position-Heap-based Privacy-Preserving Substring-of-Keyword Query over Cloud
Fan Yin, Rongxing Lu, Yandong Zheng +3
The cloud computing technique, which was initially used to mitigate the explosive growth of data, has been required to take both data privacy and users' query functionality into co…
Kernel-based Approximate Bayesian Inference for Exponential Family Random Graph Models
Fan Yin, Carter T. Butts
Bayesian inference for exponential family random graph models (ERGMs) is a doubly-intractable problem because of the intractability of both the likelihood and posterior normalizing…
CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning
Hritik Bansal, Nishad Singhi, Yu Yang +3
Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data. However, previous studies have…
Red Teaming Language Model Detectors with Language Models
Zhouxing Shi, Yihan Wang, Fan Yin +3
The prevalence and strong capability of large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. To prevent the potentially decept…
Efficient Shapley Values Estimation by Amortization for Text Classification
Chenghao Yang, Fan Yin, He He +3
Despite the popularity of Shapley Values in explaining neural text classification models, computing them is prohibitive for large pretrained models due to a large number of model e…
Spatial Heterogeneity Can Lead to Substantial Local Variations in COVID-19 Timing and Severity
Loring J. Thomas, Peng Huang, Fan Yin +4
Standard epidemiological models for COVID-19 employ variants of compartment (SIR) models at local scales, implicitly assuming spatially uniform local mixing. Here, we examine the e…
Neural Visual Decoding via Cognitive guided Adaptive Blurring and Information Constrained Alignment
Fan Yin, Chuhang Zheng, Peiliang Gong +2
EEG-based visual decoding aims to establish a mapping between neural signals and visual semantics. However, it remains constrained by the dual challenges of severe information gran…
Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning
ByteDance Seed, :, Jiaze Chen +267
We introduce Seed1.5-Thinking, capable of reasoning through thinking before responding, resulting in improved performance on a wide range of benchmarks. Seed1.5-Thinking achieves 8…
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks
Po-Nien Kung, Fan Yin, Di Wu +2
Instruction tuning (IT) achieves impressive zero-shot generalization results by training large language models (LLMs) on a massive amount of diverse tasks with instructions. Howeve…
Glyce: Glyph-vectors for Chinese Character Representations
Yuxian Meng, Wei Wu, Fei Wang +8
It is intuitive that NLP tasks for logographic languages like Chinese should benefit from the use of the glyph information in those languages. However, due to the lack of rich pict…
Bayesian Nonparametric Estimation for Point Processes with Spatial Homogeneity: A Spatial Analysis of NBA Shot Locations
Fan Yin, Jieying Jiao, Guanyu Hu +1
Basketball shot location data provide valuable summary information regarding players to coaches, sports analysts, fans, statisticians, as well as players themselves. Represented by…
Selection of Exponential-Family Random Graph Models via Held-Out Predictive Evaluation (HOPE)
Fan Yin, Nolan Edward Phillips, Carter T. Butts
Statistical models for networks with complex dependencies pose particular challenges for model selection and evaluation. In particular, many well-established statistical tools for…
A room-temperature electrical-field-enhanced ultrafast switch in organic microcavity polariton condensates
Jianbo De, Xuekai Ma, Fan Yin +7
Integrated electro-optical switches are essential as one of the fundamental elements in the development of modern optoelectronics. As an architecture for photonic systems, exciton…
On the Sensitivity and Stability of Model Interpretations in NLP
Fan Yin, Zhouxing Shi, Cho-Jui Hsieh +1
Recent years have witnessed the emergence of a variety of post-hoc interpretations that aim to uncover how natural language processing (NLP) models make predictions. Despite the su…
Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation
Fan Yin, Zifeng Wang, I-Hung Hsu +9
Large language models (LLMs) have exhibited the ability to effectively utilize external tools to address user queries. However, their performance may be limited in complex, multi-t…
Enhancing Large Vision Language Models with Self-Training on Image Comprehension
Yihe Deng, Pan Lu, Fan Yin +6
Large vision language models (LVLMs) integrate large language models (LLMs) with pre-trained vision encoders, thereby activating the perception capability of the model to understan…
Scaling Latent Reasoning via Looped Language Models
Rui-Jie Zhu, Zixuan Wang, Kai Hua +30
Modern LLMs are trained to "think" primarily via explicit text generation, such as chain-of-thought (CoT), which defers reasoning to post-training and under-leverages pre-training…
BingoGuard: LLM Content Moderation Tools with Risk Levels
Fan Yin, Philippe Laban, Xiangyu Peng +7
Malicious content generated by large language models (LLMs) can pose varying degrees of harm. Although existing LLM-based moderators can detect harmful content, they struggle to as…
Finite Mixtures of ERGMs for Modeling Ensembles of Networks
Fan Yin, Weining Shen, Carter T. Butts
Ensembles of networks arise in many scientific fields, but there are few statistical tools for inferring their generative processes, particularly in the presence of both dyadic dep…
Entity-Relation Extraction as Multi-Turn Question Answering
Xiaoya Li, Fan Yin, Zijun Sun +6
In this paper, we propose a new paradigm for the task of entity-relation extraction. We cast the task as a multi-turn question answering problem, i.e., the extraction of entities a…
Highly Scalable Maximum Likelihood and Conjugate Bayesian Inference for ERGMs on Graph Sets with Equivalent Vertices
Fan Yin, Carter T. Butts
The exponential family random graph modeling (ERGM) framework provides a flexible approach for the statistical analysis of networks. As ERGMs typically involve normalizing factors…
Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space
Xingwei Qu, Shaowen Wang, Zihao Huang +16
Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density. This token-uniform regime wastes capacity…
Should I Stop or Should I Go: Early Stopping with Heterogeneous Populations
Hammaad Adam, Fan Yin, Huibin +5
Randomized experiments often need to be stopped prematurely due to the treatment having an unintended harmful effect. Existing methods that determine when to stop an experiment ear…