Publications (31)
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models
Chi Han, Qifan Wang, Hao Peng +4
Today's large language models (LLMs) typically train on short text segments (e.g., <4K tokens) due to the quadratic complexity of their Transformer architectures. As a result, thei…
InfoPattern: Unveiling Information Propagation Patterns in Social Media
Chi Han, Jialiang Xu, Manling Li +3
Social media play a significant role in shaping public opinion and influencing ideological communities through information propagation. Our demo InfoPattern centers on the interpla…
Eliminating Position Bias of Language Models: A Mechanistic Approach
Ziqi Wang, Hanlin Zhang, Xiner Li +6
Position bias has proven to be a prevalent issue of modern language models (LMs), where the models prioritize content based on its position within the given context. This bias ofte…
LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Rui Pan, Xiang Liu, Shizhe Diao +4
The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significa…
Neural Synchrony Between Socially Interacting Language Models
Zhining Zhang, Wentao Zhu, Chi Han +2
Neuroscience has uncovered a fundamental mechanism of our social nature: human brain activity becomes synchronized with others in many social contexts involving interaction. Tradit…
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian, Chi Han, Yi R. Fung +3
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particul…
Visual Concept-Metaconcept Learning
Chi Han, Jiayuan Mao, Chuang Gan +2
Humans reason with concepts and metaconcepts: we recognize red and green from visual input; we also understand that they describe the same property of objects (i.e., the color). In…
Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
Anna Hart, Chi Han, Jeonghwan Kim +2
Modern Protein Language Models (PLMs) apply transformer-based model architectures from natural language processing to biological sequences, predicting a variety of protein function…
SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering
Xuehang Guo, Xingyao Wang, Yangyi Chen +4
Software engineering (SE) is increasingly collaborative, with developers working together on shared complex codebases. Effective collaboration in shared environments requires parti…
LDC: Learning to Generate Research Idea with Dynamic Control
Ruochen Li, Liqiang Jing, Chi Han +2
Recent advancements in large language models (LLMs) have demonstrated their potential in automating the scientific research ideation. Existing approaches primarily focus on prompti…
The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination
Yuji Zhang, Sha Li, Cheng Qian +8
Hallucination is a persistent challenge in large language models (LLMs), where even with rigorous quality control, models often generate distorted facts. This paradox, in which err…
Understanding the Effect of Data Augmentation on Knowledge Distillation
Ziqi Wang, Chi Han, Wenxuan Bao +1
Knowledge distillation (KD) requires sufficient data to transfer knowledge from large-scale teacher models to small-scale student models. Therefore, data augmentation has been wide…
Defining a New NLP Playground
Sha Li, Chi Han, Pengfei Yu +8
The recent explosion of performance of large language models (LLMs) has changed the field of Natural Language Processing (NLP) more abruptly and seismically than any other shift in…
Logical Entity Representation in Knowledge-Graphs for Differentiable Rule Learning
Chi Han, Qizheng He, Charles Yu +3
Probabilistic logical rule learning has shown great strength in logical rule mining and knowledge graph completion. It learns logical rules to predict missing edges by reasoning on…
Can Language Models Follow Multiple Turns of Entangled Instructions?
Chi Han, Xin Liu, Haodong Wang +12
Despite significant achievements in improving the instruction-following capabilities of large language models (LLMs), the ability to process multiple potentially entangled or confl…
Zero-Shot Classification by Logical Reasoning on Natural Language Explanations
Chi Han, Hengzhi Pei, Xinya Du +1
Humans can classify data of an unseen category by reasoning on its language explanations. This ability is owing to the compositional nature of language: we can combine previously s…
Learning Shared Semantic Space for Speech-to-Text Translation
Chi Han, Mingxuan Wang, Heng Ji +1
Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the ra…
Understanding Emergent In-Context Learning from a Kernel Regression Perspective
Chi Han, Ziqi Wang, Han Zhao +1
Large language models (LLMs) have initiated a paradigm shift in transfer learning. In contrast to the classic pretraining-then-finetuning procedure, in order to use LLMs for downst…
Birds of a Feather Flock Together: A Close Look at Cooperation Emergence via Multi-Agent RL
Heng Dong, Tonghan Wang, Jiayuan Liu +2
How cooperation emerges is a long-standing and interdisciplinary problem. Game-theoretical studies on social dilemmas reveal that altruistic incentives are critical to the emergenc…
Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
Yifan Hao, Chenlu Ye, Chi Han +1
Transformer based models have shown remarkable capabilities in sequence learning across a wide range of tasks, often performing well on specific task by leveraging input-output exa…
MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders
Cheng Li, May Fung, Qingyun Wang +4
Mental health disorders are one of the most serious diseases in the world. Most people with such a disease lack access to adequate care, which highlights the importance of training…
Subhalos in Galaxy Clusters: Coherent Accretion and Internal Orbits
Chi Han, Kuan Wang, Camille Avestruz +1
Subhalo dynamics in galaxy cluster host halos govern the observed distribution and properties of cluster member galaxies. We use the IllustrisTNG simulation to investigate the accr…
Prospects for using drones to test formation-flying CubeSat concepts, and other astronomical applications
John D. Monnier, Prachet Jain, Mayra Gutierrez +11
Drones provide a versatile platform for remote sensing and atmospheric studies. However, strict payload mass limits and intense vibrations have proven obstacles to adoption for ast…
Word Embeddings Are Steers for Language Models
Chi Han, Jialiang Xu, Manling Li +5
Language models (LMs) automatically learn word embeddings during pre-training on language corpora. Although word embeddings are usually interpreted as feature vectors for individua…
mCLM: A Modular Chemical Language Model that Generates Functional and Makeable Molecules
Carl Edwards, Chi Han, Gawon Lee +11
Despite their ability to understand chemical knowledge, large language models (LLMs) remain limited in their capacity to propose novel molecules with desired functions (e.g., drug-…
RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought
Tianci Xue, Ziqi Wang, Zhenhailong Wang +3
Large language Models (LLMs) have achieved promising performance on arithmetic reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting. However, LLMs face ch…
Tool Learning with Foundation Models
Yujia Qin, Shengding Hu, Yankai Lin +38
Humans possess an extraordinary ability to create and utilize tools, allowing them to overcome physical limitations and explore new frontiers. With the advent of foundation models,…
Large Language Models on Graphs: A Comprehensive Survey
Bowen Jin, Gang Liu, Chi Han +3
Large language models (LLMs), such as GPT4 and LLaMA, are creating significant advancements in natural language processing, due to their strong text encoding/decoding ability and n…
Computation Mechanism Behind LLM Position Generalization
Chi Han, Heng Ji
Most written natural languages are composed of sequences of words and sentences. Similar to humans, large language models (LLMs) exhibit flexibility in handling textual positions -…
Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play
Sha Li, Revanth Gangi Reddy, Khanh Duy Nguyen +7
Complex news events, such as natural disasters and socio-political conflicts, require swift responses from the government and society. Relying on historical events to project the f…
Towards LifeSpan Cognitive Systems
Yu Wang, Chi Han, Tongtong Wu +9
Building a human-like system that continuously interacts with complex environments -- whether simulated digital worlds or human society -- presents several key challenges. Central…