1.5k citations · 1.9k across the 37 of their papers we have counts for
37 papers
MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answering
Junbin Xiao, Jiajun Chen, Tianxiang Sun +2
Long streaming video QA remains challenging due to growing visual tokens and limited reasoning length of large language models (LLMs). KV-caching stores the Key-Value (KV) of the h…
AI Can Learn Scientific Taste
Jingqi Tong, Mingzhe Li, Hangcheng Li +20
Scientific discovery depends on expert judgement and foresight, which we call scientific taste: the ability to judge and propose research ideas with the potential for long-term sci…
DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
Zhe Xu, Jiasheng Ye, Xiaoran Liu +8
Recently, significant efforts have been devoted to enhancing the long-context capabilities of Large Language Models (LLMs), particularly in long-context reasoning. To facilitate th…
Unified Active Retrieval for Retrieval Augmented Generation
Qinyuan Cheng, Xiaonan Li, Shimin Li +7
In Retrieval-Augmented Generation (RAG), retrieval is not always helpful and applying it to every instruction is sub-optimal. Therefore, determining whether to retrieve is crucial…
Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models
Zhangyue Yin, Qiushi Sun, Qipeng Guo +9
Recent advancements in Chain-of-Thought prompting have facilitated significant breakthroughs for Large Language Models (LLMs) in complex reasoning tasks. Current research enhances…
In-Memory Learning: A Declarative Learning Framework for Large Language Models
Bo Wang, Tianxiang Sun, Hang Yan +3
The exploration of whether agents can align with their environment without relying on human-labeled data presents an intriguing research topic. Drawing inspiration from the alignme…
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
Jiasheng Ye, Peiju Liu, Tianxiang Sun +3
Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome…
Turn Waste into Worth: Rectifying Top- Router of MoE
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei +7
Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top- routing mechanism su…
Agent Alignment in Evolving Social Norms
Shimin Li, Tianxiang Sun, Qinyuan Cheng +1
Agents based on Large Language Models (LLMs) are increasingly permeating various domains of human production and life, highlighting the importance of aligning them with human value…
Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT
Zhengfu He, Xuyang Ge, Qiong Tang +3
Sparse dictionary learning has been a rapidly growing technique in mechanistic interpretability to attack superposition and extract more human-understandable features from model ac…
LLM can Achieve Self-Regulation via Hyperparameter Aware Generation
Siyin Wang, Shimin Li, Tianxiang Sun +6
In the realm of Large Language Models (LLMs), users commonly employ diverse decoding strategies and adjust hyperparameters to control the generated text. However, a critical questi…
AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Jun Zhan, Junqi Dai, Jiasheng Ye +13
We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images…
Can AI Assistants Know What They Don't Know?
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu +7
Recently, AI assistants based on large language models (LLMs) show surprising performance in many tasks, such as dialogue, solving math problems, writing code, and using tools. Alt…
DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
Xinghao Wang, Junliang He, Pengyu Wang +3
Contrastive-learning-based methods have dominated sentence representation learning. These methods regularize the representation space by pulling similar sentence representations cl…
LLatrieval: LLM-Verified Retrieval for Verifiable Generation
Xiaonan Li, Changtai Zhu, Linyang Li +3
Verifiable generation aims to let the large language model (LLM) generate text with supporting documents, which enables the user to flexibly verify the answer and makes the LLM's o…
Flames: Benchmarking Value Alignment of LLMs in Chinese
Kexin Huang, Xiangyang Liu, Qianyu Guo +9
The widespread adoption of large language models (LLMs) across various regions underscores the urgent need to evaluate their alignment with human values. Current benchmarks, howeve…
Evaluating Hallucinations in Chinese Large Language Models
Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang +8
In this paper, we establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. HalluQA c…
Secrets of RLHF in Large Language Models Part I: PPO
Rui Zheng, Shihan Dou, Songyang Gao +24
Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, h…
Improving Contrastive Learning of Sentence Embeddings from AI Feedback
Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun +2
Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings. However, the discrete nature of natural lan…
CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors
Peng Li, Tianxiang Sun, Qiong Tang +4
Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning ability on many NLP tasks. A common practice is to recast the task into a…
Origin Tracing and Detecting of LLMs
Linyang Li, Pengyu Wang, Ke Ren +2
The extraordinary performance of large language models (LLMs) heightens the importance of detecting whether the context is generated by an AI system. More importantly, while more a…
DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
Zhengfu He, Tianxiang Sun, Kuanning Wang +2
We present DiffusionBERT, a new generative masked language model based on discrete diffusion models. Diffusion models and many pre-trained language models have a shared training ob…
Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts
Xiangyang Liu, Tianxiang Sun, Xuanjing Huang +1
Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing pre-trained models (PTMs) that simply prepends a soft prompt to the input and only optimizes the promp…
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation
Tianxiang Sun, Junliang He, Xipeng Qiu +1
Automatic evaluation metrics are crucial to the development of generative systems. In recent years, pre-trained language model (PLM) based metrics, such as BERTScore, have been com…