750 citations · 1.6k across the 35 of their papers we have counts for
35 papers
Token Alignment via Character Matching for Subword Completion
Ben Athiwaratkun, Shiqi Wang, Mingyue Shang +7
Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial to…
Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda +10
This study introduces bifurcated attention, a method designed to enhance language model inference in shared-context batch decoding scenarios. Our approach addresses the challenge o…
Code Representation Learning At Scale
Dejiao Zhang, Wasi Ahmad, Ming Tan +5
Recent studies have shown that code language models at scale demonstrate significant performance gains on downstream tasks, i.e., code generation. However, most of the existing wor…
CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad +8
Code completion models have made significant progress in recent years, yet current popular evaluation datasets, such as HumanEval and MBPP, predominantly focus on code completion t…
Exploring Continual Learning for Code Generation Models
Prateek Yadav, Qing Sun, Hantian Ding +9
Large-scale code generation models such as Codex and CodeT5 have achieved impressive performance. However, libraries are upgraded or deprecated very frequently and re-training larg…
ReCode: Robustness Evaluation of Code Generation Models
Shiqi Wang, Zheng Li, Haifeng Qian +11
Code generation models have achieved impressive performance. However, they tend to be brittle as slight edits to a prompt could lead to very different generations; these robustness…
CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad +5
While pre-trained language models (LM) for code have achieved great success in code completion, they generate code conditioned only on the contents within the file, i.e., in-file c…
Multi-lingual Evaluation of Code Generation Models
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang +22
We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages and are generated…
ContraCLM: Contrastive Learning For Causal Language Model
Nihal Jain, Dejiao Zhang, Wasi Uddin Ahmad +9
Despite exciting progress in causal language models, the expressiveness of the representations is largely limited due to poor discrimination ability. To remedy this issue, we prese…
Improving Text-to-SQL Semantic Parsing with Fine-grained Query Understanding
Jun Wang, Patrick Ng, Alexander Hanbo Li +5
Most recent research on Text-to-SQL semantic parsing relies on either parser itself or simple heuristic based approach to understand natural language query (NLQ). When synthesizing…
Efficient Few-Shot Fine-Tuning for Opinion Summarization
Arthur Bražinskas, Ramesh Nallapati, Mohit Bansal +1
Abstractive summarization models are typically pre-trained on large amounts of generic texts, then fine-tuned on tens or hundreds of thousands of annotated samples. However, in opi…
DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantization
Zheng Li, Zijian Wang, Ming Tan +5
Large-scale pre-trained sequence-to-sequence models like BART and T5 achieve state-of-the-art performance on many generative NLP tasks. However, such models pose a great challenge…
Pairwise Supervised Contrastive Learning of Sentence Representations
Dejiao Zhang, Shang-Wen Li, Wei Xiao +4
Many recent successes in sentence representation learning have been achieved by simply fine-tuning on the Natural Language Inference (NLI) datasets with triplet loss or siamese los…
Improving Factual Consistency of Abstractive Summarization via Question Answering
Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu +7
A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The f…
Transductive Learning for Abstractive News Summarization
Arthur Bražinskas, Mengwen Liu, Ramesh Nallapati +2
Pre-trained and fine-tuned news summarizers are expected to generalize to news articles unseen in the fine-tuning (training) phase. However, these articles often contain specifics,…
Supporting Clustering with Contrastive Learning
Dejiao Zhang, Feng Nan, Xiaokai Wei +6
Unsupervised clustering aims at discovering the semantic categories of data according to some distance measured in the representation space. However, different categories often ove…
Entity-level Factual Consistency of Abstractive Text Summarization
Feng Nan, Ramesh Nallapati, Zhiguo Wang +5
A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models t…
Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction
Yifan Gao, Henghui Zhu, Patrick Ng +7
In open-domain question answering, questions are highly likely to be ambiguous because users may not know the scope of relevant topics when formulating them. Therefore, a system ne…
End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems
Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu +5
We propose an end-to-end approach for synthetic QA data generation. Our model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate bo…
Beyond [CLS] through Ranking by Generation
Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati +2
Generative models for Information Retrieval, where ranking of documents is viewed as the task of generating a query from a document's language model, were very successful in variou…
Embedding-based Zero-shot Retrieval through Query Generation
Davis Liang, Peng Xu, Siamak Shakeri +4
Passage retrieval addresses the problem of locating relevant passages, usually from a large corpus, given a query. In practice, lexical term-matching algorithms like BM25 are popul…
Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering
Alexander R. Fabbri, Patrick Ng, Zhiguo Wang +2
Question Answering (QA) is in increasing demand as the amount of information available online and the desire for quick access to this content grows. A common approach to QA has bee…
Who did They Respond to? Conversation Structure Modeling using Masked Hierarchical Transformer
Henghui Zhu, Feng Nan, Zhiguo Wang +2
Conversation structure is useful for both understanding the nature of conversation dynamics and for providing features for many downstream applications such as summarization of con…
Universal Text Representation from BERT: An Empirical Study
Xiaofei Ma, Zhiguo Wang, Patrick Ng +2
We present a systematic investigation of layer-wise BERT activations for general-purpose text representations to understand what linguistic information they capture and how transfe…