5 citations · 22 across the 17 of their papers we have counts for
11 papers · 1 filter
dMel: Speech Tokenization made Simple
Richard He Bai, Tatiana Likhomanenko, Ruixiang Zhang +3
Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data. Inspired by this success, researchers have inv…
Divide-or-Conquer? Which Part Should You Distill Your LLM?
Zhuofeng Wu, He Bai, Aonan Zhang +4
Recent methods have demonstrated that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first. In this paper…
Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Pratyush Maini, Skyler Seto, He Bai +3
Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data req…
Construction of Paired Knowledge Graph-Text Datasets Informed by Cyclic Evaluation
Ali Mousavi, Xin Zhan, He Bai +9
Datasets that pair Knowledge Graphs (KG) and text together (KG-T) can be used to train forward and reverse neural models that generate text from KG and vice versa. However models t…
XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing
Peng Shi, Rui Zhang, He Bai +1
In-context learning using large language models has recently shown surprising results for semantic parsing tasks such as Text-to-SQL translation. Prompting GPT-3 or Codex using sev…
Better Language Model with Hypernym Class Prediction
He Bai, Tong Wang, Alessandro Sordoni +1
Class-based language models (LMs) have been long devised to address context sparsity in -gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypot…
Cross-Lingual Training with Dense Retrieval for Document Retrieval
Peng Shi, Rui Zhang, He Bai +1
Dense retrieval has shown great success in passage ranking in English. However, its effectiveness in document retrieval for non-English languages remains unexplored due to the limi…
Segatron: Segment-Aware Transformer for Language Modeling and Understanding
He Bai, Peng Shi, Jimmy Lin +5
Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it…
Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2
He Bai, Peng Shi, Jimmy Lin +5
The semantics of a text is manifested not only by what is read, but also by what is not read. In this article, we will study how the implicit "not read" information such as end-of-…
Memory Consolidation for Contextual Spoken Language Understanding with Dialogue Logistic Inference
He Bai, Yu Zhou, Jiajun Zhang +1
Dialogue contexts are proven helpful in the spoken language understanding (SLU) system and they are typically encoded with explicit memory representations. However, most of the pre…
Source-Critical Reinforcement Learning for Transferring Spoken Language Understanding to a New Language
He Bai, Yu Zhou, Jiajun Zhang +3
To deploy a spoken language understanding (SLU) model to a new language, language transferring is desired to avoid the trouble of acquiring and labeling a new big SLU corpus. Trans…