1.6k citations · 1.9k across the 32 of their papers we have counts for
16 papers · 1 filter
Textbooks Are All You Need II: phi-1.5 technical report
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan +3
We continue the investigation into the power of smaller Transformer-based language models as initiated by \textbf{TinyStories} -- a 10 million parameter model that can produce cohe…
Generating Faithful Text From a Knowledge Graph with Noisy Reference Text
Tahsina Hashem, Weiqing Wang, Derry Tanti Wijaya +2
Knowledge Graph (KG)-to-Text generation aims at generating fluent natural-language text that accurately represents the information of a given knowledge graph. While significant pro…
Textbooks Are All You Need
Suriya Gunasekar, Yi Zhang, Jyoti Aneja +16
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for…
NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery
Farhad Moghimifar, Shilin Qu, Tongtong Wu +2
Norms, which are culturally accepted guidelines for behaviours, can be integrated into conversational models to generate utterances that are appropriate for the socio-cultural cont…
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Ronen Eldan, Yuanzhi Li
Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M p…
Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
Yue Wu, So Yeon Min, Yonatan Bisk +5
Pre-trained large language models (LLMs) capture procedural knowledge about the world. Recent work has leveraged LLM's ability to generate abstract plans to simplify challenging co…
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks,…
What Matters In The Structured Pruning of Generative Language Models?
Michael Santacroce, Zixin Wen, Yelong Shen +1
Auto-regressive large language models such as GPT-3 require enormous computational resources to use. Traditionally, structured pruning methods are employed to reduce resource usage…
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex
Terry Yue Zhuo, Zhuang Li, Yujin Huang +4
Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advancements in few-shot language models tra…
Complex Reading Comprehension Through Question Decomposition
Xiao-Yu Guo, Yuan-Fang Li, Gholamreza Haffari
Multi-hop reading comprehension requires not only the ability to reason over raw text but also the ability to combine multiple evidence. We propose a novel learning approach that h…
Towards Relation Extraction From Speech
Tongtong Wu, Guitao Wang, Jinming Zhao +4
Relation extraction typically aims to extract semantic relationships between entities from the unstructured text. One of the most essential data sources for relation extraction is…
Teaching Neural Module Networks to Do Arithmetic
Jiayi Chen, Xiao-Yu Guo, Yuan-Fang Li +1
Answering complex questions that require multi-step multi-type reasoning over raw text is challenging, especially when conducting numerical reasoning. Neural Module Networks(NMNs),…
Neural Topic Modeling with Deep Mutual Information Estimation
Kang Xu, Xiaoqiu Lu, Yuan-fang Li +5
The emerging neural topic models make topic modeling more easily adaptable and extendable in unsupervised text mining. However, the existing neural topic models is difficult to ret…
CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad +3
We present CrossSum, a large-scale cross-lingual summarization dataset comprising 1.68 million article-summary samples in 1,500+ language pairs. We create CrossSum by aligning para…
Less is More: Data-Efficient Complex Question Answering over Knowledge Bases
Yuncheng Hua, Yuan-Fang Li, Guilin Qi +3
Question answering is an effective method for obtaining information from knowledge bases (KB). In this paper, we propose the Neural-Symbolic Complex Question Answering (NS-CQA) mod…
Few-Shot Complex Knowledge Base Question Answering via Meta Reinforcement Learning
Yuncheng Hua, Yuan-Fang Li, Gholamreza Haffari +2
Complex question-answering (CQA) involves answering complex natural-language questions on a knowledge base (KB). However, the conventional neural program induction (NPI) approach e…