26 citations · 212 across the 74 of their papers we have counts for
75 papers
A Scaled-Up Empirical Study of Syntactic Language Models
Pengcheng Wang, Zhexuan Zhang, Chao Lou +1
Syntactic language models (SLMs) that generate sentences along with their syntactic tree structures have shown promise in both syntactic generalization and downstream tasks. Prior…
Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars
Zongru Liu, Pengyu Ji, Pengcheng Wang +1
Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic tr…
GiLT: Augmenting Transformer Language Models with Dependency Graphs
Tianyu Huang, Yida Zhao, Chuyan Zhou +1
Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on synt…
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Penghao Kuang, Haoyi Wu, Kewei Tu
Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computa…
Parallel Continuous Chain-of-Thought with Jacobi Iteration
Haoyi Wu, Zhihao Teng, Kewei Tu
Continuous chain-of-thought has been shown to be effective in saving reasoning tokens for large language models. By reasoning with continuous latent thought tokens, continuous CoT…
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
Zhuo Chen, Xinyu Wang, Yong Jiang +5
Despite the advancements made in Vision Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time…
A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference
You Wu, Haoyi Wu, Kewei Tu
Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techni…
Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation
Chaoyi Ai, Yong Jiang, Shen Huang +2
Named entity recognition (NER) models often struggle with noisy inputs, such as those with spelling mistakes or errors generated by Optical Character Recognition processes, and lea…
Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models
Yida Zhao, Chao Lou, Kewei Tu
Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding…
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers
Chao Lou, Zixia Jia, Zilong Zheng +1
Accommodating long sequences efficiently in autoregressive Transformers, especially within an extended context window, poses significant challenges due to the quadratic computation…
Layer-Condensed KV Cache for Efficient Inference of Large Language Models
Haoyi Wu, Kewei Tu
Huge memory consumption has been a major bottleneck for deploying high-throughput large language models in real-world applications. In addition to the large number of parameters, t…
Unsupervised Morphological Tree Tokenizer
Qingyang Zhu, Xiang Hu, Pengyu Ji +2
As a cornerstone in language modeling, tokenization involves segmenting text inputs into pre-defined atomic units. Conventional statistical tokenizers often disrupt constituent bou…
Potential and Limitations of LLMs in Capturing Structured Semantics: A Case Study on SRL
Ning Cheng, Zhaohui Yan, Ziming Wang +6
Large Language Models (LLMs) play a crucial role in capturing structured semantics to enhance language understanding, improve interpretability, and reduce bias. Nevertheless, an on…
Using Interpretation Methods for Model Enhancement
Zhuo Chen, Chengyue Jiang, Kewei Tu
In the age of neural natural language processing, there are plenty of works trying to derive interpretations of neural models. Intuitively, when gold rationales exist during traini…
RoT: Enhancing Large Language Models with Reflection on Search Trees
Wenyang Hui, Kewei Tu
Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these method…
Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts
Zhuo Chen, Xinyu Wang, Yong Jiang +3
In the era of large language models, applying techniques such as Retrieval Augmented Generation can better address Open-Domain Question-Answering problems. Due to constraints inclu…
Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
Xiang Hu, Pengyu Ji, Qingyang Zhu +2
A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPS…
Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation
Haoyi Wu, Kewei Tu
Syntactic structures used to play a vital role in natural language processing (NLP), but since the deep learning revolution, NLP has been gradually dominated by neural models that…
Conic10K: A Challenging Math Problem Understanding and Reasoning Dataset
Haoyi Wu, Wenyang Hui, Yezeng Chen +3
Mathematical understanding and reasoning are crucial tasks for assessing the capabilities of artificial intelligence (AI). However, existing benchmarks either require just a few st…
Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
Zhaohui Yan, Songlin Yang, Wei Liu +1
Entity and Relation Extraction (ERE) is an important task in information extraction. Recent marker-based pipeline models achieve state-of-the-art performance, but still suffer from…
Simple Hardware-Efficient PCFGs with Independent Left and Right Productions
Wei Liu, Songlin Yang, Yoon Kim +1
Scaling dense PCFGs to thousands of nonterminals via a low-rank parameterization of the rule probability tensor has been shown to be beneficial for unsupervised parsing. However, P…
AMR Parsing with Causal Hierarchical Attention and Pointers
Chao Lou, Kewei Tu
Translation-based AMR parsers have recently gained popularity due to their simplicity and effectiveness. They predict linearized graphs as free texts, avoiding explicit structure m…
Do PLMs Know and Understand Ontological Knowledge?
Weiqi Wu, Chengyue Jiang, Yong Jiang +2
Ontological knowledge, which comprises classes and properties and their relationships, is integral to world knowledge. It is significant to explore whether Pretrained Language Mode…
Augmenting Transformers with Recursively Composed Multi-grained Representations
Xiang Hu, Qingyang Zhu, Kewei Tu +1
We present ReCAT, a recursive composition augmented Transformer that is able to explicitly model hierarchical syntactic structures of raw texts without relying on gold trees during…