8 papers
Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars
Zongru Liu, Pengyu Ji, Pengcheng Wang +1
Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic tr…
GiLT: Augmenting Transformer Language Models with Dependency Graphs
Tianyu Huang, Yida Zhao, Chuyan Zhou +1
Augmenting Transformers with linguistic structures effectively enhances the syntactic generalization performance of language models. Previous work in this direction focuses on synt…
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
Penghao Kuang, Haoyi Wu, Kewei Tu
Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computa…
Parallel Continuous Chain-of-Thought with Jacobi Iteration
Haoyi Wu, Zhihao Teng, Kewei Tu
Continuous chain-of-thought has been shown to be effective in saving reasoning tokens for large language models. By reasoning with continuous latent thought tokens, continuous CoT…
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
Zhuo Chen, Xinyu Wang, Yong Jiang +5
Despite the advancements made in Vision Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time…
Unsupervised Morphological Tree Tokenizer
Qingyang Zhu, Xiang Hu, Pengyu Ji +2
As a cornerstone in language modeling, tokenization involves segmenting text inputs into pre-defined atomic units. Conventional statistical tokenizers often disrupt constituent bou…