1 citations · 2 across the 14 of their papers we have counts for
16 papers
Rethinking Contextualization by Reinterpreting Attention Head Channels
Hakaze Cho, Haolin Yang, Zhun Sun +3
Contextualization, the core operation of language modeling, transmits information across words to build sentence-specific word representations. Prior works mainly study contextuali…
Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes
Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan +2
Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most…
Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding
Tao Jin, Phuong Minh Nguyen, Naoya Inoue
Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tre…
Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
Haolin Yang, Hakaze Cho, Kaize Ding +1
Large Language Models (LLMs) can perform new tasks from in-context demonstrations, a phenomenon known as in-context learning (ICL). Recent work suggests that these demonstrations a…
Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis
Haolin Yang, Hakaze Cho, Naoya Inoue
We investigate the mechanistic underpinnings of in-context learning (ICL) in large language models by reconciling two dominant perspectives: the component-level analysis of attenti…
Mechanism of Task-oriented Information Removal in In-context Learning
Hakaze Cho, Haolin Yang, Gouki Minegishi +1
In-context Learning (ICL) is an emerging few-shot learning paradigm based on modern Language Models (LMs), yet its inner mechanism remains unclear. In this paper, we investigate th…
Binary Autoencoder for Mechanistic Interpretability of Large Language Models
Hakaze Cho, Haolin Yang, Yanshu Li +2
Existing works are dedicated to untangling atomized numerical components (features) from the hidden states of Large Language Models (LLMs). However, they typically rely on autoenco…
Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning
Haolin Yang, Hakaze Cho, Yiqiao Zhong +1
The unusual properties of in-context learning (ICL) have prompted investigations into the internal mechanisms of large language models. Prior work typically focuses on either speci…
Mechanistic Fine-tuning for In-context Learning
Hakaze Cho, Peng Luo, Mariko Kato +2
In-context Learning (ICL) utilizes structured demonstration-query inputs to induce few-shot learning on Language Models (LMs), which are not originally pre-trained on ICL-style dat…
Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations
Mariko Kato, Hakaze Cho, Yoshihiro Sakai +1
The performance of In-Context Learning (ICL) is highly sensitive to the selected demonstrations. Existing approaches to demonstration selection optimize different objectives, yield…
StaICC: Standardized Evaluation for Classification Task in In-context Learning
Hakaze Cho, Naoya Inoue
Classification tasks are widely investigated in the In-Context Learning (ICL) paradigm. However, current efforts are evaluated on disjoint benchmarks and settings, while their perf…
Phase Diagram of Vision Large Language Models Inference: A Perspective from Interaction across Image and Instruction
Houjing Wei, Yuting Shi, Naoya Inoue
Vision Large Language Models (VLLMs) usually take input as a concatenation of image token embeddings and text token embeddings and conduct causal modeling. However, their internal…
Revisiting In-context Learning Inference Circuit in Large Language Models
Hakaze Cho, Mariko Kato, Yoshihiro Sakai +1
In-context Learning (ICL) is an emerging few-shot learning paradigm on Language Models (LMs) with inner mechanisms un-explored. There are already existing works describing the inne…
Token-based Decision Criteria Are Suboptimal in In-context Learning
Hakaze Cho, Yoshihiro Sakai, Mariko Kato +3
In-Context Learning (ICL) typically utilizes classification criteria from output probabilities of manually selected label tokens. However, we argue that such token-based classifica…
Understanding Token Probability Encoding in Output Embeddings
Hakaze Cho, Yoshihiro Sakai, Kenshiro Tanaka +2
In this paper, we investigate the output token probability information in the output embedding of language models. We find an approximate common log-linear encoding of output token…
NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning
Yufeng Zhao, Yoshihiro Sakai, Naoya Inoue
In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned langua…