18 citations · 51 across the 23 of their papers we have counts for
24 papers
Timesteps of Mamba Align with Human Reading Times
Yuji Yamamoto, Shinnosuke Isono, Yoshinobu Kawahara +1
This study demonstrates an alignment of per-word processing time in a popular state-space language model Mamba and human readers. In Mamba, the recurrent state transition at each l…
Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings
Tomomasa Hara, Hiroto Kurita, Masaaki Imaizumi +2
For constructing text embeddings, mean pooling, which averages token embeddings, is the standard approach. This paper examines whether mean pooling actually works well in real mode…
Linear Representations of Hierarchical Concepts in Language Models
Masaki Sakata, Benjamin Heinzerling, Takumi Ito +2
We investigate how and to what extent hierarchical relations (e.g., Japan Eastern Asia Asia) are encoded in the internal representations of language models. Bui…
On Entity Identification in Language Models
Masaki Sakata, Benjamin Heinzerling, Sho Yokoi +2
We analyze the extent to which internal representations of language models (LMs) identify and distinguish mentions of named entities, focusing on the many-to-many correspondence be…
Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words
Hiroto Kurita, Goro Kobayashi, Sho Yokoi +1
The performance of sentence encoders can be significantly improved through the simple practice of fine-tuning using contrastive loss. A natural question arises: what characteristic…
Unbalanced Optimal Transport for Unbalanced Word Alignment
Yuki Arase, Han Bao, Sho Yokoi
Monolingual word alignment is crucial to model semantic interactions between sentences. In particular, null alignment, a phenomenon in which words have no corresponding counterpart…
Transformer Language Models Handle Word Frequency in Prediction Head
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1
Prediction head is a crucial component of Transformer language models. Despite its direct impact on prediction, this component has often been overlooked in analyzing Transformers.…
Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1
Transformers are ubiquitous in wide tasks. Interpreting their internals is a pivotal goal. Nevertheless, their particular components, feed-forward (FF) blocks, have typically been…
Norm of Word Embedding Encodes Information Gain
Momose Oyama, Sho Yokoi, Hidetoshi Shimodaira
Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method,…
Improving word mover's distance by leveraging self-attention matrix
Hiroaki Yamagiwa, Sho Yokoi, Hidetoshi Shimodaira
Measuring the semantic similarity between two sentences is still an important task. The word mover's distance (WMD) computes the similarity via the optimal alignment between the se…
Subspace Representations for Soft Set Operations and Sentence Similarities
Yoichi Ishibashi, Sho Yokoi, Katsuhito Sudoh +1
In the field of natural language processing (NLP), continuous vector representations are crucial for capturing the semantic meanings of individual words. Yet, when it comes to the…
Instance-Based Neural Dependency Parsing
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi +4
Interpretable rationales for model predictions are crucial in practical applications. We develop neural models that possess an interpretable inference process for dependency parsin…
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1
Transformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively a…
Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings
Masahiro Naito, Sho Yokoi, Geewook Kim +1
It is well-known that typical word embedding methods such as Word2Vec and GloVe have the property that the meaning can be composed by adding up the embeddings (additive composition…
Computationally Efficient Wasserstein Loss for Structured Labels
Ayato Toyokuni, Sho Yokoi, Hisashi Kashima +1
The problem of estimating the probability distribution of labels has been widely studied as a label distribution learning (LDL) problem, whose applications include age estimation,…
Efficient Estimation of Influence of a Training Instance
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki +1
Understanding the influence of a training instance on a neural network model leads to improving interpretability. However, it is difficult and inefficient to evaluate the influence…
Modeling Event Salience in Narratives via Barthes' Cardinal Functions
Takaki Otake, Sho Yokoi, Naoya Inoue +3
Events in a narrative differ in salience: some are more important to the story than others. Estimating event salience is useful for tasks such as story generation, and as a tool fo…
Evaluation of Similarity-based Explanations
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara +1
Explaining the predictions made by complex machine learning models helps users to understand and accept the predicted outputs with confidence. One promising way is to use similarit…
Instance-Based Learning of Span Representations: A Case Study through Named Entity Recognition
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi +4
Interpretable rationales for model predictions play a critical role in practical applications. In this study, we develop models possessing interpretable inference process for struc…
Word Rotator's Distance
Sho Yokoi, Ryo Takahashi, Reina Akama +2
A key principle in assessing textual similarity is measuring the degree of semantic overlap between two texts by considering the word alignment. Such alignment-based approaches are…
Filtering Noisy Dialogue Corpora by Connectivity and Content Relatedness
Reina Akama, Sho Yokoi, Jun Suzuki +1
Large-scale dialogue datasets have recently become available for training neural dialogue agents. However, these datasets have been reported to contain a non-negligible number of u…
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi +1
Attention is a key component of Transformers, which have recently achieved considerable success in natural language processing. Hence, attention is being extensively studied to inv…
Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm for Sparse Linguistic Expressions
Sho Yokoi, Sosuke Kobayashi, Kenji Fukumizu +2
In this paper, we propose a new kernel-based co-occurrence measure that can be applied to sparse linguistic expressions (e.g., sentences) with a very short learning time, as an alt…
Unsupervised Learning of Style-sensitive Word Vectors
Reina Akama, Kento Watanabe, Sho Yokoi +2
This paper presents the first study aimed at capturing stylistic similarity between words in an unsupervised manner. We propose extending the continuous bag of words (CBOW) model (…