9 papers
On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Models
Shunsuke Kando, Wataru Nakata, Shinnosuke Takamichi +1
Generative Spoken Language Modeling (GSLM) enables text-free speech modeling by training language models (LMs) using discrete speech representations instead of textual transcriptio…
Do Self-Supervised Speech Models Exhibit the Critical Period Effects in Language Acquisition?
Yurie Koga, Shunsuke Kando, Yusuke Miyao
This paper investigates whether the Critical Period (CP) effects in human language acquisition are observed in self-supervised speech models (S3Ms). CP effects refer to greater dif…
Tracking World States with Language Models: State-Based Evaluation Using Chess
Romain Harang, Jason Naradowsky, Yaswitha Gujju +1
Large Language Models (LLMs) exhibit emergent capabilities in structured domains, suggesting they may implicitly internalize high-fidelity representations of world models. While pr…
Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
Shunsuke Kando, Yusuke Miyao, Shinnosuke Takamichi
The purpose of speech tokenization is to transform a speech signal into a sequence of discrete representations, serving as the foundation for speech language models (SLMs). While s…
How Much Do Large Language Models Know about Human Motion? A Case Study in 3D Avatar Control
Kunhang Li, Jason Naradowsky, Yansong Feng +1
We explore the human motion knowledge of Large Language Models (LLMs) through 3D avatar control. Given a motion instruction, we prompt LLMs to first generate a high-level movement…
Does it Chug? Towards a Data-Driven Understanding of Guitar Tone Description
Pratik Sutar, Jason Naradowsky, Yusuke Miyao
Natural language is commonly used to describe instrument timbre, such as a "warm" or "heavy" sound. As these descriptors are based on human perception, there can be disagreement ov…
Self-Emotion Blended Dialogue Generation in Social Simulation Agents
Qiang Zhang, Jason Naradowsky, Yusuke Miyao
When engaging in conversations, dialogue agents in a virtual simulation environment may exhibit their own emotional states that are unrelated to the immediate conversational contex…
Textless Dependency Parsing by Labeled Sequence Prediction
Shunsuke Kando, Yusuke Miyao, Jason Naradowsky +1
Traditional spoken language processing involves cascading an automatic speech recognition (ASR) system into text processing models. In contrast, "textless" methods process speech r…
Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation
Qiang Zhang, Jason Naradowsky, Yusuke Miyao
Knowing how to end and resume conversations over time is a natural part of communication, allowing for discussions to span weeks, months, or years. The duration of gaps between con…