111 citations · 391 across the 44 of their papers we have counts for
46 papers
Tanawpamana: Benchmarking Vision-Language Models on Philippine Cultural Heritage Sites
Ivan Yuri De Leon, Hiroki Ouchi, Sakriani Sakti
Vision-language models are increasingly used to identify and describe places, yet their recognition and knowledge of specific cultures remain poorly understood, particularly in und…
Beyond One-Size-Fits-All: Personalized and Culturally Adaptive Emotional TTS via Interactive Optimization of Individual Emotion Perception Spaces
Wangzixi Zhou, Bagus Tris Atmaja, Sakriani Sakti
The rise of conversational AI has increased interest in emotional Text-to-Speech (TTS). Most systems rely on discrete emotion labels, which fail to capture the nuanced nature of hu…
WaveNeXt 2: ConvNeXt-Based Fast Neural Vocoders With Residual Denoising and Sub-Modeling for GAN and Diffusion Models
Wangzixi Zhou, Takuma Okamoto, Yamato Ohtani +2
Most neural vocoders are limited to one type: either GAN or diffusion-based. While state-of-the-art models like Vocos and WaveNeXt use powerful ConvNeXt-based generators, they have…
Toward Natural Emotional Text-To-Speech System with Fine-Grained Non-Verbal Expression Control
Wangzixi Zhou, Bagus Tris Atmaja, Sakriani Sakti
While current emotional Text-to-Speech (TTS) models have successfully controlled verbal prosody, they often ignore non-verbal vocalizations (NVs), which are essential for authentic…
SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translation
Haotian Tan, Hiroki Ouchi, Sakriani Sakti
How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems formulate SimulST as a multi-turn di…
SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition
Yuta Hirano, Sakriani Sakti
We propose Speaker-Conditioned Serialized Output Training (SC-SOT), an enhanced SOT-based training for E2E multi-talker ASR. We first probe how SOT handles overlapped speech, and w…
Indonesian-English Code-Switching Speech Synthesizer Utilizing Multilingual STEN-TTS and Bert LID
Ahmad Alfani Handoyo, Chung Tran, Dessi Puji Lestari +1
Multilingual text-to-speech systems convert text into speech across multiple languages. In many cases, text sentences may contain segments in different languages, a phenomenon know…
Continual Learning in Machine Speech Chain Using Gradient Episodic Memory
Geoffrey Tyndall, Kurniawati Azizah, Dipta Tanaya +3
Continual learning for automatic speech recognition (ASR) systems poses a challenge, especially with the need to avoid catastrophic forgetting while maintaining performance on prev…
Enhancing Indonesian Automatic Speech Recognition: Evaluating Multilingual Models with Diverse Speech Variabilities
Aulia Adila, Dessi Lestari, Ayu Purwarianti +3
An ideal speech recognition model has the capability to transcribe speech accurately under various characteristics of speech signals, such as speaking style (read and spontaneous),…
Learning Marmoset Vocal Patterns with a Masked Autoencoder for Robust Call Segmentation, Classification, and Caller Identification
Bin Wu, Shinnosuke Takamichi, Sakriani Sakti +1
The marmoset, a highly vocal primate, is a key model for studying social-communicative behavior. Unlike human speech, marmoset vocalizations are less structured, highly variable, a…
On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition
Nick Rossenbach, Ralf Schlüter, Sakriani Sakti
The rapid development of neural text-to-speech (TTS) systems enabled its usage in other areas of natural language processing such as automatic speech recognition (ASR) or spoken la…
Contrastive Feedback Mechanism for Simultaneous Speech Translation
Haotian Tan, Sakriani Sakti
Recent advances in simultaneous speech translation (SST) focus on the decision policies that enable the use of offline-trained ST models for simultaneous inference. These decision…
NAIST Simultaneous Speech Translation System for IWSLT 2024
Yuka Ko, Ryo Fukuda, Yuta Nishikawa +9
This paper describes NAIST's submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign: English-to-{German, Japanese, Chinese} speech-to-text translation and Engli…
SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain
Heli Qi, Sashi Novitasari, Andros Tjandra +2
This paper introduces SpeeChain, an open-source Pytorch-based toolkit designed to develop the machine speech chain for large-scale use. This first release focuses on the TTS-to-ASR…
Instance-level Heterogeneous Domain Adaptation for Limited-labeled Sketch-to-Photo Retrieval
Fan Yang, Yang Wu, Zheng Wang +3
Although sketch-to-photo retrieval has a wide range of applications, it is costly to obtain paired and rich-labeled ground truth. Differently, photo retrieval data is easier to acq…
NusaCrowd: Open Source Initiative for Indonesian NLP Resources
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji +44
We present NusaCrowd, a collaborative initiative to collect and unify existing resources for Indonesian languages, including opening access to previously non-public resources. Thro…
Actor-identified Spatiotemporal Action Detection -- Detecting Who Is Doing What in Videos
Fan Yang, Norimichi Ukita, Sakriani Sakti +1
The success of deep learning on video Action Recognition (AR) has motivated researchers to progressively promote related tasks from the coarse level to the fine-grained level. Comp…
Speech Artifact Removal from EEG Recordings of Spoken Word Production with Tensor Decomposition
Holy Lovenia, Hiroki Tanaka, Sakriani Sakti +2
Research about brain activities involving spoken word production is considerably underdeveloped because of the undiscovered characteristics of speech artifacts, which contaminate e…
Improved Consistency Training for Semi-Supervised Sequence-to-Sequence ASR via Speech Chain Reconstruction and Self-Transcribing
Heli Qi, Sashi Novitasari, Sakriani Sakti +1
Consistency regularization has recently been applied to semi-supervised sequence-to-sequence (S2S) automatic speech recognition (ASR). This principle encourages an ASR model to out…
Nix-TTS: Lightweight and End-to-End Text-to-Speech via Module-wise Distillation
Rendi Chevi, Radityo Eko Prasojo, Alham Fikri Aji +2
Several solutions for lightweight TTS have shown promising results. Still, they either rely on a hand-crafted design that reaches non-optimum size or use a neural architecture sear…
Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS
Katsuhito Sudoh, Takatomo Kano, Sashi Novitasari +3
This paper presents a newly developed, simultaneous neural speech-to-speech translation system and its evaluation. The system consists of three fully-incremental neural processing…
Cross-Lingual Machine Speech Chain for Javanese, Sundanese, Balinese, and Bataks Speech Recognition and Synthesis
Sashi Novitasari, Andros Tjandra, Sakriani Sakti +1
Even though over seven hundred ethnic languages are spoken in Indonesia, the available technology remains limited that could support communication within indigenous communities as…
Sequence-to-Sequence Learning via Attention Transfer for Incremental Speech Recognition
Sashi Novitasari, Andros Tjandra, Sakriani Sakti +1
Attention-based sequence-to-sequence automatic speech recognition (ASR) requires a significant delay to recognize long utterances because the output is generated after receiving en…
Incremental Machine Speech Chain Towards Enabling Listening while Speaking in Real-time
Sashi Novitasari, Andros Tjandra, Tomoya Yanagita +2
Inspired by a human speech chain mechanism, a machine speech chain framework based on deep learning was recently proposed for the semi-supervised development of automatic speech re…