169 citations · 870 across the 37 of their papers we have counts for
42 papers
Jarvis: A Proactive Speech Agent for Multi-Party Conversations
Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2
Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among…
Trigger Sound Suppression for Misophonia
Vaishnavi Vidyasagar, Jasmine Zhang, Mahima Uliyar +4
Misophonia, a disorder of decreased tolerance to specific sounds, affects 5-20% of the population, yet sufferers have no good options: therapy helps a minority, and earplugs or noi…
Fine-grained Soundscape Control for Augmented Hearing
Seunghyun Oh, Malek Itani, Aseem Gauri +1
Hearables are becoming ubiquitous, yet their sound controls remain blunt: users can either enable global noise suppression or focus on a single target sound. Real-world acoustic sc…
Proactive Hearing Assistants that Isolate Egocentric Conversations
Guilin Hu, Malek Itani, Tuochao Chen +1
We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on eg…
AV-Dialog: Spoken Dialogue Models with Audio-Visual Input
Tuochao Chen, Bandhav Veluri, Hongyu Gong +1
Dialogue models falter in noisy, multi-speaker environments, often producing irrelevant responses and awkward turn-taking. We present AV-Dialog, the first multimodal dialog framewo…
TF-MLPNet: Tiny Real-Time Neural Speech Separation
Malek Itani, Tuochao Chen, Shyamnath Gollakota
Speech separation on hearable devices can enable transformative augmented and enhanced hearing capabilities. However, state-of-the-art speech separation networks cannot run in real…
Neural Speech Extraction with Human Feedback
Malek Itani, Ashton Graves, Sefik Emre Eskimez +1
We present the first neural target speech extraction (TSE) system that uses human feedback for iterative refinement. Our approach allows users to mark specific segments of the TSE…
LLAMAPIE: Proactive In-Ear Conversation Assistants
Tuochao Chen, Nicholas Batchelder, Alisa Liu +2
We introduce LlamaPIE, the first real-time proactive assistant designed to enhance human conversations through discreet, concise guidance delivered via hearable devices. Unlike tra…
Spatial Speech Translation: Translating Across Space With Binaural Hearables
Tuochao Chen, Qirui Wang, Runlin He +1
Imagine being in a crowded space where people speak a different language and having hearables that transform the auditory space into your native language, while preserving the spat…
Wireless Hearables With Programmable Speech AI Accelerators
Malek Itani, Tuochao Chen, Arun Raghavan +2
The conventional wisdom has been that designing ultra-compact, battery-constrained wireless hearables with on-device speech AI models is challenging due to the high computational d…
Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
Bandhav Veluri, Benjamin N Peloquin, Bokai Yu +2
Despite broad interest in modeling spoken dialogue agents, most approaches are inherently "half-duplex" -- restricted to turn-based interaction with responses requiring explicit pr…
Target conversation extraction: Source separation using turn-taking dynamics
Tuochao Chen, Qirui Wang, Bohan Wu +4
Extracting the speech of participants in a conversation amidst interfering speakers and noise presents a challenging problem. In this paper, we introduce the novel task of target c…
IRIS: Wireless Ring for Vision-based Smart Home Interaction
Maruchi Kim, Antonio Glenn, Bandhav Veluri +5
Integrating cameras into wireless smart rings has been challenging due to size and power constraints. We introduce IRIS, the first wireless vision-enabled smart ring system for sma…
Knowledge boosting during low-latency inference
Vidya Srinivas, Malek Itani, Tuochao Chen +3
Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A p…
Look Once to Hear: Target Speech Hearing with Noisy Examples
Bandhav Veluri, Malek Itani, Tuochao Chen +2
In crowded settings, the human brain can focus on speech from a target speaker, given prior knowledge of how they sound. We introduce a novel intelligent hearable system that achie…
Semantic Hearing: Programming Acoustic Scenes with Binaural Hearables
Bandhav Veluri, Malek Itani, Justin Chan +2
Imagine being able to listen to the birds chirping in a park without hearing the chatter from other hikers, or being able to block out traffic noise on a busy street while still be…
Solar-powered shape-changing origami microfliers
Kyle Johnson, Vicente Arroyos, Amélie Ferran +7
Using wind to disperse microfliers that fall like seeds and leaves can help automate large-scale sensor deployments. Here, we present battery-free microfliers that can change shape…
Underwater 3D positioning on smart devices
Tuochao Chen, Justin Chan, Shyamnath Gollakota
The emergence of water-proof mobile and wearable devices (e.g., Garmin Descent and Apple Watch Ultra) designed for underwater activities like professional scuba diving, opens up op…
Wireless earbuds for low-cost hearing screening
Justin Chan, Antonio Glenn, Malek Itani +5
We present the first wireless earbud hardware that can perform hearing screening by detecting otoacoustic emissions. The conventional wisdom has been that detecting otoacoustic emi…
Real-Time Target Sound Extraction
Bandhav Veluri, Justin Chan, Malek Itani +3
We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture wit…
Underwater Acoustic Ranging Between Smartphones
Tuochao Chen, Justin Chan, Shyamnath Gollakota
We present a novel underwater system that can perform acoustic ranging between commodity smartphones. To achieve this, we design a real-time underwater ranging protocol that comput…
Underwater Messaging Using Mobile Devices
Tuochao Chen, Justin Chan, Shyamnath Gollakota
Since its inception, underwater digital acoustic communication has required custom hardware that neither has the economies of scale nor is pervasive. We present the first acoustic…
NeuriCam: Key-Frame Video Super-Resolution and Colorization for IoT Cameras
Bandhav Veluri, Collin Pernu, Ali Saffari +3
We present NeuriCam, a novel deep learning-based system to achieve video capture from low-power dual-mode IoT camera systems. Our idea is to design a dual-mode camera system where…
ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement
Ishan Chatterjee, Maruchi Kim, Vivek Jayaram +4
We present ClearBuds, the first hardware and software system that utilizes a neural network to enhance speech streamed from two wireless earbuds. Real-time speech enhancement for w…