VibraPhone: Listening through a Vibration Motor


Audio demo

Note: Waveforms are maximally amplified. Perceived volumes vary due to frequency contents (ref: perceived loudness v/s frequency).

Demo 1: (Reference in paper: section 4)

Spoken word: "Entertainment"

Vibra-motor signal: Before processing
voice_raw_vib
Vibra-motor signal: After processing
voice_proc_vib
Microphone signal
voice_mic
Info: Two out of six volunteers identified the word from the raw vibration signal itself, while Automatic Speech Recognition application worked with the processed vibration signal and microphone signal only.

Demo 2:

A few other words with different phonemes.
Words V-motor: Raw V-motor: Processed Microphone
Author
Black
Look
Can you guess the words of this table?
Words V-motor: Raw V-motor: Processed Microphone
?
?

Demo 3: (Reference in paper: section 5.2)

Spoken word: "Often"

Vibra-motor signal: Before processing
voice_raw_vib
Vibra-motor signal: After processing
voice_proc_vib
Microphone signal
voice_mic
Info: For this word signal-processing improved the sound of the fricative consonant [f] and the nasal consonant [n].

Demo 4: (Reference in paper: section 2.2)

Human speech

Vibra-motor signal: After processing

Key research elements

  • Sound sensing through back-EMF: Back-EMF is an electro-magnetic effect observed in magnet-based motors when relative motion occurs between the current carrying armature/coil and the magnetic mass’s own field. Since sound is a source of external vibration, the movable mass in the vibra-motor is expected to exhibit a (subtle) response to it. Our experiments show that, when the vibra-motor is connected to an ADC, the back-EMF generated by the ambient sound can be recorded.
    vibration motor
  • Sensor specific distortion removal: The vibra-motor’s response, on the other hand, is considerably jagged, and thereby induces distortions into the recorded signal. For example, the vibra-motor distortions on the spoken phoneme “u” alters the original formants at 266 and 600Hz to new formants at 300Hz and 1.06KHz, which changes perception of the sound. We apply the frequency domain equalizetion to restore the formant locations.

  • Reconstruction of the missing speech features: Deafness in vibra-motors implies that the motor’s response to high frequency signals (i.e., > 2KHz) is indistinguishable from noise. The erasure of this high frequency features reduces the intelligibility of a recorded voice. We recover the original speech by partially recorstructing speech features from the recorded signal, using the speech energy localization and voice source expansion techniques. energy localization

Experiment platforms

Our experimentation platform is both a Samsung Glaxy S-III smartphone and a custom circuit that uses vibra-motor chips purchased online (these chips are exactly the ones used in today’s phones and wearables).
Custom hardware setup
voice_mic
                 Smartphone setup
voice_mic

Papers

[1] Listening through a Vibration Motor (MobiSys, 2016) [paper]

Related work

[1] Gyrophone: Recognizing Speech From Gyroscope Signals
     USENIX Security Symposium 2014
     Yan Michalevsky, Dan Boneh, Gabi Nakibly [paper]

[2] AccelWord: Energy Efficient Hotword Detection through Accelerometer
     MobiSys 2015
     Li Zhang, Parth H. Pathak, Muchen Wu, Yixin Zhao, Prasant Mohapatra [paper]

[3] Acoustic eavesdropping through wireless vibrometry
     MobiCom 2015
     Teng Wei, Shu Wang, Anfu Zhou, Xinyu Zhang [paper]

[4] The Visual Microphone: Passive Recovery of Sound from Video
     SIGGRAPH 2014
     Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Fredo Durand, William T. Freeman [paper]

[5] Ripple: Communicating through Physical Vibration
     NSDI 2015
     Nirupam Roy, Mahanth Gowda, Romit Roy Choudhury [paper]

[6] Ripple II: Faster Communication through Physical Vibration
     NSDI 2016
     Nirupam Roy, Romit Roy Choudhury [paper]

People

Funding

Faculty
Romit Roy Choudhury
Associate Professor
Dept. of ECE and CS

Students
Nirupam Roy
PhD Candidate
Dept. of ECE

NSF Huawei Intel Samsung