Info: Two out of six volunteers identified the word from the raw vibration signal itself, while Automatic Speech Recognition application worked with the processed vibration signal and microphone signal only.
Sound sensing through back-EMF: Back-EMF is an electro-magnetic effect observed in magnet-based motors when relative motion occurs between the current carrying armature/coil and the magnetic mass’s own field. Since sound is a source of external vibration, the movable mass in the vibra-motor is expected to exhibit a (subtle) response to it. Our experiments show that, when the vibra-motor is connected to an ADC, the back-EMF generated by the ambient sound can be recorded.
Sensor specific distortion removal: The vibra-motor’s response, on the other hand, is considerably jagged, and thereby induces distortions into the recorded signal. For example, the vibra-motor distortions on the spoken phoneme “u” alters the original formants at 266 and 600Hz to new formants at 300Hz and 1.06KHz, which changes perception of the sound. We apply the frequency domain equalizetion to restore the formant locations.
Reconstruction of the missing speech features: Deafness in vibra-motors implies that the motor’s response to high frequency signals (i.e., > 2KHz) is indistinguishable from noise. The erasure of this high frequency features reduces the intelligibility of a recorded voice. We recover the original speech by partially recorstructing speech features from the recorded signal, using the speech energy localization and voice source expansion techniques.
Experiment platforms
Our experimentation platform is both a Samsung Glaxy S-III smartphone and a custom circuit that uses vibra-motor chips purchased online (these chips are exactly the ones used in today’s phones and wearables).
Custom hardware setup
Smartphone setup
Papers
[1] Listening through a Vibration Motor (MobiSys, 2016) [paper]
Related work
[1] Gyrophone: Recognizing Speech From Gyroscope Signals USENIX Security Symposium 2014 Yan Michalevsky, Dan Boneh, Gabi Nakibly [paper]
[2] AccelWord: Energy Efficient Hotword Detection through Accelerometer MobiSys 2015 Li Zhang, Parth H. Pathak, Muchen Wu, Yixin Zhao, Prasant Mohapatra [paper]
[3] Acoustic eavesdropping through wireless vibrometry MobiCom 2015 Teng Wei, Shu Wang, Anfu Zhou, Xinyu Zhang [paper]
[4] The Visual Microphone: Passive Recovery of Sound from Video SIGGRAPH 2014 Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Fredo Durand, William T. Freeman [paper]
[5] Ripple: Communicating through Physical Vibration NSDI 2015 Nirupam Roy, Mahanth Gowda, Romit Roy Choudhury [paper]
[6] Ripple II: Faster Communication through Physical Vibration NSDI 2016 Nirupam Roy, Romit Roy Choudhury [paper]