“It’s hard to tell what an IoT device is doing with its microphone beyond an LED signifying its microphone is in a muted or unmuted state,” says Yasha Iravantchi, a graduate student at the University of Michigan. “In the future, these devices are going to perform more tasks than just listening for speech commands and there is the possibility that a lot of audio will be captured, recorded, and stored.”

For this reason, Yasha has been leading a team developing PrivacyMic, a research project ensuring private conversations cannot be recorded and stored. By only gathering sound at frequencies above the range of human hearing, the system filters out speech and audible sound, yet can still understand what’s happening in our environment.

Sound of silence

PrivacyMic is built around Raspberry Pi and it works with ultrasonic sounds – that is, those with a frequency of 20 kilohertz or higher. Many objects and actions emit ultrasound waves, including compact fluorescent bulbs, dishwashers, computer monitors, flushing toilets, and electric toothbrushes. “It’s a frequency that’s inoffensive to humans, but can be annoying to dogs,” says Yasha.

Most traditional audio equipment won’t capture ultrasonic sounds. “Devices are tuned to focus on the range of human speech or hearing and they often actively remove the sounds outside of these ranges as ‘noise’ from the environment,” Yasha explains. By creating a HAT for Raspberry Pi Zero W using an analogue ultrasonic microphone and a filter to remove speech and audible frequencies, however, PrivacyMic can do the opposite.

“The ‘noise’ that these systems throw away is a valuable signal we can use to recognise and classify daily-use objects,” Yasha says. It means PrivacyMic can hear when a light bulb or microwave is turned on, determine when a toothbrush is being used, or when a toilet is flushed, without capturing any conversations. “We’ve also been using Raspberry Pi 3 to explore ways of performing all tasks – from capture to classification – on the computer itself to ensure no data ever leaves Raspberry Pi.”

Listen up

So how does it work? “Analogue filters remove the audible frequencies and only allow ultrasonic frequencies to pass through to the analogue-to-digital converter (ADC),” Yasha explains. “These signals are passed to a machine learning model to recognise these daily-use objects in ultrasound only, and the model then records an entry, including a time stamp and the name of the thing that turned on.”

In doing so, the filtered ultrasound is not kept. “The idea is that there are multiple layers of safeguards in place that allow us to minimise the privacy exposure to users while keeping the artefacts that matter. In the case of activities of daily living tracking, it’s the activity log, not the sounds themselves that are of value,” Yasha says.

Accomplishing this has entailed a lot of coding. C is used to efficiently send samples from the ADC to a laptop via TCP, and a new framework for interactive machine learning has been developed in Python. “It’s called T4Train and it’s a work-in-progress, but it’s already compatible with Raspberry Pi,” Yasha says.

Even so, work is far from complete. “One of the hardest challenges is trying to train PrivacyMic to learn to recognise sounds that you yourself cannot hear,” Yasha laments. It means the device won’t likely be out of the proof-of-concept stage for a few years yet, but we’ll keep our ears to the ground for more news as the project progresses.