When Your Phone Knows More About How You’re Feeling Than You Do

Your mobile phone rings. When you answer it, you hear a voice saying: “Hello. This is your iPhone. Are you feeling OK? I’m detecting hints of sadness in your voice, interspersed with anger. Should I call your therapist for you? Would you like a hug?”

OK, this is still science fiction, but the reality of it is getting closer. Scientists at the University of Rochester have already created a mobile phone application (an “app”) that can tell how you’re feeling, just from the sound of your voice. It’s like a high-tech “mood ring,” only smarter. A working prototype of the app is shown below:



Feeling blue? There’s an app for that

This article is, at this point, is more of a “high tech” piece than it is a health-related piece, but give it time. What they have done is to pave the way for apps that could detect how you’re feeling and react accordingly. Just imagine your mobile phone detecting stress in your voice, and booking you a nice massage. Or detecting sadness in your voice and automatically playing your favorite uplifting song, or beeping you with a reminder to take your antidepressants. Or, possibly more to the point, detecting sounds of panic in your voice and calling the police, or detecting chest pains and calling an ambulance. These things are all coming, and this technology makes them all possible.

What it does is to detect the presence of human emotions in speakers’ voices, without the need to analyze the words. As Wendi Heinzelman, professor of electrical and computer engineering and one of the project’s leaders says, “We actually used recordings of actors reading out the date of the month – it really doesn’t matter what they say, it’s how they’re saying it that we’re interested in.” The algorithm they invented analyzes 12 features of human speech (for example, pitch or volume) in a sound recording and searches for one of six emotions that it can detect. The working prototype achieves an 81% accuracy rate when doing this.

The algorithm works because it works the same way our brains do when trying to detect emotion. When people speak, they unconsciously alter the way they speak based on how they are feeling; they may change the volume, or even the harmonics present in their speech. “You might hear someone speak and think ‘oh, he sounds angry!’ But what is it that makes you think that? We don’t pay attention to these features individually, we have just come to learn what angry sounds like – particularly for people we know.”

But there’s a catch

It’s in that last phrase above – “for people we know.” There is still a lot of work to be done before this app is ready for prime time. At present, the 81% accuracy rating is dependent on analyzing emotions in one particular voice, one that was previously sampled and analyzed. If you use the system on a different voice, accuracy drops to 30%. The researchers are working on minimizing this by originally “training” the system with a number of voices of different ages and genders, so that it can match its “memory voices” with a new speaker’s voice and thus function with much more accuracy.

But it’s still a marvelous breakthrough, and I wish them luck with the follow-up research. There is great potential for medicine and health care in being able to easily detect a person’s emotional state.

Leave a Reply

Your email address will not be published.

Juliette Siegfried, MPH

Juliette Siegfried, MPH, has been involved in health communications since 1991. Shortly after obtaining her Master of Public Health degree, she began her career at the National Institutes of Health in Bethesda, Maryland. Juliette now lives in Europe, where she launched ServingMed(.)com, a small medical writing and editing business for health professionals all over the world.

Juliette's resume, facebook: juliette.siegfriedmph, linkedin: juliettes, (+31) 683 673 767

Recommended Articles