In this opinion piece, NSSN Co-Director and Head of the School of Electrical Engineering and Telecommunications at UNSW Sydney, Professor Julien Epps, tells us about his experience with creating smart technologies capable of sensing mental health.

During the early part of my electrical engineering career, I focused my research efforts on progressing speech signal processing research, and would never have imagined my work would lead to sensing emotion and mental state.

Around fifteen years ago, my research group at UNSW was among the earliest to experiment with applying automatic speaker authentication methods to recognising emotions from speech. Around that time, it was already becoming clear that computing devices of one kind or another – smartphones, call centres, voice over IP or smart home assistants – would be part of many, perhaps most, conversations.

The prospect that smart sensing could be easily scaled up via software applications for these platforms to benefit millions or billions of people was the spark for my team’s investigation into speech-based assessment of mental state.

Speech production is the most complex motor coordination activity performed by our bodies, so it is inevitable that our mental state affects the way we speak. We just need to work out how to sense this automatically.

*Image: NSSN Co-Director and Head of the School of Electrical Engineering and Telecommunications at UNSW Sydney, Professor Julien Epps.*

Of course, clinicians have known this for some time. Subjective descriptions of speech affected by depression as “flat” and “monotonous” have been reported for several decades. However automatically detecting states like depression is surprisingly difficult.

Speech is a very rich signal, conveying a wide variety of different sounds in rapid succession. To make things more complicated, even if different people are saying the same thing, every speaker says it differently.

My team begins by breaking the sensed speech signal up into very small chunks of time, calculates its intensity across multiple frequency bands, which we refer to as spectral features, and then uses a range of different approaches to model depression

One of these is to create a mathematical characterisation of the spectral features and then work out how ‘spread out’ this characterisation is, with more depressed speech showing less spread.

Another that we are particularly known for is to find the abrupt changes in the spectral features and use these to calculate a multitude of different timings, which we then compare with models for depressed and non-depressed speakers. This approach has yielded state-of-the-art results on a benchmark research dataset.

How would this work outside of a research lab? The methods are likely to be used for screening, i.e. automatically detecting depression, or monitoring, for example making frequent assessments of the response to medication. These are likely to be complementary to the support that health professionals can offer, and help people connect better with this support.

Partner company Sonde Health has made some of our developed methods a standard part of its smartphone app products. They will even be integrated directly into Qualcomm’s 888 and 778G chips that will be used in mid-range and top-tier phones from leading manufacturers from the end of this year.

One of the most compelling things about this kind of mental health sensing is that it can go from an idea on a research team’s whiteboard to a simulation to commercial code in a matter of months. Then, as we know from COVID-19, when you release an app it can potentially be in the hands of billions of people within hours.

For these reasons, smart sensing has enormous potential to empower Australians to manage aspects of their own health care. This is just the beginning.

Partnering with the NSSN simplifies the process of engaging with universities by creating a single point-of-contact for the leading research-intensive universities. The NSSN enables companies and government partners to access expertise and technology from across NSW's and ACT's leading universities and gain an enduring competitive advantage.

Media contact: Shahrzad (Zad) Abbasi - 0466 548 145

The NSW Smart Sensing Network (NSSN), an initiative of the NSW Government Office of the Chief Scientist and Engineer, is a consortium of eight leading universities across NSW and the ACT that brings together academia, industry and government to develop smart sensing solutions for complex challenges.

Working with the NSSN simplifies the process of engaging with universities by creating a single point-of-contact for the leading research suppliers in NSW.

Partner with us here.

Mental health sensing in the palm of your hand