Motion Optic and Sound sensors to Assess Interactive Communication (“MOSAIC”)

The Challenge

Hearing loss disrupts the pathway to healthy ageing and independent living by increasing communication difficulties. Individuals need to be able to communicate with their loved ones and communities, as failing to do so results in isolated individuals with decreased quality of life and increased health risks.

While hearing devices help to re-establish the hearing pathway, listening through devices means communication can still be difficult, and breakdowns inevitably still occur. Hearing devices need to be tailored to the dynamics of the communication environment.

Social interactions mostly take place in the presence of background noise which impacts intelligibility (e.g., TV, street traffic, cafes). Communication for even healthy hearing adults breaks down at a certain level of background noise and whether this pattern remains true for older individuals who wear hearing devices is unknown. Also unknown are the biomarkers that listeners use to signal when communication is difficult and breaks down.

The Solution

Data from sensors can predict conversational difficulty and communication breakdowns by tracking gaze and head movements in the lead-up to the breakdown.  The next step is to leverage this knowledge to selectively enhance hearing devices.

We want to identify which biomarkers have the greatest predictive power in detecting communication difficulty and which have the greatest power in predicting communication breakdowns. This will allow the MOSAIC team to fuse sensor networks with next-generation signal processing algorithms and hearing technologies. It will also allow us to provide real-time user feedback about listening environments to people and their communication partners. For example, Google Research (Australia) has a collaboration with its subsidiary, Fitbit (Australia) to explore tactile feedback signalling difficult listening situations.

We will also assess a large cohort of normal-for-age hearing listeners across the lifespan (baseline data and ageing effects) as well as older bilateral cochlear implant recipients.

Industry Partner

University Partners