Objective:
To present, analyze and evaluate a cloud-based audio-visual dialog agent and its associated automated speech, face and movement analytics for remote patient monitoring in Parkinson’s disease (PD).

Background:
The need for remote monitoring to support patients, caregivers, and healthcare professionals in their collaborative efforts for better care has never been greater, a situation which has been brought into more acute focus by the SARS-COV-2 pandemic (Motolese et al., 2020). Apart from improving ease and frequency of access, the rich data captured in the natural environment of patients’ homes can enhance our understanding of a patient’s condition and tailor his/her treatment (Papapetropoulos, Mitsi & Espay, 2015). This could improve outcomes for individual patients while substantially decreasing healthcare costs.

Method:
We present Nina, a virtual dialog agent that conducts on-demand automated screening interviews through a HIPAA-compliant, secure screening portal with patients with PD over the phone or web browser to monitor PD progression and facilitate the development of treatment plans. During the conversation, Nina engages patients in a mixture of structured speaking exercises and open-ended questions to elicit speech and facial behaviors. She also engages patients in a novel finger-tapping exercise designed to test limb motor function, and can instruct patients to complete standard survey instruments such as the PDQ-39. 

Analytics modules automatically extract multiple speech metrics (speaking rate, duration), video features (range and speed of movement of the lips and jaw), and the range, speed and periodicity of finger tapping. After the interaction, analytics are accessible through an easy-to-use dashboard which provides a high-level overview of the interaction,as well as a detailed breakdown of the individual sessions and patient responses.

Results:
We investigate speech, facial and motoric measures from initial experiments where 40 people with PD interacted with Nina. At the time of writing, over 85% of patients reported being able to complete the session independently without difficulty. We will also demonstrate the system and the associated metrics along with preliminary comparisons with offline gold-standard measurements.

Conclusion:
Leveraging multimodal conversational agents for human-in-the-loop remote monitoring has the potential to revolutionize the diagnosis and tracking of PD.

References: Motolese, F., Magliozzi, A., Puttini, F., Rossi, M., Capone, F., Karlinski, K., Stark-Inbar, A., Yekutieli, Z., Di Lazzaro, V., & Marano, M. (2020). Parkinson’s Disease Remote Patient Monitoring During the COVID-19 Lockdown. Frontiers in neurology, 11, 567413. https://doi.org/10.3389/fneur.2020.567413 Papapetropoulos, S., Mitsi, G., & Espay, A. J. (2015). Digital health revolution: is it time for affordable remote monitoring for Parkinson’s disease?. Frontiers in neurology, 6, 34.

« Back to MDS Virtual Congress 2021

MDS Abstracts - https://www.mdsabstracts.org/abstract/multimodal-conversational-ai-for-remote-patient-monitoring-and-analysis-of-parkinsons-disease/