Overview of the OBSERVER Project

In the past decade, research teams have increasingly relied on clinic visit recordings for various research projects.

These data provide a unique view into the exam room. Although the NIH and other funders have supported the collection of these valuable data, no repositories support aggregating clinic encounter data for reuse by other investigators. The reasons for that are numerous but often relate to the challenges of de-identifying video and the significant storage costs associated with these large files. However, advances in computer vision, natural language processing, and machine learning support labeling images, obscuring faces, removing words from audio and transcripts, and blurring words on video.

The Observer Repository, a novel research repository adhering to FAIR (Findable, Accessible, Interoperable, and Reusable) data-sharing principles, aims to solve this challenge. The Observer Repository Team is focusing on developing a secure, web-based platform that allows researchers to review and download multimodal visit-related data (video, audio, transcripts, room sensor) that we collectively call the visitome, based on attributes derived from natural language processing, computer vision, and machine learning methods.  The construction of this repository is guided by a multidisciplinary team of biomedical informatics, machine learning, natural language processing, systems architecture, and general computer science experts, focusing initially on aggregating telemedicine videos from studies conducted by researchers at the University of Pennsylvania.  After the Observer Repository has sufficient data (approximately 100 samples), we will invite researchers to conduct a series of pilot experiments to demonstrate the utility of this repository.  We leverage a steering committee and external advisory board to help us develop an equitable governance strategy, to provide us with a starter set of clinical visit data, and to work with us to develop a sustainability plan.

A Call for Collaboration

We invite students and researchers to help us build this novel resource. Your insights and innovations can play a pivotal role in shaping the future of healthcare. Consider these open questions as starting points for your research and exploration:

  1. How can we address data fusion challenges from diverse sources like video, audio, EHR, and patient data for a comprehensive view in the visitome repository?
  2. How can machine and deep learning be applied to identify complex patterns in multimodal visitome data, especially subtle anomalies in patient-provider interactions?
  3. What predictive modeling methods can be developed for patient outcomes using verbal and non-verbal cues in visitome data?
  4. What techniques can analyze behavioral patterns, such as micro-expressions and tone modulations, in patient-provider interactions for diagnostic accuracy?
  5. How can longitudinal visitome data be used to analyze the evolution and impact of patient-provider relationships on treatment outcomes?
  6. What methods can automate sentiment analysis of patient-provider conversations in visitome data for understanding patient satisfaction and emotional well-being?
  7. How can AI models recognize and interpret cross-cultural communication variations in visitome data for culturally competent care?
  8. What insights can be gained from analyzing video data compared to audio-only in visitome repositories, and how can these modalities complement each other for a more comprehensive understanding of patient-provider interactions?
  9. What strategies and technologies can be employed to deidentify visitome data to ensure patient privacy while retaining the integrity and utility of the data for research and analysis?
  10. How has telemedicine changed patient-provider interactions, and how can we analyze virtual visitome data for optimizing remote healthcare?