Observer Pilot Awards

The University of Pennsylvania's Artificial Intelligence for Ambulatory Care Innovation (AI-4-AI) Lab is pleased to announce its 2025 Request for Applications. The Observer Repository Pilot Awards consist of $20,000 grants for five innovative teams to explore untapped opportunities in ambulatory care research and practice using the Observer Repository — a unique dataset of video-recorded outpatient visits linked with EHR data and patient/provider metadata. These awards aim to catalyze transformative, interdisciplinary projects that leverage the dataset to generate hypotheses, test methods, advance educational innovations, or build machine learning and AI applications.

We welcome proposals that drive hypothesis generation, methodological innovation, education, or machine learning/AI development. This pilot program is supported by the Heyman AI Innovation Fund. 

Observer Dashboard

The Observer Project is a pioneering initiative housed at the University of Pennsylvania that captures real-world, multimodal data from outpatient visits to support healthcare innovation and research. The repository includes high-resolution video (both egocentric and room-view), audio, transcripts, EHR audit logs, visit metadata, and satisfaction surveys. This rich dataset enables researchers to explore clinical workflows, patient-provider interactions, cognitive support needs, and the development of novel AI and educational tools. The project was developed with input from patients, clinicians, and ethicists and employs a rigorous privacy-preserving pipeline to ensure data sharing is compliant and ethical.  

Mission:

To transform healthcare through ethically grounded, video-based clinical research by enabling open, multimodal data access for the study and improvement of real-world clinical care.  

Goals: 

• Capture Real-World Clinical Interactions: Document outpatient visits using high-resolution video (egocentric and room-view), audio, transcripts, and structured metadata. 
• Support Healthcare Innovation: Facilitate the development of AI tools, educational technologies, and workflow improvements through data-driven insights.
• Protect Patient Privacy: Implement a rigorous, privacy-preserving pipeline to ensure ethical and compliant data sharing. 
• Foster Multidisciplinary Development: Engage patients, clinicians, and ethicists in shaping responsible and meaningful research.

Learn More 

• Number of Awards: 5 
• Award Amount: $20,000  
• Duration: 12 months  
• Deliverables: Final report at pilot end date and project presentation at our virtual seminar which will be held after the pilot end date. Any deliverables reported in Awardees Data Sharing plan.  

Round 1: 

Round 2 (Invite Only): 

Note: All round two dates are currently estimated. All dates will be finalized and updated prior to the start of round two. 

We welcome applications from faculty, postdoctoral researchers, research staff, and interdisciplinary teams affiliated with the University of Pennsylvania, or any accredited university within the United States. Interdisciplinary collaborations are strongly encouraged and may include fields such as informatics, behavioral science, computer science, clinical medicine, education, design, or sociology.  

Projects must make meaningful use of the Observer Repository. Applicants may select specific visit types, modalities (e.g., egocentric video, EHR logs), or provider/patient subgroups. Combining Observer Repository data with external datasets is permissible. Projects may focus on qualitative or quantitative inquiry.  

Successful proposals will: 

• Utilize Observer’s multimodal data (video, transcription, EHR data, and metadata) 
• Explore preliminary research questions relevant to patient-clinician communication, workflow optimization, or technology integration 
• Identify potential innovative AI-driven approaches and their preliminary feasibility 
• Clearly define how the findings could guide future transformative improvements in ambulatory care 

Awardees will access data via a secure platform following execution of a Data Use Agreement. Available data includes:  

• Room-view and egocentric video  
• Audio  
• Transcripts and visit summaries  
• EHR audit logs and visit data  
• Patient and provider demographics  
• Satisfaction surveys and metadata  

Applicants are encouraged to examine the Observer Repository dashboard to familiarize themselves with the data that will be made available. There is also sample data of a clinical visit available, including audio, video, transcription, and EHR data. 

Observer Repository Dashboard

Letter of Intent

A Letter of Intent (LOI) is required for all applicants interested in submitting a project proposal. 

The LOI (up to 2 pages) should include:  

1. Project Title  
2. Principal Investigator and Team information 
3. Project Summary  
4. Expected Contribution (e.g., models, tools, insights, annotations)  
5. Data Needs (visit types, modalities)  
6. Feasibility and relevant expertise  
7. Optional: Prior Work or Collaborations 

More detailed instructions can be found here: Letter of Intent Application

LOIs should be submitted to observerproject@pennmedicine.upenn.edu by the due date (see schedule above) with the subject line ‘Observer Pilot LOI Submission’. 

Following the review of letters of intent, applicants may be invited to submit full proposals. 

Pilot Award Application

Applicants invited to submit a full proposal application will be required to submit an expanded proposal which will include:  

1. Project Title  
2. Principal Investigator and Team Information 
3. Expanded Project Summary 
4. Research Strategy  
5. Timeline and Milestones  
6. Budget and Budget Justification  
7. Biosketches 
8. Data Sharing Plan  
9. Optional: Letters of Support or Relevant Prior Work 

More detailed instructions can be found here: Full Application Application

Letters of intent will be reviewed on a number of factors, including alignment with program goals, feasibility within 12 months and $20,000, innovation and potential impact, clear and appropriate use of Observer Repository data, and diversity of perspectives or disciplines.  

Additional consideration may be given to proposals that: 

• Investigate new methodologies for enhancing patient-clinician interactions and communication 
• Conduct feasibility or pilot studies addressing workflow optimization using AI 
• Examine preliminary feasibility of new technological integrations 
• Offer clear pathways for future impactful studies based on initial results 

For more information, visit https://observer.med.upenn.edu/ or contact observerproject@pennmedicine.upenn.edu.  

An optional informational webinar is being offered prior to the LOI deadline on August 15, 2025, to review data capabilities and answer applicant questions. Details will be updated on the website.