Perelman School of Medicine at the University of Pennsylvania

AEREO: The Consortium to Advance Effective Research Ethics Oversight


Do IRBs “work”? What does that mean – and how would we know?

Launched in May 2018, AEREO is a consortium of leaders in human subjects research oversight, research ethics, and empirical methods. We have come together to advance the shared mission of evaluating and improving the effectiveness of Institutional Review Boards (IRBs) and Human Research Protection Programs (HRPPs). Consortium members work collaboratively to advance an empirical research agenda to support evidence-based approaches to research ethics review, oversight, and policy.

What does effectiveness mean in this context?

Effectiveness refers to whether and how well an intervention achieves relevant outcomes. The outcomes of interest for IRBs and HRPPs are:

  • Protection of research participants
  • Promotion of justice in research
  • Creating a culture of ethical concern amongst research stakeholders
  • Establishing public trust in the research enterprise
  • Ensuring socially valuable, scientifically valid, ethical research

These outcomes can be difficult to evaluate directly, in part because they are not well-defined or easily operationalized. They also reflect some of the very same determinations that IRBs and HRPPs are expected to make themselves and about which there may be reasonable disagreement. In light of these challenges, existing efforts to evaluate IRBs and HRPPs have largely focused on more easily measured procedural and structural factors, such as the expertise represented on boards, how they deliberate, the types of standard operating procedures they have and follow, and whether they comply with relevant regulations and guidance.

Our aims are to identify the procedural and structural factors most likely to support IRBs and HRPPs in achieving their outcomes of interest, to specify the scope of reasonable IRB and HRPP determinations, and to find ways to measure how IRBs and HRPPs influence the experiences of research participants - a critical area that has been largely overlooked in prior evaluation efforts. Rather than focusing exclusively on what IRBs and HRPPs do in reviewing proposed research, we are especially interested in studying their impact once research is allowed to proceed.

What's the difference between effectiveness and efficiency?

Much of the empirical data currently collected about IRBs and HRPPs focus on their turn-around time and how quickly they issue a decision on proposed research. These are important questions, but they relate to efficiency. The speed of protocol review does not tell us anything about how effective that review is in successfully achieving the outcomes of interest. AEREO seeks to move the discussion around IRB and HRPP quality beyond measures of efficiency to focus on the central question of effectiveness.  

What have we learned so far?

Take a look at AEREO's publication list. Work so far has led to the following recommendations for HRPPs and IRBs aiming to improve quality and effectiveness:

  • Attend to compliance but minimize “audit culture” that focuses on achieving metrics for their own sake over attention to substantive goals 
  • Prioritize assessments directly related to participant protection outcomes (e.g., quality of the informed consent process and understanding, robust review of adverse events and whether they could have been avoided, and overall participant experience in research)
  • Emphasize assessment of processes and standards likely to promote participant protection through identification of risks and implementation of safeguards (e.g., how systematically IRBs evaluate research risks and benefits, data monitoring plans, and plans to provide care and compensation for participant injury)
  • Make sure reviewers are fully informed about relevant details of proposed and ongoing research and that they have adequate time to review materials and deliberate
  • Examine the extent of meaningful discussion and debate about key ethical and regulatory issues during board meetings
  • Promote diversity, expertise, and independence among board members to ensure consideration of a variety of relevant perspectives and call in outside experts, as needed
  • Encourage reliance on prior decisions to promote consistency, where appropriate, and efficiency, as well as to clarify the meaning of ambiguous ethical and regulatory standards
  • Include attention to whether and how the HRPP monitors the conduct of research beyond protocol review (i.e., what happens after IRB approval)
  • Adopt assorted quality assessments that incorporate feedback from and about a variety of stakeholders (e.g., board members, HRPP staff, research teams, study participants and others) obtained through a variety of methods (e.g., checklists, site visits, record review, interviews, and surveys)

Learn more about the problems AEREO is trying to solve and some of our ideas so far from AEREO's co-chair, Holly Fernandez Lynch: