AI and Regulatory Innovation in Healthcare: Shaping the Future
Author: Alliance for Artificial Intelligence in Healthcare
With Contributions from:
Panel:
Paul Howard, PhD, Senior Director, Public Policy, Amicus Therapeutics (Moderator)
Panelists:
Chris Escobedo Hart, JD, Partner, Foley Hoag
Mida Pezeshkian, PhD, Founder, Principal, STEMA_cg
Brandon Rice, Chief Product Officer, Weave
Alex Zhavoronkov, PhD, CEO, Insilico Medicine
Executive Summary
At BIO 2024, the Alliance for Artificial Intelligence hosted a panel of our members to discuss the challenges of AI and Regulatory Innovation. Expertly moderated by Paul Howard, Senior Director, Amicus Therapeutics, the panel worked through a SWOT analysis on the current state of AI and regulation. The panel brought together diverse perspectives from Chris Escobedo Hart, JD, Partner, Foley Hoag; Mida Pezeshkian, PhD, Founder, Principal, STEMA_cg; Brandon Rice, Chief Product Officer, Weave; and Alex Zhavoronkov, PhD, Founder and CEO, Insilico Medicine.
The panel covered diverse topics including data privacy, best practices for training, testing, and validating machine learning models (including Generative AI) based on representative datasets, and how to grapple with challenges due to missing data, data bias, and hallucinations.
Through this rich exchange of ideas, we identified several key takeaways that highlight the intricate balance between innovation, regulation, and practical application in this rapidly evolving field. Below is the graphic as well as more detailed descriptions of the takeaway.
Content
Strengths
In the US there are many publicly accessible, high-quality datasets that can be used to train GenAI.
LLMs are highly flexible/iterative and while they need a lot of training data, smaller, highly curated datasets (like INDs) that involve text to text learning can be leveraged for high value use cases.
FDA/US regulators are extremely efficient at moving through large, complex clinical datasets, and that includes submissions leveraging AI/ML approaches.
In general, international regulators, (FDA, EMA, UK) are converging on a risk-based approach for regulating AI/ML tools in healthcare based on best practices like human oversight, explainability, transparency, monitoring model performance with real world data, and identifying/reducing data bias.
Weaknesses
EU vs US vs UK – very different approaches when it comes to issues like data privacy and comprehensiveness of regulation. In general, EU AI regulations are more comprehensive/centralized, and the US is very fragmented. The UK is evolving but leans towards prioritizing speed of innovation. But even in the EU, member standards can be more stringent that EU wide regulations.
US FDA AI/ML regulation can vary depending on the type of product and which center (CDRH, CDER, CBER) is regulating it. This can cause confusion among sponsors and slow adoption/investment.
Regulatory harmonization is lacking; regulations aren’t consistent enough for global companies to scale across jurisdictions, and smaller companies can be exposed to significant liability for model performance, or costs for “red-teaming.”
Generative AI systems need public benchmarking and characterization, especially in the fields of reinforcement learning and training/validation.
Data is still not always consistent in terms of structure, format, quality and information; interoperability is still lacking.
Opportunities
AI is extremely useful where repetitive tasks are the focus of the output. Allows for guardrails to be easily implemented and oversight is more straightforward.
Preventing hallucinations through use of data and results that are traceable, verifiable, etc. Being able to build and demonstrate controls in the beginning is important and a differentiator.
Humans hallucinate too and by combining the expertise of humans and the expertise of AI you can help prevent some of these issues. Human oversight in the AI lifecycle is key.
Defining benchmarking and characterization is important to prevent unchecked hallucinations
Understanding how you can train LLMS to perform simple tasks and even use LLMS to check the work of your LLM (models can check the work of other models).
Bringing in diverse teams (legal, science, regulatory, compliance, ethics, business) early can help ensure better implementation and decrease potential risks.
Synthetic data help address some of the issues of representation in data sets, as well as provide enhanced privacy protections.
Leveraging learnings from other fields (finance, aviation, retail, etc) can help accelerate adoption in healthcare.
Threats
Dark data, missing data, biased data is still an important issue when it comes to identifying bias and ensuring representativeness; identifying data gaps should be a first order priority.
Policymaking around AI is very much a moving target, especially in the US, with both state and federal legislators considering additional safeguards that could lead to overregulation.
Hype around AI damages confidence in its performance both on the upside (utopian) and downside (apocalyptic).
The “automation paradox”: over reliance on AI models/outputs can reduce human competence and lead to worse outcomes over time; AI implementation must remain human-centric, and emphasize “human-in-the-loop oversight” with real world feedback.
Privacy laws causing increase in impact assessment, with smaller firms less able to compete in more complex regulatory environments.
Alignment with human values across societies and over time will be an ongoing challenge, especially for international regulators.
Conclusion
As a rule of thumb, using AI/ML models in the pre-clinical space (for finding molecules or optimizing their design, for instance) does not require regulatory engagement. But as soon as one of these tools is used to drive assessments (including diagnostic assessment) or analysis of clinical datasets, it is prudent to consult with regulators to assess any concerns around the model’s performance, risks, and validation. Depending on the FDA Center that regulates the products, there are multiple contact points to begin these discussions. Reaching out to the agency for early regulatory advice for these activities is always prudent, and will help weed out potential problems as early as possible to increase the likelihood of eventual regulatory acceptance.