FDA's Collaborative Approach to Promoting Responsible AI in Medical Products
Author: The Alliance for Artificial Intelligence in Healthcare
With Contributions from:
Hin Au, JD, Associate, Wilson Sonsini Goodrich & Rosati
Esther W.B. Bleicher, JD, MPH, General Counsel and Chief Compliance Officer, Hello Heart, Board, AAIH
Meredith Brown-Tuttle, RAC, FRAPS, Chief Regulatory Strategist, Regulatorium
Elaine Hamm, PhD, Executive Director, AAIH
Paul Howard, PhD, Senior Director, Public Policy, Amicus Therapeutics
Rafael Rosengarten, PhD, Chief Executive Officer, Genialis, Board, AAIH
Eva F. Yin, PhD, MPH, JD, Partner, Wilson Sonsini Goodrich & Rosati
Chandrasekhar Sharma, MBA, Senior Vice President, Course5i
Alex Yue, MBA Senior Director, Quality, Hello Heart
James Zanewicz, JD, LLM, Chief Strategy Officer Tulane School of Medicine, Treasurer AAIH
Executive Summary
The Alliance for Artificial Intelligence in Healthcare commends the recent joint publication by the FDA's Centers for Biologics Evaluation and Research (CBER), Drug Evaluation and Research (CDER), Devices and Radiological Health (CDRH), and the Office of Combination Products (OCP). This article outlined four key recommended areas of focus:
1. Fostering collaboration to safeguard public health
2. Advancing regulatory approaches that support innovation
3. Promoting standards, guidelines, best practices, and tools
4. Supporting research related to evaluating and monitoring AI performance
This article underscores the transformative potential of AI in healthcare but also highlights the crucial steps needed to ensure we safely harness this potential. This is one of the key missions of the Alliance for AI in Healthcare.
Content
A key component to ensuring success in each of the above-stated recommendations will be collaboration across the entire spectrum of healthcare. The FDA recognizes the importance of engaging with a range of stakeholders, including patient groups and caregivers, payors and provider systems, drug and diagnostic developers, med- and health-tech innovators, as well as academics and other interested parties. By soliciting input and promoting educational initiatives, the Agency aims to cultivate a patient-centered, collaborative regulatory approach that emphasizes transparency, accountability, and health equity. Most notably, the Agency has met with quite a few AI developers and innovators through the “Critical Path Innovation” meeting process. Advances in AI are bringing new innovators to medical products, swelling the ranks in addition to incumbents with existing relationships with the FDA. AAIH encourages the FDA to include these newcomer organizations with less regulatory experience as part of its stakeholder engagement to develop a regulatory approach that is fully responsive to the breadth of innovations AI can generate for the healthcare field.
Furthermore, the FDA is committed to developing policies that provide regulatory clarity and predictability, both supporting regulatory science efforts to evaluate AI algorithms and issuing guidance on various aspects of AI in medical product development and use. AAIH applauds the FDA’s efforts to provide regulatory certainty, along with the FDA’s readiness to revise policies over time as innovation continues to evolve.
Promoting the development of harmonized standards, guidelines, best practices, and tools is another key focus area. The Agency plans to refine considerations for evaluating the safe, responsible, and ethical use of AI, identify best practices for long-term safety monitoring, and develop frameworks for quality assurance of AI-enabled tools and systems.
Finally, the FDA intends to support demonstration projects that identify potential sources of bias in the AI development life cycle, address health inequities associated with AI use, and monitor the ongoing performance and reliability of AI tools in medical product development.
Best practices for the safe, effective, and ethical use of AI are critical to the long-term success of AI-enabled FDA-regulated products. AAIH recognizes the proactive steps the FDA has taken to learn from AI developers to understand better the unique benefits and risks of AI, as well as to identify potential solutions. Many companies and academic research organizations have already taken on projects to address bias, inequalities, hallucinations, and other quality standards. Here the FDA does not need to reinvent the wheel, but rather should strive to consider the best practices derived from ongoing work. We also encourage the FDA to continue to stay abreast of advances in responsible AI, both in areas of their responsibility, such as medicine, as well as in other fields outside their direct purview, like law enforcement and banking. By learning from other areas where the deployment of AI-related tools has raised serious concerns, the FDA may be able to avoid potential pitfalls and adapt existing solutions.
Overall, this FDA publication provides a strong step toward advancing responsible AI innovation in healthcare. By taking a proactive, collaborative, and multifaceted approach, the FDA aims to safeguard public health while fostering responsible and ethical innovation in the application of AI to medical products. This effort is aligned with the Agency's mission to ensure patient access to safe and effective medical products and highlights its dedication to facilitating innovation in the healthcare sector. As a key industry convening organization, the AAIH urges all stakeholders to actively collaborate and engage as we work together to address not just the numerous challenges - but also incredible opportunities - that adopting AI presents to improve patient care and promote health equity. To assist in these commendable endeavors, the AAIH is working on a series of papers on these topics.
The commitment to supporting research that evaluates and monitors AI performance in medical product development is crucial for ensuring equitable and effective healthcare delivery. This initiative by the FDA underscores the importance of addressing potential biases and health inequities arising from AI technologies.
In this context, AAIH would like to highlight the critical need for additional datasets that capture Social Determinants of Health (SDOH) and caregiver perspectives to inform AI development and address biases effectively in medical product development. Doing so will appropriately emphasize the importance of leveraging comprehensive datasets to drive innovation and promote health equity through AI technologies in healthcare. One of the fundamental risks for AI is the bias that could result in inequality in healthcare, as AI and machine learning commonly make predictions and outputs via correlation using the dataset they were trained on, which may be biased, to begin with. The traditional wisdom of “correlation does not imply causation” is still highly applicable. The conclusions we draw by using AI and machine learning algorithms are reached with varying levels of assumptions (on various diversity - or lack thereof - of data sets), and we must ensure these assumptions are well-characterized and well-justified.
One example of this is a risk-prediction software applied to millions of individuals that failed to identify black American patients with more chronic illnesses as needing extra care, when compared to their white counterparts. The flaw stemmed from the algorithm's reliance on healthcare spending data, which were distorted by systemic disparities in healthcare access and utilization between racial groups. This introduced racial biases in the algorithm’s recommendations. Researchers eventually identified the issue and collaborated with the software manufacturer to address the bias, highlighting the importance of ongoing scrutiny and refinement in AI applications to mitigate unintended consequences and ensure equitable healthcare outcomes. (source: Stanford Social Innovation Review)
Incorporating additional SDOH datasets into AI development projects can help identify points of bias introduction in the AI life cycle more effectively. By analyzing how SDOH factors interact with AI algorithms, researchers will learn to better understand and address biases through robust risk management strategies. Incorporating such an approach is essential for enhancing the fairness and effectiveness of AI-driven medical products.
Caregivers, as integral parts of healthcare, offer invaluable insights into patient experiences and needs. Their unique perspective is crucial in understanding the impact of AI tools on medical product development. It's vital to gather datasets that encompass the diverse patient demographics and caregiving scenarios to ensure this perspective is properly integrated. Examples of potential information sources/datasets include those available from the Centers for Disease Control and Prevention (CDC) [https://www.cdc.gov/aging/caregiving/caregiver-brief.html] and the National Health and Aging Trends Study (NHATS) [https://www.nhats.org/researcher/nsoc]. These can serve as foundational resources for designing and deploying AI technologies that are inclusive and reflective of the communities they support.
The FDA’s initiative to support research on AI performance evaluation and monitoring reflects a commitment to advancing healthcare innovation responsibly. To fully realize AI's potential in addressing health disparities and promoting equity, we must advocate for collecting and utilizing diverse datasets encompassing SDOH and caregiver perspectives. Integrating these datasets into AI development processes can pave the way for more inclusive, effective, and equitable healthcare solutions.
In addition to the FDA’s collaborative approach within the Agency, the FDA is also collaborating with other federal agencies, such as the United States Patent and Trademark Office (USPTO) and the National Institute of Standards and Technology (NIST) to explore new considerations raised by AI. For example, the FDA and the USPTO currently have collaboration initiatives that are aimed at protecting and promoting innovations while advancing marketplace competition that can help to lower drug prices for Americans. Both agencies have hosted joint events over the past few years encouraging the public to provide comments on areas for collaboration and engagement between the agencies. In February 2024, the USPTO issued guidance that discusses inventorship considerations for AI-assisted inventions. Unlike traditional patents for drugs and medical devices, AI adds a layer of complexity and requires special considerations with respect to the patenting (or the patentability) of AI healthcare innovations.
In view of the different regulatory frameworks and considerations that apply to the use of AI in medical products, it is becoming increasingly important for companies to coordinate their product development plans with their clinical data plans, FDA regulatory strategy, data privacy, and security strategy, and intellectual property strategies, and to incorporate safeguards against potential biases and hallucinations. Companies in the healthcare space are advised to monitor new AI policies and guidelines issued by the FDA, USPTO, and other government agencies, such as the Federal Trade Commission.
Conclusion
AAIH will work with regulatory and legal experts to publish more in-depth articles on various topics in the AI space, including topics on AI in drug manufacturing and development, AI in medical devices, intellectual property considerations, and data privacy considerations. Stay tuned for more!