- Epic launches open-source tool for healthcare organizations to validate and monitor AI models.
- The software suite, available on GitHub, enables integration with EHR systems and supports validation of both Epic-developed and external AI models.
- Epic aims to foster AI best practices and ensure equity by providing a platform for standards and practices integration.
- The tool automates data collection and mapping, offering real-time metrics and analysis, with plans for expansion to include more AI models.
- Health AI Partnership (HAIP) plans to utilize Epic’s tool for local AI model validation, marking a collaborative effort in advancing responsible AI use in healthcare.
- Despite concerns about potential conflicts of interest, Epic emphasizes the tool’s neutrality and commitment to “doing good.”
Main AI News:
Epic unveiled a groundbreaking open-source tool on Wednesday, designed to empower healthcare institutions in assessing and overseeing artificial intelligence (AI) models. The AI validation software suite, as disclosed by Corey Miller, Vice President of Research and Development at Epic to Fierce Healthcare, is freely accessible on GitHub, enabling health systems to seamlessly integrate it into their electronic health record (EHR) systems.
Healthcare entities can leverage this tool to validate AI models seamlessly integrated with EHR systems. These models span those developed by Epic itself and those created by external organizations. Epic executives emphasize that as AI best practices evolve, this open-source framework will facilitate the incorporation of standards and practices alongside AI validation capabilities.
This marks Epic’s inaugural venture into open-source tools, a move underscored by Miller’s assertion, “By publishing on GitHub, it’s truly available to everyone; it’s not behind any lock or key that we control. We’re excited to dive into this world.” The tool’s aim to ensure health AI equity aligns with its public availability and openness to global contributors, Miller affirmed.
In early April, Epic announced its intention to release an AI validation software suite, enabling healthcare institutions to evaluate AI models locally and monitor them over time. Seth Hain, Senior Vice President of R&D at Epic, elucidated that the suite, labeled an “AI trust and assurance software suite,” automates data collection and mapping, offering near real-time metrics and analysis on AI models. The automation streamlines processes, eliminating the arduous task of data mapping for healthcare organization data scientists, Hain emphasized.
The pivotal objective is to enable AI testing and validation at a local level, while ensuring scalable ongoing monitoring, Hain reiterated. Although the current version of the open-source tool doesn’t validate the performance of generative AI models, Miller revealed Epic’s plans to broaden its scope to encompass a wider array of AI models in the future.
The Health AI Partnership (HAIP), comprising entities like Duke Health, Mayo Clinic, and Kaiser Permanente, intends to utilize Epic’s AI trust and assurance software suite for local AI model validation, according to Miller. Furthermore, Epic plans to collaborate with HAIP and the University of Wisconsin to explore predictive model usage using the validation tool.
“We will be able to learn quickly how the tool needs to evolve and develop,” Miller asserted. He expressed excitement regarding HAIP’s decision to leverage the tool, underscoring its alignment with the tool’s intended purpose of fostering responsible AI utilization to enhance the healthcare system comprehensively.
Healthcare organizations have embraced predictive AI models and machine learning for nearly a decade. However, the advent of large language models (LLMs) and generative AI tools presents novel challenges. While health systems rapidly adopt LLMs and generative AI for tasks like medical record summarization and clinical notetaking automation, they grapple with validating these models to ensure accuracy, performance, and safety.
Hain highlighted in April that the AI software suite boasts intuitive reporting dashboards, automatically updated to provide analyses stratified by various demographics. Moreover, the software incorporates a standardized monitoring template and data schema, simplifying its extension to new AI models in the future.
The open-source tool empowers healthcare organizations to conduct AI validation within their EHR systems, tailored to their patient populations and workflows, Epic executives emphasized. Miller elucidated, “It will pull in downstream outcomes and interventions. You can slice and dice the data across different patient cohorts.”
He further noted, “The tool is targeted primarily at data scientists and clinicians, but we hope that it will be easy enough to understand that a clinician without a data science background can also dig in and learn.”
Despite concerns raised by some stakeholders about potential conflicts of interest stemming from Epic’s role as a health IT vendor, Miller emphasized the tool’s neutrality. “This is completely agnostic to any of our code,” he asserted. “At Epic, the motto we like to live by is ‘Do good, have fun, make money,’ and we feel like this one really fits in that ‘Do good’ part.“
Conclusion:
Epic’s introduction of an open-source AI validation tool signifies a pivotal shift in the healthcare market. By providing accessible and standardized tools for AI validation, Epic not only empowers healthcare organizations to ensure the accuracy and safety of AI models but also fosters collaboration and innovation in the field. This move underscores the increasing importance of transparency and accountability in AI-driven healthcare solutions, ultimately driving the industry towards greater efficiency and effectiveness in patient care.
