This new team in Technology and Digital Solutions ( TDS), announced in March 2022, will carry out Stanford Health Care’s long-term vision to harness artificial intelligence to support and enhance every aspect of health care delivery, AI research and medical education. At its core, our efforts will center on how Stanford Health Care accelerates innovations around artificial intelligence — from development and implementation to maintenance and optimization. We will have projects advancing the scientific understanding of disease, improving the practice of clinical medicine and orchestrating the delivery of health care. As part of this effort, we will be actively searching for situations where the application of risk-stratification and proactive action can provide cost-effective, health system level benefits.

Check out the launch announcement, solving for R2O in health care, and our immediate priorities. Our initial efforts reviewing the existing recommendations for responsible adoption of AI, found that across 15 community guidelines, there are 220 items “to report’’. We also found that there is very limited guidance on how to assess fairness or utility/usefulness.

To address the gaps found, we developed a framework to estimate usefulness, and a way to assess fairness in terms of the consequences of using a model to guide care. To translate to practice, we conducted a fairness audit which required 115 person-hours across 8–10 months. To disseminate our work, we joined in the founding team for  The Coalition for Health AI, whose mission is to provide guidelines regarding an ever-evolving landscape of health AI tools to ensure high quality care, increase credibility amongst users, and meet health care needs.

The research that underpins our work is supported by the Stanford Medicine Program for AI in Healthcare and the framework guiding the development and evaluation of Fair, Useful, and Reliable Models (FURM) is below.

The four stages are: 1) problem specification and clarification, 2) development and validation of the model, 3) analysis of utility and impacts on the clinical workflow that is triggered by the model, and 4) monitoring and maintenance of the deployed model as well as evaluation of the running system comprised of the model-triggered workflow