Dr. Karandeep Singh Doesn’t Trust AI Blindly. That’s Exactly Why UC San Diego Health Trusts Him With It

Key Podcast Moments

• Why the nephrologist who exposed a flawed sepsis model is now the one deciding what AI gets deployed
• The model card said it predicted sepsis before it happened. His research said otherwise
• Why generative AI feels transformative and predictive AI barely feels like anything at all
• The AI phone call that now reaches 100% of a hospital's colonoscopy patients, without cutting a single nurse
• What a Chief Health AI Officer actually does across evaluation, governance, and implementation
• His case for regulatory sandboxes, and the policy gap keeping AI translation out of patient care

Dr. Karandeep Singh is a practicing nephrologist, an informaticist, and the inaugural Chief Health AI Officer at UC San Diego Health, where he also leads the Joan and Irwin Jacobs Center for Health Innovation. He learned to code at age eight, has worked across 14 programming languages over his career, and built tools to solve his own frustrations at every stage of training, from acid-base calculators in medical school to a JavaScript admissions tool during residency. He didn't discover biomedical informatics as a formal field until his nephrology fellowship, by which point he'd already been practicing it for years without a name for it.

Karandeep's reputation in AI circles was built on a piece of research most people never get credit for: proving that a widely deployed model wasn't doing what it claimed. At the University of Michigan, frontline staff on two hospital units told him Epic's sepsis model was behaving strangely. Rather than dismiss the feedback, he investigated, and found the tool was strong at flagging sepsis after the fact but not, as its own model card claimed, at predicting it before it happened. Mentors told him validating someone else's model wouldn't get him published. He did the work anyway, folded it into formal AI governance, and later repeated the evaluation at other health systems.

That same instinct, question the tool before you deploy the tool, now defines his day job. Karandeep and Graham Walker dig into why predictive AI creates work that's almost impossible to feel, even when it's saving lives, while generative AI's smaller, more frequent time savings earn trust faster. They walk through a real case study of UC San Diego Health using agentic AI to automate pre-colonoscopy prep calls, a shift that let the system reach every English-speaking patient scheduled for the procedure while giving nurses back time for actual clinical work. And they get into what it actually means to hold a job that barely existed five years ago, and what happens when a health system hands that job to someone who doesn't take AI's word for it.

Thank you to our wonderful sponsors for supporting the podcast:

Evidently - Leading AI-powered clinical data intelligence https://evidently.com/

Top 4 Takeaways

Karandeep isn't arguing for caution as an end in itself. He's arguing that the fastest path to AI actually helping patients runs through the same rigor that exposed the sepsis model's flaws in the first place. Here's what that looks like in practice.

• The Model Card Said One Thing. The Data Said Another.
  Karandeep's research on Epic's sepsis model didn't start as an academic exercise. Frontline nurses on two units told him the tool felt off, and instead of trusting the vendor's numbers over their instincts, he investigated. What he found was a model that excelled at identifying sepsis cases after they'd already developed, not before, despite a model card that claimed early prediction. It's the kind of gap that matters enormously once a tool is deployed at national scale, and it became the foundation for how he thinks about AI governance today.
• Predictive AI Creates Work That's Almost Impossible to Feel
  Karandeep draws a sharp distinction between how physicians experience generative AI versus predictive AI. Generative tools save small, frequent chunks of time, twenty minutes here, an hour there, and doctors notice. Predictive tools like sepsis or deterioration models operate on a different scale entirely: the best published literature shows roughly a 2% absolute risk reduction in mortality, a number too small for any individual clinician to consciously register. The result is that physicians remember every extra alert the model generates, but almost never the rare case where it was the reason a patient survived.
• Automating a Phone Call Still Requires Keeping Humans at the Center
  When UC San Diego Health identified pre-colonoscopy prep calls as ripe for automation, the goal was never to remove nurses from the process. The old system relied on floor nurses working through a lengthy script covering everything from parking to medication holds, often eating up to half an hour per patient, with real clinical consequences when calls were rushed or missed. The AI system now handles the script directly with patients and escalates to a nurse only when something requires real judgment, and every call is reviewed weekly against patient feedback. The health system now reaches all of its English-speaking colonoscopy patients, up from a fraction before, and nursing teams report preferring the new workflow because their time is spent on nursing, not logistics.
• Regulatory Sandboxes Are How Medicine Moves Forward.
  Karandeep sees a real tension in the current AI literature: studies showing near-superhuman diagnostic performance sit alongside studies documenting hallucination, and he argues those two findings are too often treated as an equal coin flip rather than fundamentally different risks to weigh. His answer isn't to greenlight AI-based care broadly. It's to build structured, consented sandboxes that let health systems test new models of care, starting with lower-risk use cases and moving carefully toward harder ones, in places where the shortage of care is already the bigger risk. He points to a concrete policy gap as an example: AI-based medical translation is currently barred in situations requiring accuracy, even though patients already use unsupervised consumer translation tools as a workaround today.

Resources and Links

• UC San Diego Health, Jacobs Center for Health Innovation
• UC San Diego Health AI
• Connect with Dr. Karandeep Singh on LinkedIn