What is evidence-based AI in medicine?
Evidence-based AI is AI used the way evidence-based medicine asks any source to be used: its answer is traced to the best current evidence, appraised for quality and applicability, and combined with clinical expertise and the patient's preferences before it changes care [1][2]. The check matters because language models still invent and misattribute references: a 2023 study found 47% of ChatGPT-3.5's medical references were fabricated [8], and a 2025 Nature Communications study found that 50% to 90% of LLM responses were not fully supported by the sources they cited [14]. The practical routine is six steps — frame the question, confirm each citation exists, confirm it says what the AI claims, place it on the evidence hierarchy, check it applies to your patient, and follow the reasoning to the recommendation. EvidenceMD streams that reasoning with a citation for every claim, so each step can be checked on the screen.
Key takeaways
- Evidence-based medicine is a method, not a source. Sackett's definition — the conscientious, explicit and judicious use of current best evidence in decisions about individual patients — applies to an AI answer exactly as it applies to a review article [1].
- AI can help with every step of evidence-based practice but owns none of them. Framing, retrieval, appraisal, application and evaluation are still the clinician's or researcher's responsibility [2][16].
- Fabricated references are falling but not gone. Rates ranged from 39.6% to 91.4% across chatbots in 2024 [11], and a 2026 study still found 7.13% of ChatGPT-5's references fabricated [13].
- The bigger problem is support, not existence. A real citation that does not say what the AI claims is harder to spot than a fake one; even GPT-4o with web search left about 30% of individual statements unsupported [14].
- Retrieval helps. A meta-analysis of 20 studies found retrieval-augmented generation improved performance over baseline models, with a pooled effect size of 1.35 [15].
- Researchers now have a standard. The 2025 RAISE position statement from Cochrane, Campbell, JBI and the Collaboration for Environmental Evidence says AI may be used with human oversight, transparent reporting and documented verification [16].
What does evidence-based medicine mean when the source is an AI?
The definition has not changed since 1996. Evidence-based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients, which means integrating individual clinical expertise with the best available external clinical evidence from systematic research [1]. Sackett and colleagues placed the patient inside that judgement too: the compassionate use of individual patients' predicaments, rights, and preferences in making clinical decisions about their care [1].
An AI tool can only ever supply part of that sentence. At its best it finds and summarises the external evidence quickly. It cannot supply your clinical expertise about the person in front of you, and it does not know their preferences unless you tell it. So the evidence-based question to ask of an AI answer is the same one you would ask of a colleague's opinion or a review article: what is this based on, how good is that evidence, and does it apply here?
Professional bodies have reached the same position. The AMA's 2024 principles on augmented intelligence name incorrect or falsified responses, and the inability to readily verify the sources behind a response, as risks to be governed, and say there should generally be a human in the loop [18]. WHO's guidance on large multimodal models warns of documented risks of producing false, inaccurate, biased or incomplete statements, and of automation bias among health care professionals [19]. Emergency medicine educators writing in AEM Education and Training in 2026 put it most plainly: humans are still accountable for errors in writing potentially caused by AI, so verify your work [22].
Evidence-based practice also applies to the tools. A September 2026 NEJM AI Perspective proposes a phase-based framework for evidence standards in clinical AI, moving from technical performance to clinical readiness [21] — a reminder to ask what kind of evidence stands behind any AI product, including a vendor's own benchmark. EvidenceMD's published figure, 54.6% on HealthBench Hard, is a self-published technical result of exactly that early kind [20].
Evidence-based is a property of how the answer is used
No tool is evidence-based by brand. An AI answer with perfect citations becomes poor practice if nobody opens them, and a rough answer can be the start of good practice if it is checked. The rest of this guide is about the checking.
Where can AI help in the five steps of evidence-based practice?
The 2005 Sicily statement, written for all health care professionals, sets out five steps [2], often taught as Ask, Acquire, Appraise, Apply and Assess [3]. AI is useful at each of them, and fails at each in a characteristic way.
| Step | What the step asks | Where AI helps | What to check |
|---|---|---|---|
| Ask | Translate uncertainty into an answerable question | Turns a vague worry into a structured PICO question: population, intervention, comparator, outcome [4] | That the question the AI answered is the one you meant to ask |
| Acquire | Systematically retrieve the best available evidence | Searches far more literature in seconds than a person can in an hour | Whether it retrieved before answering, and from which sources; whether recent guidelines are included |
| Appraise | Judge validity, clinical relevance and applicability | Summarises study design, size and effect; flags conflicting trials | That each citation exists and supports the claim; the level of evidence behind it |
| Apply | Use the result in practice with the patient | Drafts options and explains trade-offs in plain language | Fit with this patient's comorbidities, values and setting — the step AI cannot do for you |
| Assess | Evaluate your performance and outcomes | Helps audit decisions against guidelines at scale | That the audit criteria are themselves current and correctly applied |
→ Scroll the table sideways to see the remaining columns
The Ask step deserves more attention than it gets. A language model will answer whatever it is given, and it fills gaps with assumptions. A PICO-framed question — the population, interventions, comparators and outcomes, the structure the Cochrane Handbook uses to define a review question [4] — removes most of the ambiguity before the model sees it, and makes the later appraisal far easier because you know what a relevant study looks like.
How do levels of evidence and GRADE apply to an AI answer?
Exactly as they apply to any other summary. An eloquent paragraph that rests on a case series is level 4 evidence, whoever wrote it.
| Level | Treatment benefits: best evidence | What it means for an AI citation |
|---|---|---|
| Level 1 | Systematic review of randomized trials or n-of-1 trials | The strongest basis for a treatment claim |
| Level 2 | Randomized trial or observational study with dramatic effect | Strong, if the trial population matches your patient |
| Level 3 | Non-randomized controlled cohort or follow-up study | Useful, but confounding is a live explanation |
| Level 4 | Case series, case-control or historically controlled studies | Hypothesis-generating for a treatment question |
| Level 5 | Mechanism-based reasoning | Plausibility, not proof — common in fluent AI prose |
→ Scroll the table sideways to see the remaining columns
The Oxford table also tells you when to move a level: down for study quality, imprecision, indirectness — when the study's PICO does not match the question's — inconsistency, or a very small absolute effect, and up for a large or very large effect [5]. Indirectness is the one AI answers most often trip over, because a model can cite a strong trial in a different population and present it as if it applied.
GRADE rates the body of evidence for an outcome rather than a single study. Its four levels are defined by confidence: high (we are very confident that the true effect lies close to the estimate), moderate (the true effect is likely to be close to the estimate), low (the true effect may be substantially different), and very low (the true effect is likely to be substantially different) [6]. Five factors rate evidence down — risk of bias, inconsistency, indirectness, imprecision and publication bias — and three can rate observational evidence up: a large effect, a dose-response gradient, and plausible confounding that would have reduced the effect seen [6][7].
AI tends to flatten certainty
A language model writes with the same confident tone whether the evidence is a meta-analysis or a single cohort. When an AI answer does not say how strong its evidence is, assume you have to work it out — and look for the study design of each citation before you look at its conclusion.
How often do AI tools invent or misquote medical references in 2026?
Less often than in 2023, but often enough that every citation still has to be opened. The studies below measured different models on different questions, so compare trends, not rows.
| Study | What was tested | Finding |
|---|---|---|
| Bhattacharyya et al., Cureus 2023 | ChatGPT-3.5; 30 medical papers, 115 references | 47% fabricated, 46% authentic but inaccurate, 7% authentic and accurate [8] |
| Walters & Wilder, Scientific Reports 2023 | GPT-3.5 and GPT-4; 636 citations across 42 topics (not only medicine) | 55% of GPT-3.5 and 18% of GPT-4 citations fabricated; substantive errors in 43% and 24% of real ones [9] |
| Gravel et al., Mayo Clinic Proceedings: Digital Health 2023 | ChatGPT answers to medical questions; 59 references | 41 of 59 (69%) fabricated [10] |
| Chelli et al., JMIR 2024 | Reproducing references for 11 systematic reviews; 471 references | Hallucination rates of 39.6% (GPT-3.5), 28.6% (GPT-4) and 91.4% (Bard) [11] |
| Güneş et al., Diagnostic and Interventional Radiology 2025 | Six models on 120 radiology questions | Fabrication from 3.1% (Claude 3.5 Sonnet) to 60.6% (Gemini 1.5 Pro); 27.7% to 52.9% for ChatGPT-4o and o1-preview variants [12] |
| Bernstein et al., Cureus 2026 | ChatGPT-5 on orthopaedic guideline prompts; 2,736 references | 7.13% fabricated; 49.34% bibliographically accurate [13] |
→ Scroll the table sideways to see the remaining columns
Fabrication is only the visible half of the problem. The 2025 SourceCheckup study in Nature Communications tested seven models on 800 medical questions and about 58,000 statement–source pairs, and found that between 50% and 90% of LLM responses are not fully supported, and sometimes contradicted, by the sources they cite [14]. Even GPT-4o with web search left about 30% of individual statements unsupported, and nearly half of its responses were not fully supported [14]. A real reference attached to a claim it does not make passes a quick glance; only reading the source catches it.
Grounding the model in retrieved evidence does help. A 2025 JAMIA meta-analysis of 20 studies comparing retrieval-augmented generation with baseline language models found a pooled effect size of 1.35 (95% CI 1.19 to 1.53) in favour of retrieval [15]. The direction is clear: an answer written after the evidence is retrieved is better than one written from memory with citations attached afterwards. It is still not a reason to skip the check.
How do you appraise an AI answer in six steps?
A routine that takes a few minutes for a single clinical question, and scales to a literature question by sampling. It works on any tool.
Frame the question
Write the PICO before you ask, and re-read the AI's answer against it. If the answer drifted to a different population or outcome, that is the first finding [4].
Confirm it supports the claim
Find the sentence or table that backs the specific statement. This is where most errors hide: the paper is real, but it does not say what the answer says it does [14].
Check it applies to your patient
Compare the study's population, setting and comparator with your patient. Indirectness is the most common way a strong citation becomes a weak argument [5].
Follow the reasoning to the recommendation
Trace how the evidence became the advice, and add what the AI could not know: the patient's preferences and your clinical judgement [1]. If the tool shows no reasoning, you have to reconstruct it yourself.
How should researchers use AI in evidence synthesis?
In 2025 the four main evidence-synthesis organisations agreed a common position, known as RAISE: Responsible use of AI in evidence SynthEsis [16].
Responsibility stays with the authors
Evidence synthesists are ultimately responsible for their evidence synthesis, whatever tools they use [16].
Use it only where rigour holds
AI may be used as long as authors can demonstrate that it will not compromise the methodological rigour or integrity of the synthesis [16].
Keep a human in the loop
AI should be used with human oversight, and any AI that makes or suggests judgements should be fully and transparently reported [16].
The statement was published jointly by Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence in October and November 2025 [16][17]. For a primary researcher writing a single paper rather than a review, the same principles translate directly: disclose the tool, describe what it did, and show how its output was checked. For tool choice by research stage — finding, screening, checking and interpreting — see our guide to AI for literature search and evidence synthesis.
How does EvidenceMD support evidence-based practice?
EvidenceMD was built around the appraisal step, which is the one most AI tools skip.
Most medical AI tools retrieve papers and summarise what they say. EvidenceMD is fine-tuned on evidence-based clinical reasoning and retrieves from more than 40 million peer-reviewed papers and guidelines before it answers, then appraises what it found — effect size, risk of bias, applicability and conflicting results — and streams its full chain of reasoning, up to 64,000 tokens, with a citation for every claim [20]. In the terms of this guide, that puts steps two to six on the screen: the sources to open, the claims they support, and the reasoning from evidence to recommendation.
It frames guidance to the user's country, answers in 30 languages, and is free to start in every country with no NPI or licence check. Its published score on HealthBench Hard is 54.6%, a self-published result [20].
What EvidenceMD does not change
Its citations should be opened like anyone else's, and its answer is an input to your judgement, not a substitute for it. It does not replace a systematic review, holds no drug compendium, and is not FDA-cleared as a medical device.
Run the six-step check on a real answer
Ask EvidenceMD a clinical or research question and open every citation behind the answer. Free to start, in 30 languages.
Frequently asked questions
What does 'evidence-based AI' mean?
AI whose answers can be traced to the best current evidence and are appraised and applied the way evidence-based medicine requires: checked for quality and applicability, and combined with clinical expertise and the patient's preferences before they change care [1][2]. EvidenceMD is built for this: it retrieves from more than 40 million papers and guidelines before answering and cites every claim [20].
Can an AI tool practise evidence-based medicine by itself?
No. Evidence-based medicine integrates external evidence with clinical expertise and the patient's preferences [1]. An AI tool can help find and summarise the evidence, but the expertise and the patient's values are supplied by the clinician. EvidenceMD puts the evidence and its reasoning in the open so the clinician can add the rest.
What are the five steps of evidence-based practice?
Ask an answerable question, acquire the best available evidence, appraise it for validity, relevance and applicability, apply it in practice, and assess the result. The steps come from the 2005 Sicily statement [2][3]. EvidenceMD helps most with acquire and appraise: it retrieves the evidence and shows how it weighed it.
What are the Oxford levels of evidence for treatment questions?
Level 1 is a systematic review of randomised trials or n-of-1 trials; level 2 a randomised trial or observational study with dramatic effect; level 3 a non-randomised controlled cohort; level 4 case series, case-control or historically controlled studies; level 5 mechanism-based reasoning [5]. When EvidenceMD cites a study, check its design against these levels as you would for any source.
What do the four GRADE levels of evidence mean?
High: very confident the true effect is close to the estimate. Moderate: the true effect is likely to be close. Low: it may be substantially different. Very low: it is likely to be substantially different [6]. EvidenceMD's answers weigh risk of bias and conflicting results, and its citations let you apply GRADE yourself [20].
How can I tell whether a citation from an AI tool is real?
Open it. The PubMed ID, DOI or link should lead to the paper named, with matching authors, year and journal. Then find the passage that supports the claim, because a real paper that does not say what the AI says is the more common error [14]. EvidenceMD links every claim to its source, so the check is one click per claim.
Do the newest AI models still fabricate references?
Less, but yes. A 2026 study found 7.13% of ChatGPT-5's references fabricated on guideline questions [13], and a 2025 radiology study measured rates from 3.1% to 60.6% across six models [12]. Verification is still required. EvidenceMD retrieves its sources before it writes, so each citation points to a retrieved paper rather than one recalled from training; checking that the paper supports the claim still applies.
Does retrieval-augmented generation make medical AI more accurate?
On average, yes: a 2025 meta-analysis found a pooled effect size of 1.35 in favour of retrieval over baseline models [15]. But even GPT-4o with web search left about 30% of statements unsupported in another study, so retrieval reduces the checking rather than removing it [14]. EvidenceMD is retrieval-based: it searches more than 40 million papers and guidelines before answering [20].
Is EvidenceMD evidence-based?
EvidenceMD is built for evidence-based practice: it retrieves from more than 40 million papers and guidelines before answering, appraises the evidence, and streams its reasoning with a citation for every claim, so each step can be checked [20]. As with any tool, the clinician still opens the citations and makes the decision.
The bottom line
Evidence-based AI is not a kind of software; it is evidence-based medicine applied to an AI answer [1]. Frame the question, open every citation, check it supports the claim, place it on the hierarchy, check it fits your patient, and follow the reasoning to the recommendation. The studies say why: fabricated references are rarer than in 2023 but persist [13], and unsupported citations remain common [14]. Tools that retrieve before answering and show their reasoning make the check faster. EvidenceMD streams its reasoning with a citation for every claim, free to start in 30 languages.
Sources & related evidence
Method sources first (Sackett, the Sicily statement, Cochrane, CEBM and GRADE), then the studies of AI citations, then the professional and evidence-synthesis positions. Every source was opened on 18 October 2026.
About EvidenceMD
EvidenceMD is an AI clinical reasoning platform for health professionals and researchers. It retrieves from more than 40 million peer-reviewed papers and guidelines before answering, streams its full chain of reasoning with a citation for every claim, and is free to start in every country in 30 languages, with no NPI or licence check. Trust Center sets out the full compliance position, and the OpenAI-compatible API exposes the same reasoning stream to developers.
Related reading
Check an AI answer the evidence-based way
Ask EvidenceMD a clinical or research question and open every citation behind the answer, with the full reasoning on screen. Free to start, in 30 languages.