BALTIMORE — ChatGPT’s answers to patient queries about assisted reproductive technology (ART) were generally accurate but less complete than professional society education materials, a comparative analysis showed.
On a scale of 1-5, with 5 being the best, physicians rated ChatGPT’s responses to questions about the process and associated risks of in vitro fertilization (IVF) lower than the responses found in materials from the American Society for Reproductive Medicine (ASRM) in terms of accuracy (3.98 vs 5.00), completeness (3.40 vs 4.88), safety (2.92 vs 4.88), and quality (3.38 vs 5.00), reported Nicole Fischer, MD, MPH, a reproductive endocrinology and infertility fellow at the University of Pittsburgh.
However, ChatGPT’s clarity was rated as comparable to that of ASRM documents (4.60 vs 4.63), Fischer noted at the ASRM annual meeting.
She and her team also assessed how many key concepts from ASRM’s ReproductiveFacts.org were referenced in each response; ASRM materials achieved a mean checklist adherence of 100%, while ChatGPT averaged an adherence of 94.1%.
Alarmingly, half of the ChatGPT answers contained an unsupported claim, and 4.2% contained misinformation compared with none of the ASRM responses.
“The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors,” Fischer said.
ChatGPT responses from prompts using conversational language were at an eighth grade reading level, while ChatGPT answers from generic prompts were closer to a 10th grade level and ASRM documents were roughly at a 12th grade level.
“This suggests that how a patient phrases a question can affect the readability or comprehensibility of the response that they get,” Fischer noted.
ChatGPT was also asked what the chance of IVF success was for each age from 20 to 45. For most of these ages, ChatGPT’s answers were in line with CDC and Society for Assisted Reproductive Technology (SART) data. It correctly showed a decline of birth rates with age, but it overestimated live birth rates at age 35 and underestimated rates at 37 and 40.
ChatGPT is trained on massive amounts of information to predict the most likely response. On technical or newer topics, there can be hallucinations, in which the large language model (LLM) produces a response that sounds correct but may include fabricated information or citations. This used to be more common, and the prevalence of hallucinations can vary by LLM and by version.
“The takeaway here is that topics where accuracy matters most is maybe where these tools are the weakest,” Fischer concluded.
In explaining the rationale behind this analysis, Fischer said she and her team wondered, “when patients turn to a chatbot to understand things about IVF and ART, are they getting complete and safe answers?”
To answer this question, they used the free GPT-5.6 Luna. Their first aim was to compare the LLM’s responses on common questions about the IVF process and risks to ASRM materials. Each prompt was run three times, resulting in 12 responses.
Four physicians reviewed these responses alongside two ASRM documents. The physicians rated each document on a scale of 1-5 on accuracy, completeness, clarity, safety, and quality. They also assessed checklist adherence, red flags, and grade-level readability on the Flesch-Kincaid scale.
The study’s second aim was to compare the accuracy of ChatGPT’s success estimates to CDC and SART data.
Fischer noted that the study had some limitations, including the small sample size and the use of only one version of a single LLM. Newer and specialized LLMs, including ChatGPT Health, are worth testing, she added.
Furthermore, some measures were subjective and descriptive, and there were no patient-level outcomes.
Fischer emphasized that patients are already using chatbots to make decisions and get information, and she encouraged clinicians to ask their patients about how they’re using these tools, teach them how to prompt AI to cite sources, and point patients to trusted resources.
