false
OasisLMS
zh-CN,en,fr,de,ja,es
Catalog
35th AM (2025) - Poster Session
Automated Audit of Ai-Generated Patient Education ...
Automated Audit of Ai-Generated Patient Education Materials A Case Study in Opioid Use Disorder
Back to course
Pdf Summary
This study addresses the challenge of ensuring clarity and readability in patient education materials related to opioid use disorder, a critical issue in Addiction Psychiatry where misinformation must be corrected quickly during clinical visits. Existing materials like leaflets and FAQs often lack systematic evaluation, potentially undermining patient understanding and treatment adherence. Traditional quality audits are manual and slow, limiting their effectiveness.<br /><br />The researchers developed an automated pipeline to quantitatively evaluate and compare pairs of texts—specifically, original opioid-safety FAQs and GPT-4o-generated responses to the same questions—using 46 objective metrics covering readability indices, the Patient Education Materials Assessment Tool (PEMAT) for understandability, and linguistic tone analysis via LIWC. They analyzed 50 opioid-related FAQs, generating 1900 data points.<br /><br />Results showed that GPT-4o-generated answers were lexically more complex, with the Coleman–Liau index increasing from 13 to 16.7 and Flesch Reading Ease dropping by 19 points, indicating denser language. However, these AI-generated responses improved structural clarity significantly, with the PEMAT metric for "information broken into clear chunks" rising from 52% to 64%. Other metrics, including tone and minor readability scores, remained statistically unchanged.<br /><br />The automated tool offers a fast, transparent means for clinicians and researchers to audit educational content, highlighting a trade-off wherein GPT-4o enhances information organization at the expense of increased complexity. Such objective assessments enable rapid content refinement, reproducible research endpoints for large language models, and data-driven quality benchmarks for policymakers.<br /><br />The study suggests future research should combine these metrics with actual patient comprehension testing, expand analyses to other topics, and refine AI prompting to balance detail with accessibility. The authors plan to release all code and anonymized data openly to foster cross-disciplinary collaboration and improve patient education in substance use treatment contexts.
Keywords
opioid use disorder
patient education materials
Addiction Psychiatry
GPT-4o
automated evaluation pipeline
readability metrics
Patient Education Materials Assessment Tool (PEMAT)
linguistic tone analysis
AI-generated responses
health communication quality
×