Authors:Antonio Verdone, Aidan Cardall, Fardeen Siddiqui, Motaz Nashawaty, Danielle Rigau, Youngjoon Kwon, Mira Yousef, Shalin Patel, Alex Kieturakis, Eric Kim, Laura Heacock, Beatriu Reig, Yiqiu Shen

View PDF

Abstract:Objective: Radiology residents require timely, personalized feedback to develop accurate image analysis and reporting skills. Increasing clinical workload often limits attendings’ ability to provide guidance. This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings. Methods: We analyzed 5,000 resident-attending report pairs from routine practice at a multi-site U.S. health system. GPT-4o was prompted with clinical instructions to identify common errors and provide feedback. A reader study using 100 report pairs was conducted. Four attending radiologists and four residents independently reviewed each pair, determined whether predefined error types were present, and rated GPT-4o’s feedback as helpful or not. Agreement between GPT and readers was assessed using percent match. Inter-reader reliability was measured with Krippendorff’s alpha. Educational value was measured as the proportion of cases rated helpful. Results: Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings. GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types. Inter-reader reliability showed moderate variability ({\alpha} = 0.767, 0.595, 0.567), and replacing a human reader with GPT-4o did not significantly affect agreement ({\Delta} = -0.004 to 0.002). GPT’s feedback was rated helpful in most cases: 89.8%, 83.0%, and 92.0%. Discussion: ChatGPT-4o can reliably identify key educational errors. It may serve as a scalable tool to support radiology education.

Submission history

From: Yiqiu Shen [view email] [v1] Mon, 22 Sep 2025 20:51:09 UTC (2,181 KB)