The use of generative AI may not be helpful in reducing burnout in health care, new research suggests. Previous research indicated that increased time spent using electronic health record systems and handling administrative responsibilities has been a burden on doctors. So some people had heralded artificial intelligence as a potential solution — yet recent investigations by U.S. health systems found that large language models did not simplify clinicians’ day-to-day responsibilities.
For instance, a 2023 observational study at Brigham and Women’s Hospital in Boston, Massachusetts, examined the impact of using AI for electronic patient messaging. Researchers prompted a large language model to respond to simulated questions from cancer patients — then compared its output to responses from six board-certified radiation oncologists. Medical professionals then edited the AI-generated responses into 'clinically acceptable' answers to send to patients.
The researchers concluded that LLM-assisted results displayed a 'best-of-both-worlds scenario' — reducing physician workload while ensuring that patients get accurate information. 'These early findings … indicate the need to thoroughly evaluate LLMs in their intended clinical contexts, reflecting the precise task and level of human oversight,' the study concluded.
The research, published in the journal NEJM AI, found that all tested LLMs performed poorly on medical code querying, 'often generating codes conveying imprecise or fabricated information.' The study concluded, 'LLMs are not appropriate for use on medical coding tasks without additional research.' The study was funded by the AGA Research Foundation and National Institutes of Health .
The assumption was that generative AI drafts would lessen a physician's time spent doing these tasks — yet the results showed otherwise. 'Generative AI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length and some perceived benefits,' the study found. Researchers suggested that 'rigorous empirical tests' are needed to further assess AI’s performance and patients' experiences.