Towards AI-Driven Healthcare: Systematic Optimization, Linguistic Analysis, and Clinicians’ Evaluation of Large Language Models for Smoking Cessation Interventions

要旨

Creating intervention messages for smoking cessation is a labor-intensive process. Advances in Large Language Models (LLMs) offer a promising alternative for automated message generation. Two critical questions remain: 1) How to optimize LLMs to mimic human expert writing, and 2) Do LLM-generated messages meet clinical standards? We systematically examined the message generation and evaluation processes through three studies investigating prompt engineering (Study 1), decoding optimization (Study 2), and expert review (Study 3). We employed computational linguistic analysis in LLM assessment and established a comprehensive evaluation framework, incorporating automated metrics, linguistic attributes, and expert evaluations. Certified tobacco treatment specialists assessed the quality, accuracy, credibility, and persuasiveness of LLM-generated messages, using expert-written messages as the benchmark. Results indicate that larger LLMs, including ChatGPT, OPT-13B, and OPT-30B, can effectively emulate expert writing to generate well-written, accurate, and persuasive messages, thereby demonstrating the capability of LLMs in augmenting clinical practices of smoking cessation interventions.

受賞
Honorable Mention
著者
Paul Calle
University of Oklahoma, Norman, Oklahoma, United States
Ruosi Shao
The University of Oklahoma, Oklahoma City, Oklahoma, United States
Yunlong Liu
University of Oklahoma, Norman, Oklahoma, United States
Emily T. Hébert
UTHealth School of Public Health, Austin, Texas, United States
Darla Kendzor
Stephenson Cancer Center, Oklahoma City, Oklahoma, United States
Jordan Neil
OUHSC, Oklahoma City, Oklahoma, United States
Michael Businelle
Stephenson Cancer Center, Oklahoma City, Oklahoma, United States
Chongle Pan
University of Oklahoma, Norman, Oklahoma, United States
論文URL

https://doi.org/10.1145/3613904.3641965

動画

会議: CHI 2024

The ACM CHI Conference on Human Factors in Computing Systems (https://chi2024.acm.org/)

セッション: Health Ecosystems

313B
5 件の発表
2024-05-16 01:00:00
2024-05-16 02:20:00