Jie Yuan,1,* Xiao Chen,2,* Chun Yang,3 JianYou Chen,3 PengFei Han,3 YuHong Zhang,2 YuXia Zhang2 

1School of Nursing, Fudan University, Shanghai, 200032, People’s Republic of China; 2Department of Nursing, Zhongshan Hospital of Fudan University, Shanghai, 200032, People’s Republic of China; 3Department of Information, Zhongshan Hospital of Fudan University, Shanghai, 200032, People’s Republic of China

*These authors contributed equally to this work

Correspondence: YuXia Zhang, Department of Nursing, Zhongshan Hospital of Fudan University, Room 501, Building 5, Fenglin Road No. 180, Xuhui District, Shanghai, 200032, People’s Republic of China, Tel +86 13816881925, Email zhang.yx@aliyun.com YuHong Zhang, Department of Nursing, Zhongshan Hospital of Fudan University, Room 501, Building 5, Fenglin Road No. 180, Xuhui District, Shanghai, 200032, People’s Republic of China, Tel +86 13816881925, Email zhang.yuhong@zs-hospital.sh.cn

Background: Open-ended questions in patient experience surveys provide a valuable opportunity for people to express and discuss their authentic opinions. The analysis of free-text comments can add value to quantitative measures by offering information which matters most to patients and by providing detailed descriptions of the service issues that closed-ended items may not cover.
Objective: To extract useful information from large amounts of free-text patient experience comments and to explore differences in patient satisfaction and loyalty between patients who provided negative comments and those who did not.
Methods: We collected free-text comments on a broad, open-ended question in a cross-sectional patient satisfaction survey. We adopted a mixed-methods approach involving a literature review, human annotation, and natural language processing technique to analyze free-text comments. The associations of patient satisfaction and loyalty scores with the occurrence of certain patient comments were tested via logistic regression analysis.
Results: In total, 28054 free-text comments were collected (comment rate: 72.67%). The accuracy of the machine learning approach and the deep learning approach for topic modeling and sentiment analysis was 0.98 and 0.91 respectively, indicating a satisfactory prediction. Participants tended to leave positive comments (69.0%, 19356/28054). There were 22 patient experience themes discussed in the open-ended comments. The regression analysis showed that the occurrence of negative comments about “humanity of care”, “information, communication, and education”, “sense of responsibility of staff”, “technical competence”, “responding to requests”, and “continuity of care” was significantly associated with a worse patient satisfaction and loyalty, while the occurrence of negative comments about other aspects of healthcare services had no impact on patient satisfaction and loyalty.
Conclusion: The results of this study highlight the interpersonal and functional aspects of care, especially the interpersonal aspects, which are often the “moment of truth” during a service encounter when patients critically evaluate hospital services.

Keywords: patient experience, natural language processing, sentiment analysis, topic modelling, free-text comments