文章：

评估隐私保护型大型语言模型在计算脊柱肿瘤不稳定评分（SINS）中的准确性

Evaluating the Accuracy of Privacy-Preserving Large Language Models in Calculating the Spinal Instability Neoplastic Score (SINS)

原文发布日期：20 June 2025

DOI: 10.3390/cancers17132073

类型: Article

开放获取: 是

英文摘要：

Background:Large language models (LLMs) have emerged as powerful tools in healthcare. In diagnostic radiology, LLMs can assist in the computation of the Spine Instability Neoplastic Score (SINS), which is a critical tool for assessing spinal metastases. However, the accuracy of LLMs in calculating the SINS based on radiological reports remains underexplored.Objective:This study evaluates the accuracy of two institutional privacy-preserving LLMs—Claude 3.5 and Llama 3.1—in computing the SINS from radiology reports and electronic medical records, comparing their performance against clinician readers.Methods:A retrospective analysis was conducted on 124 radiology reports from patients with spinal metastases. Three expert readers established a reference standard for the SINS calculation. Two orthopaedic surgery residents and two LLMs (Claude 3.5 and Llama 3.1) independently calculated the SINS. The intraclass correlation coefficient (ICC) was used to measure the inter-rater agreement for the total SINS, while Gwet’s Kappa was used to measure the inter-rater agreement for the individual SINS components.Results:Both LLMs and clinicians demonstrated almost perfect agreement with the reference standard for the total SINS. Between the two LLMs, Claude 3.5 (ICC = 0.984) outperformed Llama 3.1 (ICC = 0.829). Claude 3.5 was also comparable to the clinician readers with ICCs of 0.926 and 0.986, exhibiting a near-perfect agreement across all individual SINS components [0.919–0.990].Conclusions:Claude 3.5 demonstrated high accuracy in calculating the SINS and may serve as a valuable adjunct in clinical workflows, potentially reducing clinician workload while maintaining diagnostic reliability. However, variations in LLM performance highlight the need for further validation and optimisation before clinical integration.

摘要翻译： 

背景：大型语言模型已成为医疗保健领域的重要工具。在放射诊断学中，大型语言模型可辅助计算脊柱肿瘤不稳定评分，这是评估脊柱转移瘤的关键工具。然而，基于放射学报告计算脊柱肿瘤不稳定评分时，大型语言模型的准确性仍有待深入研究。 目的：本研究评估两种机构隐私保护型大型语言模型（Claude 3.5和Llama 3.1）根据放射学报告和电子病历计算脊柱肿瘤不稳定评分的准确性，并将其表现与临床医师阅片者进行比较。 方法：对124份脊柱转移瘤患者的放射学报告进行回顾性分析。三位专家阅片者建立了脊柱肿瘤不稳定评分计算的参考标准。两名骨科住院医师和两种大型语言模型独立计算脊柱肿瘤不稳定评分。采用组内相关系数衡量总体脊柱肿瘤不稳定评分评估者间一致性，使用Gwet's Kappa系数衡量各脊柱肿瘤不稳定评分组成要素的评估者间一致性。 结果：大型语言模型与临床医师在总体脊柱肿瘤不稳定评分方面均与参考标准呈现几乎完全一致。两种大型语言模型中，Claude 3.5的表现优于Llama 3.1。Claude 3.5与临床医师阅片者的表现相当，在所有脊柱肿瘤不稳定评分组成要素中均展现出近乎完美的一致性。 结论：Claude 3.5在计算脊柱肿瘤不稳定评分方面表现出较高准确性，可作为临床工作流程中有价值的辅助工具，在保持诊断可靠性的同时可能减轻临床医师工作负担。然而，大型语言模型表现的差异性提示在临床整合前仍需进一步验证和优化。

原文链接：

Evaluating the Accuracy of Privacy-Preserving Large Language Models in Calculating the Spinal Instability Neoplastic Score (SINS)