Representing narrative evidence as clinical evidence logic statements

Ronilda Lacson; Mahsa Eskian; Laila Cochon; Isha Gujrathi; Andro Licaros; Anna Zhao; Nicole Vetrano; Louise Schneider; Ali Raja; Ramin Khorasani

doi:10.1093/jamiaopen/ooac024

Representing narrative evidence as clinical evidence logic statements

JAMIA Open. 2022 Apr 11;5(2):ooac024. doi: 10.1093/jamiaopen/ooac024. eCollection 2022 Jul.

Authors

Ronilda Lacson^{1

2}, Mahsa Eskian¹, Laila Cochon¹, Isha Gujrathi¹, Andro Licaros¹, Anna Zhao^{1

2}, Nicole Vetrano¹, Louise Schneider^{1

2}, Ali Raja^{1

2

3}, Ramin Khorasani^{1

2}

Affiliations

¹ Department of Radiology, Center for Evidence-Based Imaging, Brigham and Women's Hospital, Boston, Massachusetts, USA.
² Harvard Medical School, Boston, Massachusetts, USA.
³ Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA.

Abstract

Objective: Clinical evidence logic statements (CELS) are shareable knowledge artifacts in a semistructured "If-Then" format that can be used for clinical decision support systems. This project aimed to assess factors facilitating CELS representation.

Materials and methods: We described CELS representation of clinical evidence. We assessed factors that facilitate representation, including authoring instruction, evidence structure, and educational level of CELS authors. Five researchers were tasked with representing CELS from published evidence. Represented CELS were compared with the formal representation. After an authoring instruction intervention, the same researchers were asked to represent the same CELS and accuracy was compared with that preintervention using McNemar's test. Moreover, CELS representation accuracy was compared between evidence that is structured versus semistructured, and between CELS authored by specialty-trained versus nonspecialty-trained researchers, using χ² analysis.

Results: 261 CELS were represented from 10 different pieces of published evidence by the researchers pre- and postintervention. CELS representation accuracy significantly increased post-intervention, from 20/261 (8%) to 63/261 (24%, P value < .00001). More CELS were assigned for representation with 379 total CELS subsequently included in the analysis (278 structured and 101 semistructured) postintervention. Representing CELS from structured evidence was associated with significantly higher CELS representation accuracy (P = .002), as well as CELS representation by specialty-trained authors (P = .0004).

Discussion: CELS represented from structured evidence had a higher representation accuracy compared with semistructured evidence. Similarly, specialty-trained authors had higher accuracy when representing structured evidence.

Conclusion: Authoring instructions significantly improved CELS representation with a 3-fold increase in accuracy. However, CELS representation remains a challenging task.

Keywords: clinical decision support; health information technology; knowledge representation.