Latest NEQAS News
Aug 9, 2021
The Use and Effectiveness of an Online Diagnostic Support System for Blood Film Interpretation: Comparative Observational Study The Use and Effectiveness of an Online Diagnostic Support System for Blood Film Interpretation: Comparative Observational Study Authors of this article: 2University Hospitals Plymouth NHS Trust, Plymouth, United Kingdom 3Manchester Foundation Trust, Manchester, United Kingdom 4UK NEQAS Haematology, Watford, United Kingdom 5The Christie NHS Foundation Trust, Manchester, United Kingdom 6Division of Cancer Sciences, University of Manchester, Manchester, United Kingdom Corresponding Author: Abstract Background: The recognition and interpretation of abnormal blood cell morphology is often the first step in diagnosing underlying serious systemic illness or leukemia. Supporting the staff who interpret blood film morphology is therefore essential for a safe laboratory service. This paper describes an open-access, web-based decision support tool, developed by the authors to support morphological diagnosis, arising from earlier studies identifying mechanisms of error in blood film reporting. The effectiveness of this intervention was assessed using the unique resource offered by the online digital morphology Continuing Professional Development scheme (DM scheme) offered by the UK National External Quality Assessment Service for Haematology, with more than 3000 registered users. This allowed the effectiveness of decision support to be tested within a defined user group, each of whom viewed and interpreted the morphology of identical digital blood films. Objective: The primary objective of the study was to test the effectiveness of the decision support system in supporting users to identify and interpret abnormal morphological features. The secondary objective was to determine the pattern and frequency of use of the system for different case types, and to determine how users perceived the support in terms of their confidence in decision-making. Methods: This was a comparative study of identical blood films evaluated either with or without decision support. Selected earlier cases from the DM scheme were rereleased as new cases but with decision support made available; this allowed a comparison of data sets for identical cases with or without decision support. To address the primary objectives, the study used quantitative evaluation and statistical comparisons of the identification and interpretation of morphological features between the two different case releases. To address the secondary objective, the use of decision support was assessed using web analytical tools, while a questionnaire was used to assess user perceptions of the system. Results: Cases evaluated with the aid of decision support had significantly improved accuracy of identification for relevant morphological features (mean improvement 9.8%) and the interpretation of those features (mean improvement 11%). The improvement was particularly significant for cases with higher complexity or for rarer diagnoses. Analysis of website usage demonstrated a high frequency of access for web pages relevant to each case (mean 9298 for each case, range 2661-24,276). Users reported that the decision support website increased their confidence for feature identification (4.8/5) and interpretation (4.3/5), both within the context of training (4.6/5) and also in their wider laboratory practice (4.4/5). Conclusions: The findings of this study demonstrate that directed online decision support for blood morphology evaluation improves accuracy and confidence in the context of educational evaluation of digital films, with effectiveness potentially extending to wider laboratory use. J Med Internet Res 2021;23(8):e20815 Effect on Decision-making in Complex Morphological Cases The case of adult T-cell leukemia lymphoma (ATLL) demonstrated the challenge of detailed morphological evaluation and interpretation of abnormalities when there were small numbers of abnormal cells ( Figure 4 A). The case required participants first to recognize that the lymphoid cell population was abnormal, then to make a detailed morphological assessment of the cells present, and ascribe a likely diagnosis based on that assessment. This case was difficult for participants when first released, with only around 16% making the correct diagnosis. When rereleased, the case attracted particularly frequent use of the decision support website, with more than 18,000 relevant pages accessed by participants, of which more than 3700 were specific for the diagnosis of ATLL. This high use of support was associated with an improved accuracy in the different stages of the diagnostic process; in the initial release, 829 participants answered the case, with 1282 participants in the more recent release. Although participants identified the lymphocytosis with equal frequency, those in the more recent case release group were more likely to identify the cells as neoplastic (n=615, 48% versus n=328, 40%, P<.001, chi-square test, Fisher exact test) and were also significantly more likely to suggest that the cells represented a neoplastic T-cell disorder (934/1282 participants versus 414/829, 73% versus 50%; P<.001, chi-square test with Yates correction; Figure 4 B). This was associated with a significantly greater likelihood of ascribing a precise diagnosis to the case (rather than a generic diagnosis of T-cell lymphoproliferative disorder) as well as a much higher recognition that the diagnosis was ATLL (513/1282, 45% versus 108/829, 13%; P<.001, chi-square test with Yates correction; Figure 4 C). These findings suggest that support for the detailed cytological evaluation of rare cell types was widely used, and that this resulted in improved accuracy in the morphological evaluation of uncommon disorders. Effect on Decision-making in the Presence of Multiple Abnormal Features For those diagnoses where a range of abnormal morphological features are present, interpretation requires strategies that depend on the identification, classification, and prioritization of the abnormal features. The accuracy of these strategies depends very much on the skill and knowledge of the morphologist, and may result in unhelpful simplification processes such as elimination bias (where features are ignored or assigned inappropriately low significance for diagnosis) or framing bias (where features are considered to fit a single preferred diagnosis, irrespective of whether this is the most likely interpretation). Consistent with this, the most complex case in this set combined the red blood cell disorder Southeast Asian ovalocytosis (SAO) with an accompanying Epstein-Barr virus infection, leading to distinctive but complex morphology ( Figure 5 A). This case proved difficult for participants when first released, with a tendency for errors related to simplification strategies; in the initial release, the marked red blood cell abnormalities were all reported relatively infrequently, particularly the diagnostically important stomatocytes ( Figure 5 B), and red blood cell diagnoses were offered by only 245 of 991 (25%) participants. The correct interpretation of SAO was made by just 69 of 991 participants (7%) in the initial release ( Figure 5 C). In contrast, the features of viral illness were well recognized (reported and correctly interpreted by 961/991 participants, 97%; Figure 5 D). When the case was rereleased, the decision support tool was heavily used, with more than 11,000 views of pages relevant to the case, and this was associated with a more accurate identification of abnormal morphological features ( Figure 5 B), with the overall consideration of red blood cell disorders in the final diagnosis increased to 841/1402 participants (60%; P<.001; Figure 5 C) and the correct diagnosis of SAO given by 365/1402 participants, (26%; P<.001, chi-square test with Yates correction). Interestingly, the increased awareness of the red blood cell features led to indications that some participants used a framing bias, attempting to unify the white blood cell and red blood cell features into a single diagnosis of liver disease or malignancy ( Figure 5 D). A small proportion showed elimination bias, as they simply omitted any white blood cell diagnosis at all. Overall, the reduced rate of diagnosis of viral disorder was not significant (P=.19, chi-square test with Yates correction). Discussion Principal Results In this study, digital blood films provided within the online UK NEQAS Haematology DM CPD scheme were used to evaluate the effectiveness of a new decision support system developed by the authors to support morphological assessment in hematology. This analysis revealed that use of the decision support site was associated with a significantly improved accuracy of reporting for both morphological feature identification and the interpretation of those features. That information, together with web page analytic data and direct user feedback, has shown that the system was widely used by CPD participants and improved the performance and confidence of users. The detailed analysis of the two most difficult cases was particularly useful to evaluate the effectiveness of support. For white blood cell types, the system focused on providing detailed help in assessing cytology of individual cell types to support recognition of characteristic features that allow discrimination between different cell forms. This help was very highly used in the assessment of the uncommon lymphoproliferative disorder ATLL, and this use was associated with a very substantial improvement both in recognizing the unusual (T-cell) nature of the disorder and in suggesting the correct diagnosis (ATLL). For red blood cell diagnoses, the aim of the system was (as for white blood cells) to provide help in recognizing abnormal forms, and additionally focused on identifying pitfalls in cell recognition and showing how combinations of features could be used to make a diagnosis. In the case of SAO, the participants more frequently reported those relevant red blood cell features present and also showed considerable improvement in suggesting the correct diagnosis (SAO). Possible Limitations of the Study The decision support was evaluated using an online education system and compared present users examining DM CPD cases with the historical performance of users assessing the same case. We recognize that the study design therefore contains potentially confounding factors related to the separate time points. In particular, users in the current period were likely to have increased familiarity with technology and online resources and were more likely to have access to better technical resources, screen quality, and download speed [ 17 , 18 ]. The study partly addresses these limitations, showing that irrespective of other factors the decision support pages were frequently accessed at times when CPD cases were available for assessment. In addition, the web pages accessed corresponded closely to the subject of the case, consistent with directed use of the system by participants. Furthermore, there was evidence of selective use of pages according to case difficulty, with a marked correlation between the number of times relevant pages were accessed and the difficulty of the case (as indicated by performance in the earlier period). It is also worth noting that the system does not address all sources of error. The case of SAO also had reactive (“viral”) lymphocytes, which were very widely reported in the earlier release; interestingly, the improved awareness of red blood cell features was associated with a small reduction in those reporting the likely viral disorder. These errors were similar to the “heuristic errors” observed in our previous study of cases with a highly complex group of features [ 15 ] and may reflect familiar sources of bias in complex cases, such as “anchoring” or “elimination” bias, where morphologists focused only on red blood cell features. Context and Prior Work Providing effective support for morphological diagnosis requires an understanding of the process; the complexity and number of cells present on a blood film means that (consciously or unconsciously) morphologists employ approaches to reduce the complexity of the evaluation to allow a timely conclusion [ 19 , 20 ]. Briefly, the process has two phases. The first phase is “perception,” a process where the morphologist perceives whether the observed appearances differ from an expectation of normality; this is a rapid and largely subconscious process [ 21 ], whose effectiveness depends significantly on the experience of the observer [ 22 ]. The second phase employs the skills of “recollection and analysis” to evaluate particular findings in detail [ 23 ], requiring the active application of knowledge through techniques that simplify analysis, such as classification (allowing abnormal cells to be considered as classes or groups rather than as separate entities), prioritization (to focus on those elements that are considered most important to diagnosis), and elimination (to remove from consideration those elements considered unimportant to diagnosis). These techniques all require an appropriate knowledge base for accurate outcomes; when incorrectly performed, the process can lead to errors of various types (reviewed in [ 15 , 24 , 25 ]). The support system in this paper describes an approach to providing effective “bench side” diagnostic support for the analytical phase of blood film evaluation by providing a rapid and accessible information source available at the time of evaluation focusing on the central processes of cell recognition, classification, prioritization, and interpretation. A digital platform was selected based on the advantages offered by such a system in the context of morphology. First, digital platforms enable the uploading and displaying of many images, as well as the ability to select and magnify features or to drive links or visual menu systems. Second, digital platforms are accessible to all users and are accessed through computers or mobile devices that can be freely available at the point of need. Third, this digital approach offers a flexibility that can be iteratively modified to suit user needs or changing information; this platform can be linked to other resources and websites that provide supporting or detailed information. It is important to reflect that global health resources should offer accessibility across geographical barriers; in this case, the adoption of a MediaWiki platform and free access to all users facilitates wide use. It is recognized though that language barriers often remain. However, while the decision support system developed by the authors is written solely using English language, there are positive features for future development. First, the use of a MediaWiki platform allows access to the tools developed to facilitate easier page translation for Wikipedia and related applications. These tools support potential future collaborative development of the system in different language formats. Second, the use of image-driven menus within the system was intended to support easier navigation for those not fully familiar with morphological descriptions, but this may also enable the use of the system by a wider international group. Conclusions In this paper, we have demonstrated the effectiveness of this specifically designed online tool to improve the performance of morphologists in the setting of an online CPD system. However, the broader question is whether there is an expanded role for its use in general diagnosis within laboratories. The internet is gaining wider penetration in all areas of society; the public is increasingly using and familiar with internet resources, often accessed by mobile devices. Online diagnosis support can be used either at the microscope or as an adjuvant to computer-assisted diagnostic systems. The flexibility of a web-based approach may also be extended, particularly to support fast-evolving areas such as molecular diagnosis. Finally, the system may have particular value in developing countries where internet access via mobile devices may become a major point of access for teaching, training, or other services, and where low-cost flexible support can provide wider benefit if problems of adoption can be overcome [ 12 , 26 ]. Acknowledgments The authors would like to thank Celgene (UK) and Sysmex (UK) for support of this work through educational grants. Authors' Contributions CH and MB contributed equally to the manuscript, and both took images, analyzed data, and wrote the paper. JS and BDLS provided and analyzed data and edited the manuscript. RC, JA, and RB wrote text and reviewed the paper. KH contributed to the design of the study. KRU wrote and edited the paper. JB designed the study, took images, analyzed data, and wrote the paper. Conflicts of Interest Multimedia Appendix 1 The EQATE system. (A) The menu system in the left panel allows users to select information or provide responses to the case; the panel can be expanded by users as required to provide a larger view of the image, which can be expanded or contracted to allow higher magnification viewing. (B) The full image field in the right panel shows around 50 images taken at high magnification using an oil immersion lens allowing maximum resolution of detail, while the left panel shows the narrative that guides users through the major features of the film following completion of the case. (C) Image viewed at high power (equivalent to magnification at the time of image acquisition). (D) Additional features of the case are shown at highest magnification with an accompanying annotation following completion of the case. References Comar SR, Malvezzi M, Pasquini R. Evaluation of criteria of manual blood smear review following automated complete blood counts in a large university hospital. Rev Bras Hematol Hemoter 2017;39(4):306-317 [ FREE Full text ] [ CrossRef ] [ Medline ] Adewoyin AS, Nwogoh B. Peripheral blood film - a review. Ann Ib Postgrad Med 2014 Dec;12(2):71-79 [ FREE Full text ] [ Medline ] Palmer L, Briggs C, McFadden S, Zini G, Burthem J, Rozenberg G, et al. ICSH recommendations for the standardization of nomenclature and grading of peripheral blood cell morphological features. Int J Lab Hematol 2015 Jun;37(3):287-303. [ CrossRef ] [ Medline ] Buoro S, Da Rin G, Fanelli A, Lippi G. Harmonization of interpretative comments in laboratory hematology reporting: the recommendations of the Working Group on Diagnostic Hematology of the Italian Society of Clinical Chemistry and Clinical Molecular Biology (WGDH-SIBioC). Clin Chem Lab Med 2018 Dec 19;57(1):66-77. [ CrossRef ] [ Medline ] Caruso C. Negative impacts of shiftwork and long work hours. Rehabil Nurs 2014;39(1):16-25 [ FREE Full text ] [ CrossRef ] [ Medline ] Brown M, Tucker P, Rapport F, Hutchings H, Dahlgren A, Davies G, et al. The impact of shift patterns on junior doctors' perceptions of fatigue, training, work/life balance and the role of social support. Qual Saf Health Care 2010 Dec 25;19(6):e36-e36 [ FREE Full text ] [ CrossRef ] [ Medline ] Criel M, Godefroid M, Deckers B, Devos H, Cauwelier B, Emmerechts J. Evaluation of the Red Blood Cell Advanced Software Application on the CellaVision DM96. Int J Lab Hematol 2016 Aug;38(4):366-374. [ CrossRef ] [ Medline ] Yu H, Ok CY, Hesse A, Nordell P, Connor D, Sjostedt E, et al. Evaluation of an automated digital imaging system, Nextslide Digital Review Network, for examination of peripheral blood smears. Arch Pathol Lab Med 2012 Jun;136(6):660-667 [ FREE Full text ] [ CrossRef ] [ Medline ] Briggs C, Longair I, Slavik M, Thwaite K, Mills R, Thavaraja V, et al. Can automated blood film analysis replace the manual differential? An evaluation of the CellaVision DM96 automated image analysis system. Int J Lab Hematol 2009 Feb;31(1):48-60. [ CrossRef ] [ Medline ] Safi S, Thiessen T, Schmailzl KJ. Acceptance and Resistance of New Digital Technologies in Medicine: Qualitative Study. JMIR Res Protoc 2018 Dec 04;7(12):e11072 [ FREE Full text ] [ CrossRef ] [ Medline ] Hutchinson CV, Brereton ML, Burthem J. Digital imaging of haematological morphology. Clin Lab Haematol 2005 Dec;27(6):357-362. [ CrossRef ] [ Medline ] Ahmed L, Seal LH, Ainley C, De la Salle B, Brereton M, Hyde K, et al. Web-Based Virtual Microscopy of Digitized Blood Slides for Malaria Diagnosis: An Effective Tool for Skills Assessment in Different Countries and Environments. J Med Internet Res 2016 Aug 11;18(8):e213 [ FREE Full text ] [ CrossRef ] [ Medline ] Burthem J, Brereton M, Ardern J, Hickman L, Seal L, Serrant A, et al. The use of digital 'virtual slides' in the quality assessment of haematological morphology: results of a pilot exercise involving UK NEQAS(H) participants. Br J Haematol 2005 Jul;130(2):293-296. [ CrossRef ] [ Medline ] Brereton M, De La Salle B, Burthem J, Ardern J, Hickman L, Seal L, et al. Review of the UK NEQAS (H) digital morphology pilot scheme for continuing professional development accessed via the internet. Int J Lab Hematol 2008 Oct;30(5):365-371. [ CrossRef ] [ Medline ] Brereton M, De La Salle B, Ardern J, Hyde K, Burthem J. Do We Know Why We Make Errors in Morphological Diagnosis? An Analysis of Approach and Decision-Making in Haematological Morphology. EBioMedicine 2015 Sep;2(9):1224-1234 [ FREE Full text ] [ CrossRef ] [ Medline ] Haematologyetc. URL: https://haematologyetc.co.uk [accessed 2021-07-26] Haag M, Igel C, Fischer MR, German Medical Education Society (GMA)‚ Committee “Digitization – Technology-Assisted Learning and Teaching”, Joint working group “Technology-enhanced Teaching and Learning in Medicine (TeLL)” of the German Association for Medical Informatics‚ Biometry and Epidemiology (gmds) and the German Informatics Society (GI). Digital Teaching and Digital Medicine: A national initiative is needed. GMS J Med Educ 2018;35(3):Doc43 [ FREE Full text ] [ CrossRef ] [ Medline ] Thimbleby H. Technology and the future of healthcare. J Public Health Res 2013 Dec 01;2(3):e28 [ FREE Full text ] [ CrossRef ] [ Medline ] Shah AK, Oppenheimer DM. Heuristics made easy: an effort-reduction framework. Psychol Bull 2008 Mar;134(2):207-222. [ CrossRef ] [ Medline ] Wegwarth O, Gaissmaier W, Gigerenzer G. Smart strategies for doctors and doctors-in-training: heuristics in medicine. Med Educ 2009 Aug;43(8):721-728. [ CrossRef ] [ Medline ] DiCarlo JJ, Zoccolan D, Rust NC. How does the brain solve visual object recognition? Neuron 2012 Feb 09;73(3):415-434 [ FREE Full text ] [ CrossRef ] [ Medline ] Bar M. A cortical mechanism for triggering top-down facilitation in visual object recognition. J Cogn Neurosci 2003 May 15;15(4):600-609. [ CrossRef ] [ Medline ] Henson RN, Rugg MD, Shallice T, Josephs O, Dolan RJ. Recollection and Familiarity in Recognition Memory: An Event-Related Functional Magnetic Resonance Imaging Study. J Neurosci 1999 May 15;19(10):3962-3972 [ FREE Full text ] [ CrossRef ] [ Medline ] Gunderman RB. Biases in radiologic reasoning. AJR Am J Roentgenol 2009 Mar;192(3):561-564. [ CrossRef ] [ Medline ] Blumenthal-Barby JS, Krieger H. Cognitive biases and heuristics in medical decision making: a critical review using a systematic search strategy. Med Decis Making 2015 May;35(4):539-557. [ CrossRef ] [ Medline ] Maware S, Sai KOS. An investigation on e-resource utilisation among university students in a developing country: A case of Great Zimbabwe University. South African Journal of Information Management 2018;20(1):a860.