Jul 28, 2021

January 17, 2021 . Safety and Acceptability of a Natural Language Artificial Intelligence Assistant to Deliver Clinical Follow-up to Cataract Surgery Patients: Proposal Safety and Acceptability of a Natural Language Artificial Intelligence Assistant to Deliver Clinical Follow-up to Cataract Surgery Patients: Proposal Authors of this article: 2Centre for Health Technology, University of Plymouth, Plymouth, United Kingdom 3Imperial College London, London, United Kingdom 4University of Oxford, Oxford, United Kingdom 5Department of Primary Care and Public Health, Imperial College London, London, United Kingdom Corresponding Author: Centre for Health Technology Abstract Background: Due to an aging population, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high-volume workloads, driving increasing costs for providers. Artificial intelligence (AI), in the form of conversational agents, presents a possible opportunity to enable efficiency in the delivery of care. Objective: This study aims to evaluate the effectiveness, usability, and acceptability of Dora agent: Ufonia’s autonomous voice conversational agent, an AI-enabled autonomous telemedicine call for the detection of postoperative cataract surgery patients who require further assessment. The objectives of this study are to establish Dora’s efficacy in comparison with an expert clinician, determine baseline sensitivity and specificity for the detection of true complications, evaluate patient acceptability, collect evidence for cost-effectiveness, and capture data to support further development and evaluation. Methods: Using an implementation science construct, the interdisciplinary study will be a mixed methods phase 1 pilot establishing interobserver reliability of the system, usability, and acceptability. This will be done using the following scales and frameworks: the system usability scale; assessment of Health Information Technology Interventions in Evidence-Based Medicine Evaluation Framework; the telehealth usability questionnaire; and the Non-Adoption, Abandonment, and Challenges to the Scale-up, Spread and Suitability framework. Results: The evaluation is expected to show that conversational technology can be used to conduct an accurate assessment and that it is acceptable to different populations with different backgrounds. In addition, the results will demonstrate how successfully the system can be delivered in organizations with different clinical pathways and how it can be integrated with their existing platforms. Conclusions: The project’s key contributions will be evidence of the effectiveness of AI voice conversational agents and their associated usability and acceptability. International Registered Report Identifier (IRRID): PRR1-10.2196/27227 JMIR Res Protoc 2021;10(7):e27227 Keywords Clinical Problem In the United Kingdom, aging population is causing an increased demand for health care services that is exceeding clinical capacity [ 1 ]. With staff pressured to deliver high-volume workloads, the resulting burnout crisis is increasing costs for health care providers. Demand has been further exacerbated by the COVID-19 pandemic, as the widespread cancelation of elective care has created a large backlog of clinical work [ 2 ]. However, a large proportion of this clinical work is taken up by highly repetitive and low-skill tasks, preventing staff from working at the top of their license. Therefore, there is a need to improve the efficiency in the delivery of care and to collect data that can be analyzed to support routine improvement and optimization through the automation of routine clinical interactions. One area of care where improved efficiency is urgently needed is cataract surgery. Cataract surgery is the most common operation in the National Health Service (NHS), with approximately 450,000 procedures conducted per year [ 3 ]. COVID-19 has caused record delays in receiving planned surgeries [ 4 ], and the average wait time for cataract surgery was approximately 2.5 months [ 5 ]. The aging population will also have a significant impact on the number of patients with cataracts, which is expected to double between now and 2050 [ 6 ]. The cataract pathway is also an ideal case for optimization because there is little variability and high levels of patient safety; the most significant complication (endophthalmitis) occurs in fewer than 1 in 1000 cases [ 7 ]. To address this clinical need, this study aims to collect data on the effectiveness, usability, and acceptability of an artificial intelligence (AI) natural language assistant for delivering cataract surgery follow-up checks. Current System and Its Limitations Similar to most operations, cataract surgery requires a postoperative check to monitor for complications and assess success. This has historically been performed with a face-to-face visit; prior to the COVID-19 pandemic, this was the standard procedure for 72% of NHS Trusts [ 8 ]. However, this postoperative system has a high operational demand and is not always necessary; a recent ophthalmology Getting It Right First Time (GIRFT) report stated that a hospital review of cataract surgery patients is not required, as long as follow-up arrangements are in place [ 8 ]. In the current context of the COVID-19 pandemic, face-to-face visits also pose a high risk for virus transmission due to the proximity of patients and clinicians. Although this project focuses on cataract surgery follow-up, the underlying platform will be applicable to a wide range of routine clinical tasks. The need for an effective automated tool is especially great, as a new normal of widespread remote clinical care has been established in the wake of COVID-19. User Needs Using AI-enabled automation is a novel approach for patient care, and patient perspectives are mixed [ 9 , 10 ]. Research on cataract surgery patients during the development of this solution found that although many patients stated that they would prefer a clinician to provide follow-up, this opinion was contingent on the quality of the system and service [ 11 ]. Most patients rated the solution highly on simplicity and ease of use and expressed appreciation for the additional convenience of a telephone follow-up [ 11 ]. The patients also suggested that the AI system might be less rushed than a clinician and would have the benefit of freeing up nurses’ time for other clinical work. This is supported by internal company research on the clinical outcomes of 300 cataract surgery follow-up calls, which identified that only 10% (30/300) of patients telephoned by expert clinicians were determined to need a face-to-face review. Therefore, an automated telephone follow-up could significantly reduce the number of clinical appointments necessary to deliver postoperative care. Solution Overview General Description The solution developed to improve clinical efficiency for cataract surgery follow-up is a natural language, voice telemedicine conversation delivered to patients via telephone calls. For the patient, this is intended to be no different than a regular telemedicine consultation with a doctor or nurse; it does not require the download of an app, the provision of a device, or any training. This is important because the populations that consume the majority of health care services (older adults and socioeconomically disadvantaged populations) tend to be relatively more digitally disenfranchised. The solution, a natural language AI assistant called Dora agent: Ufonia’s autonomous voice conversational agent, has been developed by Ufonia Limited. By the start of the clinical testing described in this proposal, validation with Ufonia’s development partners at Buckinghamshire Healthcare NHS Trust will be complete, and Dora will be at Technical Readiness Level (TRL)5. By the end of this study, the solution will be at TRL6, which has been demonstrated in a relevant clinical setting. Key Features Dora uses a variety of AI technologies to deliver the patient follow-up call, including speech transcription, natural language understanding, a machine learning conversation model to enable contextual conversations, and speech generation. Together, these technologies cover the input, processing and analysis, and output needed to maintain a natural conversation. Dora is currently delivered via a telephone connection as a real-time, stand-alone system. Benefits of Proposed Solution For Digitally Enabled Care The solution is aligned with the aims of NHS to improve efficiency and increase digitally enabled care [ 8 , 12 ]. Cataract surgery follow-up provides a good launching point for further development of digital solutions, as it typifies many routine telemedicine care processes that could be similarly automated. For Health Care Staff The solution is expected to have significant benefits for health care staff by reducing their clinical burden. Preliminary company research suggests that Dora will reduce the number of patients who require follow-up with a clinician by approximately 80%. This reduces the type of high-volume, repetitive task that contributes to burnout and allows clinicians to be redeployed to higher-value activities, where their skills, insight, and empathy can be best used. In addition, telephone follow-up reduces the risk of COVID-19 transmission and frees up hospital space to help meet the increasing patient demand. For Patients The system also has several benefits for patients. Dora will provide a reliable and consistent safety net after surgery. Patients will be able to ask questions about their recovery (such as when to drive, swim, and stop taking eye drops) just as they would with a human clinician. The system is convenient because it does not require them to travel to the hospital and can take place at the time or duration that suits them. Reducing the number of in-person follow-ups will allow clinicians to perform other clinical activities, making patients more likely to receive timely care for their initial cataract surgery or for other conditions. This will be evaluated as part of the study by examining the number of surgeries conducted. Aims and Objectives The purpose of this study is to evaluate the evidence for clinical safety, study design feasibility, usability, acceptability, satisfaction, appropriateness, and cost-effectiveness of the autonomous cataract follow-up call system (Dora) for detecting patients that require further assessment. The primary aims are to establish preliminary evidence that Dora is safe, to evaluate its sensitivity and specificity, and to determine what can be learned to improve its design for future studies. To achieve these aims, there are 5 key objectives: To establish baseline rates of efficacy for Dora’s detection of patients requiring further assessment To evaluate patient acceptability of an autonomous call in comparison with existing standards of care To evaluate the cost-effectiveness of autonomous calls in comparison with existing standards of care To capture conversational data to train future versions of the system To capture data and assess study feasibility to inform the development of future trials Research Questions On the basis of these aims and objectives, 4 main research questions were defined to guide the project: What are the factors impacting the effectiveness of Dora’s conversational call follow-up to determine patients who require further assessment? Can Dora sufficiently support conversation and patient engagement to collect the data needed to allow Dora’s computational capabilities to perform an accurate assessment? What are the perceived benefits and barriers of using conversational agents for patient follow-up? Is Dora more cost-effective than existing standards of care? Methods Study Design Using an implementation science construct, the study will be a phase 1 pilot study to develop evidence regarding the feasibility, acceptability, and potential effectiveness of Dora and to identify factors influencing effectiveness. The study will last for 18 months: 6 months of evaluation and intervention refinement, 9 months of implementation and follow-up, and 3 months of postevaluation analysis and write-up. Research Participants The study population will incorporate two clinical sites: the Imperial College Healthcare NHS Trust and the Oxford University Hospitals NHS Trust. The population at the Imperial College site is drawn from the North West London Collaboration of Clinical Commissioning Groups (CCGs) and is densely populated, highly diverse, highly mobile, and relatively young [ 13 ]. Black, Asian, and other minority ethnic groups make up 37% of the resident population in the West London CCG [ 14 ], and 20% of London residents do not speak English as their first language [ 15 ]. In contrast to London, the Oxfordshire population is less diverse but more rural [ 16 ]. Although it is growing in ethnic diversity, residents are primarily of a White British background; in 2011, approximately 84% of Oxfordshire residents identified as White British (compared with the national average of approximately 80%) [ 17 ]. It was also found that 16% of the Oxfordshire population did not speak English as their main language (compared with the national average of 8%) [ 18 ]. In terms of age, the population of Oxfordshire is similar to the national profile, particularly for older ages [ 19 ]. Recruitment Recruitment will take place at sites in the Imperial College Healthcare and Oxford University Hospitals NHS Trusts that conduct cataract surgeries. Study information will be shared with cataract surgery patients at their initial visit (and via post or telephone call for patients who are delayed by COVID-19 or cannot visit in person). Informed consent will be obtained at the time of preassessment (in-person or virtually depending on the current COVID-19 guidelines). In the discharge lounge following surgery, patients who have consented to participate will be given further information to remind them about the call from Dora. This work will be performed by a dedicated research nurse at each site. Study Duration and Follow-up This study will conduct telemedicine calls with patients in addition to their standard of care. Patients will receive their call from Dora between 25 and 27 days post surgery. The evaluation of Dora in two different NHS Trusts will help demonstrate that Dora can be integrated into different pathways. It will also provide a range of sensitivities to input into the health economic models to determine whether Dora provides cost-effectiveness in different settings. Procedure Overview Although patients typically have cataracts in both eyes, procedures are typically performed on one eye at a time in the United Kingdom. As this is a pragmatic study, patients undergoing either first or second eye surgery will be recruited. This should be in a balanced proportion due to the random nature of operating timing, but differences between groups will be examined in the posthoc analysis. Given that the usual interval between cataract surgeries is more than 6 months, it is unlikely that the same patient will be recruited twice in the study. If there are any cases where the same patient is recruited, they will be excluded from participating for the second time. Dora Call Greeting and introduction Closure of call The entire conversation will be supervised by an expert clinician (an ophthalmology research fellow). This clinician will be able to interrupt the call at any point if the system fails, the patient struggles to interact with it, or Dora does not collect sufficient information from the patient. The cataract follow-up questions will classify 5 key symptoms: redness, pain, reduced vision, flashing lights, and floaters. Both Dora and the supervising clinician (masked to each other) will independently indicate for each symptom, whether the symptom is: Absent (eg, no pain) Present and clinically significant (eg, deep and persistent pain) Insufficient information for classification Issues identified in response to any of these questions will prompt the need for a face-to-face review. The complexity of the model comes from evaluating the exact nature of these symptoms, for example, distinguishing between improving redness in the corner of the eye (due to the local anesthetic injection) and widespread redness that has progressed, which may represent infection. The conversational nature of the model enables it to ask patients further questions to clarify their responses and assess the significance of the reported symptoms. If clarifying points are necessary for the clinician to make a decision, Dora will enable the clinician who will ask the necessary questions, and then record updated assessments for symptoms and overall management before handing back to Dora for frequently asked questions and patient evaluation. If the call is uninterrupted, Dora will make a decision about the patient’s management plan. Once the cataract follow-up questions have been completed, the clinician will also set their own decision, masked to the decision made by the system ( Table 1 ). For this project, the supervising clinician’s decision, which is made based on the information from the call with Dora’s decision masked, is considered the gold standard for evaluation. For each patient, the final decision regarding the management plan can be as follows: Discharge (and/or add to waiting list for second eye cataract surgery and/or continue follow-up as previously planned in the ophthalmology clinic) Eye casualty review within 1 week Same day eye casualty review Table 1. Outcome communicated to the patient at the end of the call depending on the blinded decision of Digital Online Recording Automation agent: Ufonia’s autonomous voice conversational agent and the clinician. Clinician decision References Oliver D, Foot C, Humphries R. Making our health and care systems fit for an ageing population. In: The King's Fund: Ideas that change health care. London: The King's Fund; 2014. Illman J. Exclusive: NHS prepares to cancel elective ops in readiness for covid-19 surge Internet. Health Service Journal. URL: https:/​/www.​hsj.co.uk/​free-for-non-subscribers/​exclusive-nhs-prepares-to-cancel-elective-ops-in-readiness-for-covid-19-surge/​7027110.​article [accessed 2020-06-20] Donachie P, Sparrow J. National Ophthalmology Database Audit. United Kingdom: The Royal College of Ophthalmologists; 2020. Griffin S. Covid-19: Waiting times in England reach record highs. In: BMJ. Covid-19: Waiting times in England reach record highs. BMJ Internet British Medical Journal Publishing Group; Sep 11, 2020:m3557. Dayan M, Ward D, Gardner T, Kelly E. How good is the NHS? The Health Foundation, the Institute for Fiscal Studies, The King's Fund, and the Nuffield Trust. London: The Health Foundation; 2018. URL: https://www.nuffieldtrust.org.uk/files/2018-06/the-nhs-at-70-how-good-is-the-nhs.pdf [accessed 2020-06-30]