Delphi

Nov 16, 2023

Experts who exhibit a rigorous scientific research attitude and can provide comprehensive and professional insights. Individuals who possess a certain level of understanding of the Delphi method and voluntarily express their willingness to participate in the study. Patients with e-consultation experience: Having had at least three online full electronic consultationsspecialists who have been engaged in relevant fields for at least three years. Individuals who possess a certain level of understanding of the Delphi method and voluntarily express their willingness to participate in the study. Keeping these criteria in mind, we have selected a diverse and knowledgeable group of experts, including doctors with experience in online doctor-patient communication, experts in doctor-patient interpersonal communication, and patients who have undergone online medical service, to ensure a multidimensional and comprehensive analysis of the quality indicators for electronic consultations. Expert Consultation Questionnaire After conducting a thorough search and review of relevant literature, combined with the existing scale, we summarized the dimensions for evaluating the quality of online doctor-patient communication. Through extensive discussions within the research group, a preliminary indicator system was established consisting of three first-level indicators, ten second-level indicators, and twenty-nine third-level indicators. To gather expert opinions and feedback, a questionnaire was developed for the first round of expert consultation. The questionnaire consists of three parts: Explanation of the questionnaire. This section provides the research background, purpose, and significance of the study, ensuring that experts have a clear understanding of the survey’s objectives. Evaluation of each indicator by experts. Using a Likert 5 scale, experts rate the importance and feasibility of each indicator. The scale ranges from “not important” to “very important” for importance, and from “not feasible” to “very feasible” for feasibility. Each indicator is assigned a corresponding rating from 1 to 5. Experts are encouraged to provide suggestions during this part, such as proposing additional indicators or reasons for removing certain indicators. Survey of experts’ basic information. In this section, experts are asked to assess their familiarity with the consultation content and the impact of different judgment bases, such as practical or scientific research experience, medical experience, theoretical knowledge, domestic and foreign references, and visual judgment. By including these components in the questionnaire, we aim to gather comprehensive and insightful feedback from experts, ensuring that their assessments align with both their expertise and individual perspectives on the subject matter. The Implementation of Expert Consultation Two rounds of expert consultation were conducted from December 2022 to January 2023. For the consulting panel, we carefully selected experts in nursing, clinical medicine, and medical communication expert who possessed in-depth knowledge and authority in their respective fields. These experts were well-informed about the subject matter and displayed a willingness to collaborate as consultants. Additionally, we also invited patients with online medical experience to ensure a comprehensive perspective. After considering the practical constraints, a total of 14 experts were ultimately consulted, reflecting a diverse range of viewpoints. The consultation process took the form of an email survey, allowing for efficient communication and data collection. After receiving the responses from the first round of questionnaires, indicators that did not meet the following basic conditions were eliminated. These criteria included an average rating for importance and feasibility below 3.5, a coefficient of variation above 0.3, and a standard deviation higher than 1. Then, according to the experts’ suggestions and the discussion of the research group, the indicator system was revised. Then the revised results were then shared with the experts for the second round of negotiations to gather their valuable insights and opinions. Finally, three first-level indicators, ten second-level indicators and thirty-two third-level indicators were determined, reflecting the consensus achieved through this comprehensive consultation process. Ethical Aspects The study protocol was approved by the Ethics Committee of Chongqing Medical University (record number 2018011). The participants gave their informed consent before answering the questionnaire. Statistical Analysis The collected data were processed using Microsoft Excel before being input into a database for further analysis. Data analysis was performed using SPSS 26.0 (IBM Corporation, Armonk, NY, US). Descriptive statistics are used to calculate the mean, standard deviation, and coefficient of variation of the importance and feasibility of each indicator. The reliability of the expert’s correspondence is measured by means of authority, enthusiasm, and coordination. The consistency of the two rounds of consulting experts was determined by Kendall’s coefficient and P values <0.05 were considered statistically significant. The weights of indicators were determined using the Analytical Hierarchy Process (AHP). Through assigning importance values to each indicator, we established the Saaty scale, constructed the judgment matrix, and conducted a hierarchical ranking and consistency test, and conduct a hierarchical ranking and consistency test. These steps allowed us to determine the weight of each indicator at all levels of the hierarchy. Result In the first round of survey, the participant demographics included 7 (50%) patient experts and 7 (50%) doctor experts. Additionally, there were 7 (50%) female participants and 7 (50%) male participants. Regarding age, 5 (36%) participants fell into the 20–30 age group, while 9 (64%) fell into the 30–50 age group. The number of participants in the second round decreased by one compared to the first round ( Table 1 ). By utilizing these various analytical techniques and considering demographic factors, we aimed to ensure a comprehensive and rigorous evaluation process for the expert consultation phase of our study. Table 1 Demographic Characteristics of Participants Round 1 Fourteen participants were invited to partake in the first round of consultation, and all of them agreed to participate, resulting in a response rate of 100% ( Table 1 ), nine experts (64.29%) provided valuable suggestions during the consultation, indicating a high level of enthusiasm and engagement. It is worth noting that the self-evaluation of the experts revealed a familiarity coefficient (Cs) of 0.86 and a judgment coefficient (Ca) of 0.91. Based on these coefficients, the expert opinion authority (Cr) was calculated as 0.89 ( Table 2 ), highlighting the credibility and expertise of the participating experts. Table 2 Degree of Authority of Experts To assess the degree of coordination among experts’ opinions, we examined both the coefficient of variation (CV) and Kendall’s consistency coefficient (Kendall’s W). Following the first round of expert consultation, the Kendall’s W values for indicators at all levels ranged from 0.134 to 0.307 ( Table 3 ). These values indicate a reasonable level of agreement and consistency among the experts’ opinions. Table 3 Concordance Coefficients of Respondent Experts According to the experts’ suggestions and the discussion of the research group, a consensus was reached on the three Primary indicators, ten secondary indicators and twenty-nine tertiary indicators proposed in the first round. During the second round of consultation, all primary indicators, secondary indicators and twenty-eight tertiary indicators were retained. However, one tertiary indicator did not meet the basic conditions and was excluded, while four new tertiary indicators were added. In addition, based on the experts’ suggestions, appropriate modifications were made to the content of various indicators to enhance their relevance and effectiveness. Overall, these iterative rounds of consultation and refinement help ensure that the indicator system is comprehensive, reliable, and well-aligned with the expert opinions and knowledge in the field. For example, in consideration of the corresponding diagnosis and treatment process in online doctor-patient communication, we have made adjustments to the order of the five secondary indicators under the primary dimension. This change aligns with the sequential flow of diagnosis and treatment, where patients first report their clinical conditions, followed by further communication, and finally receiving treatment suggestions ( Table 4 ). Table 4 A Change in the Content Order Additionally, we have revised the definition of the primary indicator “responsiveness” to capture a more comprehensive understanding. It now encompasses not only the timeliness and pertinence of the patient or doctor’s reply message but also the patient’s compliance with the doctor’s diagnosis and treatment suggestions. Furthermore, under the Primary indicator “responsiveness”, we revised the secondary indicator “whether the patient refers to the doctor’s advice (the patient follows the doctor’s advice)” to “whether the patient referred to the doctor’s advice regarding medication, dietary guidelines, and follow-up visits”. This modification reflects the three specific areas where patient compliance is assessed. Finally, the modified indicator system, along with the data from all the indicators in the first round, was carried forward to the second round. Round 2 Finally, thirteen experts participated in the second round of voting, resulting in a response rate of 92.86%, and three of them proposed amendments (21.42%). The experts demonstrated a high level of authority, with a familiarity coefficient (Cs) of 0.860 and a judgment coefficient (Ca) of 0.88, resulting in a combined coefficient (Cr) of 0.880. This indicates a strong level of expertise among the experts. After the second round of expert consultation, the Kendall W value was 0.133–0.372 ( Table 3 ), with statistical significance (P<0.05). In the second round of negotiation, the importance and feasibility scores of various indicators met the basic conditions (see Table 5 for specific data). The indicator system now consists of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators. We also made some modifications to the content of the indicator. For example, the secondary indicator “special reminder given by the doctor” was refined to include additional specifics, such as the doctor told the patient the contraindications, side effects, and possible sequelae of drug use etc. (especially the part easily misunderstood by the general population). Table 5 Importance and Feasibility Coefficient of Variation of Indicators These iterative rounds of expert consultation and negotiation have ensured the comprehensiveness, relevance, and accuracy of the indicator system. The modifications made during the second round have not only enhanced the content of specific indicators but also addressed potential areas of misunderstanding, providing a more comprehensive assessment of online doctor-patient communication quality. Generation of e-Consultation Quality Evaluation Indicator System The Delphi process involved extensive research and evaluation of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators. After constantly revising and updating the indicators, the importance and feasibility scores of the indicators satisfied the basic conditions (See Table 5 for specific data). As a result of this rigorous process, a final set of three first-level indicators, ten second-level indicators, and thirty-two third-level indicators were determined to form the e-consultation quality evaluation indicator system ( Table 6 ).The results of AHP showed that the weights of three first-level indicators from high to low were Joint decision-making between doctors and patients (0.6232), Patient responsiveness (0.2395), and Interpersonal relationship between doctors and patients (0.1373). The weights of each level index are listed in Table 7 . It is worth noting that the consistency ratio (CR) for each judgment matrix was found to be lower than 0.10, indicating a high level of consistency among the judgments made. Table 6 Quality Evaluation Index of Online Doctor-Patient Communication Table 7 Weight Calculation Results of Online Doctor-Patient Communication Quality Evaluation Index System Overall, the Delphi process, combined with the AHP analysis, has led to the development of a comprehensive and well-structured indicator system for evaluating the quality of e-consultations. The thoroughness of the process, as evidenced by the consistency of the judgment matrices, ensures the reliability and accuracy of the evaluation results. Discussion With the advancement of medical informatization, there is a growing expectation for doctors to enhance their communication skills during doctor-patient interactions. 24 , 25 The communication between doctors and patients in online medical communities is complex, diverse, and the time span is relatively large. 26–28 In order to quantify and evaluate the quality of doctor-patient communication, it is essential to employ scientifically validated evaluation methods. This not only encourages doctors to actively engage in communication but also facilitates the improvement of their communication skills. In summary, this study uses the Delphi method to construct an electronic consultation quality evaluation scale. The key to predicting success or failure using the Delphi method is the selection of expert consultation. In this study, experts were selected who had more than 5 years of work experience and a bachelor’s degree or higher. They included clinical doctors, experts in doctor-patient communication, and patients with online consultation experience, all of whom had rich experience in electronic consultation. The reply rates of experts in the two rounds of consultations are 100% and 92.86%, and 12 experts provided opinions or suggestions, indicating a high level of enthusiasm and cooperation among the experts, and a relatively high importance attached to this research work. The authority coefficient of the two rounds of consultation experts was both >0.80, indicating a high level of expertise. The average importance rating of all indicators in the e-consultation quality evaluation scale constructed in this study was ≥3.5, and the coefficient of variation was <0.3, indicating a high level of consensus among the experts. In summary, this study is scientific and reasonable, and the e-consultation quality evaluation scale constructed based on the Delphi method is relatively scientific and reliable. This study sought to gather insights from both doctors and patients’ experts in order to obtain a comprehensive and multi-faceted assessment of e-consultation quality. In the first and second rounds of the Delphi process, the Kendall coefficients for the feasibility of the third-level indicators were found to be relatively low, measuring 0.134 and 0.133 respectively. This indicates that the feasibility of the tertiary indicators is low in terms of the harmony between doctors and patients, considering that it may be due to different positions of both parties in the medical service process. We conducted a separate data analysis on the consultation form of doctors and patient experts, and the results showed that Kendall coefficients, which separately counted the feasibility of the second round of entries for doctors and patient specialists, were not low, at 0.274 and 0.355 ( Table 8 ). This indicates that the consistency between doctors and the consistency between patients is relatively high. Research shows that doctors and patients have different positions in the process of medical services. 29–31 As the payers of the online consultation service, patients seek high-quality and targeted health information while also considering the efficiency of doctors’ responses and their overall satisfaction with the treatment process. On the other hand, doctors, as service providers in online consultations, doctors aim to deliver quality medical services to their patients. 32 However, due to work commitments, doctors may not always be able to respond promptly, leading to longer wait times for users. Additionally, users may not be able to describe the condition completely, resulting in low efficiency of consultation. These different perspectives of doctors and patients may contribute to the variation in their evaluation of the quality of doctor-patient communication which is consistent with our research results. Table 8 Kendal Coefficient of Doctors and Patients The e-consultation quality evaluation index system, tailored to the characteristics of Internet medical treatment, plays a vital role in enriching the theoretical framework for assessing the quality of online doctor-patient communication. It also broadens the theoretical perspectives within the field of Internet medical treatment, thereby advancing the research progress of online medical consultation services. From a practical standpoint, the evaluation and research of e-consultation quality offer valuable guidance for both online medical community operators and users. For operators, utilizing the evaluation system to assess the quality of e-consultation services aids in identifying platform issues, promptly improving service quality, enhancing user satisfaction and loyalty, and expanding the user base. Furthermore, the system provides users with a reference standard for quality evaluation, enabling them to assess the platform usage experience from multiple perspectives. This facilitates effective selection of e-consultation services, resulting in energy and cost savings. Conclusion Online consultation is a relatively new and evolving medical service that has some limitations in terms of quality, especially when it comes to non-specialist doctors answering specialist questions. One common shortcoming is the lack of clear and understandable responses to user queries, often lacking in human care and focusing solely on the disease, neglecting the emotional experiences of the users. Effective communication between doctors and patients is crucial in online consultation services, yet there is a lack of objective evaluation tools to assess the quality of electronic consultations in clinical practice. Therefore, it is essential to develop an electronic consultation quality evaluation scale that can serve as a basis for assessing the quality of dialogue in medical consultation services and further enrich the theoretical framework of online medical evaluation. Additionally, this scale will help improve the efficiency of communication between doctors and patients, enhance the quality of platform consultation services, advance the technology of platform electronic consultation services, and optimize the overall platform consultation environment. In constructing the electronic consultation quality evaluation scale, the Delphi method is employed. It is important to acknowledge that the research results may have a certain degree of subjectivity. Currently, in China’s internet medical environment, the emphasis is primarily on the consultation process, particularly the interaction with clinical doctors.It is worth noting that in China, online electronic consultation service platforms do not have dedicated pharmaceutical malls, and there is almost no communication with clinical pharmacists, which, to some extent, limits the selection of experts for this study. It is important to note that the evaluation indicators may evolve over time as new evidence emerges and our understanding of the quality of doctor-patient communication continues to advance. Therefore, continuous research and updates to the index system in the future will be necessary to ensure its relevance and effectiveness in evaluating the quality of e-consultation. Acknowledgments We grateful to all participants of the Delphi surveys and other experts who have participated in this process, provided guidance, or critically appraised our manuscript. Disclosure The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. References 1. Machengyu M. Analysis of the development status of online medical communities. Chin Hospit Manag. 2018;38(5):3. 2. Huihui H, Shu W, Di W, et al. PEST analysis and countermeasures research on the development of internet healthcare in China. Chin Health Care Manag. 2020;11:37. 3. Yuan F, Weijun H, Yanni Y, et al. Conducted a study on the perceived value of users in online medical community information consulting services. Chin Med Literat Informat J. 2019;28:12. 4. Jun D, Mingle H. Influencing factors of online medical community information service quality based on user perception. Information Science; 2019. 5. Jiang J, Yang M, Kiang M, et al. Exploring the freemium business model for online medical consultation services in China. Inf Process Manag. 2021;58(3):102515. doi:10.1016/j.ipm.2021.102515 6. Mallen CD, Peat G, Porcheret M. 10-minute consultation: chronic knee pain. BMJ Br Med J. 2007;2007:335. 7. Bai J. Internet+health solution and practice in public hospitals in 2016 - smart healthcare and big health industry development summit; 2016. 8. Chen X, Zhou X, Li H, et al. The value of WeChat application in chronic diseases management in China. Comput Methods Programs Biomed. 2020;196:105710. doi:10.1016/j.cmpb.2020.105710 9. Qi M, Cui J, Li X, Han Y. Perceived factors influencing the public intention to use e-consultation: analysis of web-based survey data (Preprint). J Med Internet Res. 2020;23:1. 10. Xu C . Research on the application of remote consultation in medical service institutions. Inform Commun. 2016;2016:8. 11. Tatsumi H, Mitani H, Haruki Y, Ogushi Y. Internet medical usage in Japan: current situation and issues. J Med Internet Res. 2001;3(1):e12. doi:10.2196/jmir.3.1.e12 12. Qi M, Cui J, Li X, Han Y. Influence of E-consultation on the intention of first-visit patients to select medical services: results of a scenario survey. J Med Internet Res. 2023;25(e 40993):e40993. doi:10.2196/40993 13. Bokolo AJ. Application of telemedicine and eHealth technology for clinical services in response to COVID‑19 pandemic. Health Technol (Berl). 2021;11(2):359–366. doi:10.1007/s12553-020-00516-4 14. Zhang W, Deng Z, Hong Z, et al. Unhappy patients are not alike: content analysis of the negative comments from china’s good doctor website. J Med Internet Res. 2018;20(1):e35. doi:10.2196/jmir.8223 15. Describing X, Hongbing T, Liting D, et al. Discussed the analysis and countermeasures for the quality and safety issues of mobile healthcare under the background of Internet+. Chin Health Qual Manag. 2017;24(3):4. 16. Shengwu L Conducted a study on the innovative service model for regional medical consortiums’ quality management under the background of “Internet+”. Chin Health Qual Manag. 2019;2019:1. 17. Linn LS, Brook RH, Clark VA, et al. Physician and patient satisfaction as factors related to the organization of internal medicine group practices. Med Care. 1985;23(10):1171–1178. doi:10.1097/00005650-198510000-00006 18. Stewart AL, Pringer AN, Pérez-Stable EJ. Interpersonal processes of care in diverse populations. Milbank Quarterly. 1999;77(3):305–339. doi:10.1111/1468-0009.00138 19. Elwyn G, Edwards A, Wensing M, et al. Shared decision making: developing the OPTION scale for measuring patient involvement. Qual Saf Health Care. 2003;12(2):93–99. doi:10.1136/qhc.12.2.93 20. Mercer SW, Maxwell M, Heaney D, Watt GC. The consultation and relational empathy (CARE) measure: development and preliminary validation and reliability of an empathy-based consultation process measure. Fam Pract. 2004;21(6):699–705. doi:10.1093/fampra/cmh621 21. Jie Z, Xiangfu G. Communication Skills Between Doctors and Patients. People’s Health Publishing House; 2015. 22. Chen L. Analysis of the influencing factors and countermeasures of the quality of doctor-patient communication management. J Tradit Chin Med Manag. 2020;28(7):2. 23. Liqun Z, Xin L. Practical research on the cultivation of medical communication ability to improve the quality of oral medicine students’ surgical internship. Chin Health Indust. 2020;17(17):3. 24. Jing X. An analysis of the current situation and possible reasons for the communication skills of medical students with patients. Higher Educ J. 2015;2015:1. 25. Tian Z, Li X, Wang S, et al., The mechanism and role of “internet plus” in promoting medical institutions to achieve social governance – taking medical colleges in qigihar as an example; 2020. 26. Yijia L. Study on Factors Influencing Patients’ Choice of Doctors in Online Medical Communities; 2021. Anhui Medical University. 27. Chengyu M. Empirical study on doctor-patient interaction behavior in online medical communities - taking haodf.com as an example. Chin J Health Polic Res. 2016;9(11):5. 28. Limandow A, Kaihatu TS. Perception of potential patients and doctors against doctors based online consulting; 2017. 29. Shamalova E, Kostromina E. Research of significant factors of choosing a medical organization for consumers of medical services in the Russian Federation. Bulletin of the Moscow University named S U Vitte Series 1 Economics and management; 2023. 30. Sirkis T, Maitland S. Monitoring real-time junior doctor sentiment from comments on a public social media platform: a retrospective observational study. Postgrad Med J. 2023;99:423–427. doi:10.1136/pmj-2022-142080 31. Maharjan RK, Koirala S, Maharjan R, Sherchand JB. Analysis of online medical services availability during covid-19 pandemic in Nepal. Tribhuv Univer J. 2020; 27:59–68. 32. George CE, Duquenoy P. Online Medical Consultations: Legal, Ethical, and Social Perspectives. Social Science Electronic Publishing; 2008. © 2023 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License .By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms .