400+ Validation Studies
Evidence-based Simulation Training
Find here a selection of validation studies, the culmination of extensive research and rigorous validation processes providing evidence of the validity and reliability of our simulation technology, which helped pave the way for revolutionizing surgical education and training. Based on some of the studies we have established proficiency-based curricula which are integrated into our simulators.
Filter by Simulator
Background: There is a lack of evidence regarding the effectiveness of virtual simulators as a means to acquire hands-on exposure to endoscopic retrograde cholangiopancreatography (ERCP). The present study aimed to assess the outcome and construct validity of virtual ERCP when training on the GI II Mentor simulator.
Methods: A group of seven experienced endoscopists were compared with 31 novices. After a short introduction, they were requested to carry out three virtual ERCP procedures: diagnosing and removing a common bile duct (CBD) stone; diagnosing and taking brush cytology from a hilar stenosis; and, finally, diagnosing and treating a cystic leakage with a BD stent. For each task, the total time required to complete the task, time required to correctly view the papilla, total time of irradiation, time to deep cannulation, time to define diagnosis, time to complete sphincterotomy, and time to complete the respective intervention were measured. Cannulation of the BD, correct diagnosis, sphincterotomy, and time to complete intervention were assessed by an assessor blinded to the status of the endoscopist who performed the virtual ERCP.
Results: The time required to visualize the papilla and to cannulate deeply when removing the BD stone was significantly shorter for the experts (both p < 0.05). The time to visualize the papilla, cannulate deeply, reach a diagnosis, complete sphincterotomy, and complete the intervention was significantly shorter for the experts when managing cystic leakage (all p < 0.05). In diagnosing and taking brush cytology from a hilar stenosis, there was only a trend toward the experts needing less time for the deep cannulation of the BD (p = 0.077).
Conclusion: The performance differed between experts and novices, especially in the management of cystic leakage. This corroborates the construct validity of the GI II Mentor simulator.
Background The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score.
Methods Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach’s alpha. Test–retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves.
Results A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants’ level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach’s alpha) was 0.82. Test–retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21.
Conclusions The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.
Introduction: Surgical residents have learned flexible endoscopy by practicing on patients in hospital settings under the strict guidance of experienced surgeons. Simulation is often used to “pretrain” novices on endoscopic skills before real clinical practice; nonetheless, the optimal method of training remains unknown. The purpose of this study was to compare endoscopic virtual reality and physical model simulators and their respective roles in transferring skills to the clinical environment.
Methods: At the beginning of a skills development rotation, 27 surgical postgraduate year 1 residents performed a baseline screening colonoscopy on a real patient under faculty supervision. Their performances were scored using the Global Assessment of Gastrointestinal Endoscopic Skills (GAGES). Subsequently, interns completed a 3-week flexible endoscopy curriculum developed at our institution. One-third of the residents were assigned to train with the GI Mentor simulator, one-third trained with the Kyoto simulator, and one-third of the residents trained using both simulators. At the end of their rotations, each postgraduate year 1 resident performed one posttest colonoscopy on a different patient and was again scored using GAGES by an experienced faculty.
Results: A statistically significant improvement in the GAGES total score (p < 0.001) and on each of its subcomponents (p = 0.001) was observed from pretest to posttest for all groups combined. Subgroup analysis indicated that trainees in the GI Mentor or both simulators conditions showed significant improvement from pretest to posttest in terms of GAGES total score (p = 0.017 vs 0.024, respectively). This was not observed for those exclusively using the Kyoto platform (p = 0.072). Nonetheless, no single training condition was shown to be a better training modality when compared to others in terms of total GAGES score or in any of its subcomponents.
Conclusion: Colonoscopy simulator training with the GI Mentor platform exclusively or in combination with a physical model simulator improves skill performance in real colonoscopy cases when measured with the GAGES tool.
Keywords: GAGES; Patient Care; Practice-Based Learning and Improvement; Systems-Based Practice; flexible endoscopy; simulation; skills assessment.
Background Fundamentals of Endoscopic Surgery (FES) has become a board certification requirement for general surgery residents. While the FES program provides a robust didactic curriculum, an endoscopic skills curriculum is lacking for this high stakes assessment. The aims of this study were to develop a proficiency-based endoscopic skills curriculum and assess its effectiveness on success in the FES exam.
Methods Endoscopy experts developed a multiple-choice questionnaire based on the FES online didactics. Five training cases from the GI Mentor II simulator were selected, and expert performance defined proficiency levels for each case. Participating surgery residents were required to review online didactics and achieve proficiency twice on selected simulator cases. The multiple-choice questionnaire, simulator-generated metrics of two endoscopy cases, Global Assessment of Gastrointestinal Endoscopic Skills (GAGES), NASA-Task Load Index (TLX), and the manual portion of the FES exam were used for assessment before and after training. The curriculum was implemented either alongside a clinical endoscopy rotation or independent of the rotation. Clinical endoscopic skills of participants with a dedicated rotation were assessed using GAGES.
Results Twenty-eight general surgery residents (PGY 2–5) participated in the study, of which 25 (89%) completed the curriculum. Scores of the multiple-choice questionnaire and all simulator-generated metrics improved in the post-training assessment, with the exception of the percentage of mucosal surface examined, which was slightly decreased. Simulated and clinical GAGES scores and the NASA-TLX score improved after training. Performance scores on four of five FES exam tasks were significantly improved.
Conclusions The proficiency-based endoscopic skills curriculum was successfully implemented both alongside the clinical endoscopy rotation and independent of the rotation. Participating residents acquired skills to pass the FES exam. This curriculum will be valuable to general surgery residency programs
Background and Study Objectives: Operator competency is essential for esophagogastroduodenoscopy (EGD) quality, which makes appropriate training with a final test important. The aims of this study were to develop a test for assessing skills in performing EGD, gather validity evidence for the test, and establish a credible pass/fail score.
Methods: An expert panel developed a practical test using the Simbionix GI Mentor II simulator (3 D Systems) and an EGD phantom (OGI 4, CLA Medical) with a diagnostic (DP) and a technical skills part (TSP) for a prospective validation study. During the test a supervisor measured: 1) total time; 2) degree of mucosal visualization; and 3) landmarks and pathology identification. The contrasting groups standard setting method was used to establish a pass/fail score.
Results: We included 15 novices (N), 10 intermediates (I), and 10 experienced endoscopists (E). The internal structure was high with a Cronbach’s alpha of 0.76 for TSP time consumption and 0.74 for the identification of landmarks.
Mean total times, in minutes, for the DP were N 15.7, I 11.3, and E 7.0, and for TSP., they were N 7.9, I 8.9, and E 2.9. The total numbers of identified landmarks were N 26, I 41, and E 48. Mean visualization percentages were N 80, I 71, and E 71. A pass/fail standard was established requiring identification of all landmarks and performance of the TSP in < 5 minutes. All experienced endoscopists passed, while none of the endoscopists in the other categories did.
Conclusions: We established a test that can distinguish between participants with different competencies. This enables an objective and evidence-based approach to assessment of competencies in EGD.
Background: After the American Board of Surgery announcement of the Flexible Endoscopy Curriculum requirement in 2014, we implemented a dedicated endoscopy rotation at the post graduate year (PGY)2 level including a simulation curriculum for Fundamentals of Endoscopic Surgery skills. Here we evaluate the outcomes of this implementation.
Methods: Beginning in 2015, we developed a clinical endoscopy and simulation-based rotation to prepare for Fundamentals of Endoscopic Surgery testing. Originally, our curriculum was based on the published Texas Association of Surgical Skills Laboratories curriculum using the GI Mentor and transitioned to a mastery learning curriculum using the Endoscopy Training System in 2016. We evaluated the success of the curriculum in terms of first-time pass rates, training time required, and comparison to previously published benchmarks based on clinical experience.
Results: Since 2015, a total of 37 general surgery residents in our program were Fundamentals of Endoscopic Surgery tested (PGY2 = 24, PGY3 = 4, PGY5 = 9); 84% (31) completed the Endoscopy Training System curriculum. At the time of testing, 73% (27) had performed <25 esophagogastroduodenoscopies, and 46% had performed <25 colonoscopies. Ninety-two percent (34) spent 10 hours or less completing the curriculum. The first-time pass rate for those completing the Endoscopy Training System curriculum was 97% vs 67% for those not completing the Endoscopy Training System curriculum (P = .01). For residents completing the Endoscopy Training System curriculum, total Fundamentals of Endoscopic Surgery scores were discernibly higher (472 vs 389, P < .01), as were 3/5 task scores (Nav1 80 vs 67, P = .02; Loop2 36 vs 8, P = .02; Retro3 89 vs 71, P = .02). Despite clinical inexperience (<25 esophagogastroduodenoscopies and <50 colonoscopies), PGY2s yielded a mean score of 454 and a pass rate of 92%. This was similar to PGY5s (427, 89%; P = .3) and compares to benchmark data of endoscopists with >300 cases.
Conclusion: Early implementation of flexible endoscopy training with a simulation-based curriculum results in Fundamentals of Endoscopic Surgery performance equal to a clinical experience not often gained during surgical residency. Often requiring <10 hours, this represents a fantastic return on investment for this training.