400+ Validation Studies

Background

Point-of-care ultrasound is rapidly gaining traction in clinical practice, including primary care. Yet, logistical challenges and geographical isolation hinder skill acquisition. Concurrently, an evidentiary gap exists concerning such guidance’s effectiveness and optimal implementation in these settings.

Methods

We developed a lung point-of-care ultrasound (POCUS) curriculum for primary care physicians in a rural, medically underserved region of the south of Israel. The course included recorded lectures, pre-course assessments, hands-on training, post-workshop lectures, and individual practice. To evaluate our course, we measured learning outcomes and physicians’ proficiency in different lung POCUS domains using hands-on technique assessment and gathered feedback on the course with a multi-modal perception approach: an original written pre- and post-perception and usage questionnaire.

Results

Fifty primary care physicians (PCPs) showed significant improvement in hands-on skills, increasing from 6 to 76% proficiency (p < 0.001), and in identifying normal versus abnormal views, improving from 54 to 74% accuracy (p < 0.001). Ten weeks after training, primary care physicians reported greater comfort using lung ultrasound, rising from 10 to 54% (p < 0.001), and improved grasp of its potential and limits, increasing from 27.5% to 84% (p < 0.001). Weekly usage increased from none to 50%, and the number of primary care physicians not using at all decreased from 72 to 26% (p < 0.001).

Conclusions

A two-day focused in-person and remote self-learning lung-POCUS training significantly improved primary care physicians’ lung ultrasound skills, comfort, and implementation.

Background: Basic thoracic ultrasound is being used more frequently by clinicians in several settings due to its high diagnostic accuracy for many common causes of respiratory failure and dyspnoea. However, ultrasound examinations are operator-dependent, and sufficient competences are needed to obtain high sensitivity and specificity of examinations. Additionally, it is crucial for ultrasound operators to perceive the competence to interpret the images and integrate them into the patient history and other examinations. This study aims to explore and gather validity evidence for an objective structured clinical examination test of basic thoracic ultrasound competences and establish a pass/fail score.

Methods An expert panel created the test which included two theoretical and five practical stations representing cases with different diagnoses that cause respiratory symptoms and which are possible to establish by basic thoracic ultrasound. Twenty-five participants with different levels of experience in basic thoracic ultrasound completed the test. Data of the test scores were used for item analysis, and exploring validity evidence was done according to Messick’s framework which is recommended. The contrasting groups’ standard-setting method was used to establish a pass/fail score.

Results The summarised internal consistency reliability was high with a Cronbach’s alpha of 0.87. The novice group (n = 4) had a mean test score of 42 ± 10.1 points, the intermediate group (n = 8) scored 79.1 ± 8.1 points, and the experienced group (n = 13) 89.0 ± 6.2 points (one-way ANOVA, p < 0.001). A pass/fail score of 71 points was thus derived (maximum test score = 105 points).

Conclusion We developed a test for the assessment of clinical competences in basic thoracic ultrasound with solid validity evidence, and a pass/fail standard with no false positives or false negatives.

Objective:  To assess the effectiveness of simulation-based teaching vs traditional teaching of TEE knowledge and skills of cardiology fellows.

Design, Setting, and Participants:  Between November 2020 and November 2021, all consecutive cardiology fellows inexperienced in TEE from 42 French university centers were randomized (1:1; n = 324) into 2 groups with or without simulation support.

Main Outcomes and Measures:  The co-primary outcomes were the scores in the final theoretical and practical tests 3 months after the training. TEE duration and the fellows’ self-assessment of their proficiency were also assessed.

Results:  While the theoretical and practical test scores were similar between the 2 groups (324 participants; 62.6% male; mean age, 26.4 years) before the training (33.0 [SD, 16.3] points vs 32.5 [SD, 18.5] points; P = .80 and 44.2 [SD, 25.5] points vs 46.1 [SD, 26.1] points; P = .51, respectively), the fellows in the simulation group (n = 162; 50%) displayed higher theoretical test and practical test scores after the training than those in the traditional group (n = 162; 50%) (47.2% [SD, 15.6%] vs 38.3% [SD, 19.8%]; P < .001 and 74.5% [SD, 17.7%] vs 59.0% [SD, 25.1%]; P < .001, respectively). Subgroup analyses showed that the effectiveness of the simulation training was even greater when performed at the beginning of the fellowship (ie, 2 years or less of training) (theoretical test: an increase of 11.9 points; 95% CI, 7.2-16.7 vs an increase of 4.25 points; 95% CI, −1.05 to 9.5; P = .03; practical test: an increase of 24.9 points; 95% CI, 18.5-31.0 vs an increase of 10.1 points; 95% CI, 3.9-16.0; P < .001). After the training, the duration to perform a complete TEE was significantly lower in the simulation group than in the traditional group ( 8.3 [SD, 1.4] minutes vs 9.4 [SD, 1.2] minutes; P < .001, respectively). Additionally, fellows in the simulation group felt more ready and more confident about performing a TEE alone after the training (mean score, 3.0; 95% CI, 2.9-3.2 vs mean score, 1.7; 95% CI, 1.4-1.9; P < .001 and mean score, 3.3; 95% CI, 3.1-3.5 vs mean score, 2.4; 95% CI, 2.1-2.6; P < .001, respectively).

Conclusions and Relevance:  Simulation-based teaching of TEE showed a significant improvement in the knowledge, skills, and self-assessment of proficiency of cardiology fellows, as well as a reduction in the amount of time needed to complete the examination. These results should encourage further investigation of clinical performance and patient benefits of TEE simulation training.

Background:  Basic thoracic ultrasound is being used more frequently by clinicians in several settings due to its high diagnostic accuracy for many common causes of respiratory failure and dyspnoea. However, ultrasound examinations are operator-dependent, and sufficient competences are needed to obtain high sensitivity and specificity of
examinations. Additionally, it is crucial for ultrasound operators to perceive the competence to interpret the images and integrate them into the patient history and other examinations. This study aims to explore and gather validity evidence for an objective structured clinical examination test of basic thoracic ultrasound competences and establish
a pass/fail score.

Methods: An expert panel created the test which included two theoretical and five practical stations representing cases with different diagnoses that cause respiratory symptoms and which are possible to establish by basic thoracic ultrasound. Twenty-five participants with different levels of experience in basic thoracic ultrasound completed
the test. Data of the test scores were used for item analysis, and exploring validity evidence was done according to Messick’s framework which is recommended. The contrasting groups’ standard setting method was used to establish a pass/fail score.

Results:  The summarised internal consistency reliability was high with a Cronbach’s alpha of 0.87. The novice group (n = 4) had a mean test score of 42 ± 10.1 points, the intermediate group (n = 8) scored 79.1 ± 8.1 points, and the experienced group (n = 13) 89.0 ± 6.2 points (one-way ANOVA, p < 0.001). A pass/fail score of 71 points was thus
derived (maximum test score = 105 points).

Conclusion: We developed a test for the assessment of clinical competences in basic thoracic ultrasound with solid validity evidence, and a pass/fail standard with no false positives or false negatives.

Purpose: To analyze the learning curves of ultrasound novices in fetal echocardiography during structured simulation-based ultrasound training (SIM-UT) including a virtual, randomly moving fetus.

Methods: 11 medical students with minimal (< 10 h) prior obstetric ultrasound experience underwent 12 h of structured fetal echocardiography SIM-UT in individual hands-on sessions during a 6-week training program. Their learning progress was assessed with standardized tests after 2, 4, and 6 weeks of SIM-UT. Participants were asked to obtain 11 fetal echocardiography standard planes (in accordance with ISUOG and AHA guidelines) as quickly as possible. All tests were carried out under real life, examination-like conditions on a healthy, randomly moving fetus. Subsequently, we analyzed the rate of correctly obtained images and the total time to completion (TTC). As reference groups, 10 Ob/Gyn physicians (median of 750 previously performed Ob/Gyn scans) and 10 fetal echocardiography experts (median of 15,000 previously performed Ob/Gyn scans) were examined with the same standardized tests.

Results: The students showed a consistent and steady improvement of their ultrasound performance during the training program. After 2 weeks, they were able to obtain > 95% of the standard planes correctly. After 6 weeks, they were significantly faster than the physician group (p < 0.001) and no longer significantly slower than the expert group (p = 0.944).

Conclusion: SIM-UT is highly effective to learn fetal echocardiography. Regarding the acquisition of the AHA/ISUOG fetal echocardiography standard planes, the students were able to reach the same skill level as the expert group within 6 weeks.

Objectives: To compare the contribution of training using virtual simulation with theoretical instruction alone in the learning of pelvic ultrasound for gynecological emergencies in non-experienced students.

Methods: A unicentre randomised controlled trial was conducted. The participants were students in the last year of medical studies during their internship in the obstetrics and gynecology department at the University Hospital Center of Angers. Twenty participants were randomised to a 1:1 ratio in the training group and control group of 10 students each. All participants received a two hours theoretical course on the physical basis of ultrasound, the use of probes, as well as the basic principles of an trans-vaginal ultrasound examination centreed on the different standardised images to be given to gynecologic emergencies. For the training group an additional 20-minute pelvic ultrasound training on a trans-vaginal virtual simulator was performed. The participants were evaluated on images taken from patients consulting our gynecological emergencies centre. The pictures evaluated were sagittal and coronal sections of the uterus, left and right ovarian sections and a Morrison pouch view. The primary outcome was based on picture quality scores described by Salomon et al. with the French Society for the Improvement of Ultrasound Practices (SFAPE) and by Popowski et al. for Standardisation Acute Female Echography (SAFE). Secondaries outcomes focused on the general ultrasound skills and the duration of acquisition of the pictures.

Results: The mean SFAPE and SAFE scores were significantly higher in the training group than in the control group (14.5 ± 3.1, p = 0.046 and 10.1 ± 2.08, p = 0.016, respectively).

Conclusions: After a very short training session, using virtual simulation, inexperienced students produce proper pictures for the management of gynecological emergencies and better quality ultrasound pictures than students trained only theoretically.

Background: Clinical lung ultrasound (LUS) is a fast bedside diagnostic tool which can assist clinicians in decisions regarding the treatment and monitoring of patients with respiratory symptoms. LUS training and education differ widely, and is often done in a clinical setting, with potential risks for patients if decisions are made based on the wrong interpretations. No clear guidelines or recommendations for objective and standardized assessment of LUS skills exist, and those that do are often based on a fixed time-frame or an arbitrary number of examinations performed; this does not ensure adequate competencies.

Objectives: The study aimed to develop and gather validity evidence for a practical, simulation-based test in LUS.

Methods: Nine cases were developed in collaboration with 3D Systems Healthcare, Littleton, CO, USA, representing the most common diagnosis and sonographic findings in patients with respiratory symptoms. Thirty-six participants with different levels of competence in LUS, completed the test. The participants were divided into groups, i.e., novices, intermediates, and experienced, according to their experience with LUS, the number of examinations they had performed, and any research they had conducted. Their answers were used for item analyses.

Results: The intraclass correlation coefficient, Cronbachs’ α, was 0.69 summarized, and there was a statistically significant difference (p < 0.001) between the novices and the trained participants (intermediates and experienced). A pass/fail score of 16 points was calculated according to the contrasting-groups method.

Conclusion: We developed a test for the assessment of clinical competencies in LUS. The test proved solid validity evidence, and a pass/fail standard without any false-negatives, and only 2 explained false-positives.