• Prehosp Emerg Care · Jan 1998

    Evaluating the evaluators: interrater reliability on EMT licensing examinations.

    • W Snyder and S Smit.
    • Mercy Memorial Hospital, Monroe, MI 48162, USA. snyder@mailgw-vtc.oto.med.umich.edu
    • Prehosp Emerg Care. 1998 Jan 1;2(1):37-46.

    IntroductionCurrent methods of evaluating the technical competence of Michigan emergency medical technician (EMT) licensure candidates are subjective and potentially unreliable. Evaluators are required to attend a workshop before evaluating practical examination candidates. Despite the workshop, there is too much score variation and not enough observational consistency on the standardized examination.ObjectiveTo determine the level of rater reliability for evaluators of EMT practical examinations.MethodsData were collected from 104 licensed instructor-coordinators (ICs). Participants watched and scored two practical examinations simulated on videotape, one passing and one failing performance. Variation in student score and level of evaluator agreement concerning skill performance were determined. Michigan's basic EMT practical examination scoring instrument was used.ResultsNine basic EMT, 9 EMT specialist, and 86 EMT-paramedic ICs participated. Thirty-four percent had high school diplomas, 43% associate's degrees, 19% bachelor's degrees, 3% master's degrees, and 1% a doctoral degree. The ICs averaged 14 years of provider experience, and 6.35 years as an IC. The average score for scenario 1 (passing) was 86.4% (SD = 9.15, range = 53). An 80% score is required to pass. Nine ICs (9%) failed the student. Scenario 1 evaluator agreement was 79.4%. The average score for scenario 2 (failing) was 60.9% (SD = 11.57, range = 57). Seventeen ICs (18%) gave the student a passing score. Scenario 2 evaluator agreement was 67.8%. There was no significant difference in student scores based on evaluator level of education, licensure, or evaluator workshop attendance for either scenario.ConclusionNotable variation in scores given by evaluators for a single observed student, combined with low levels of evaluator agreement about skill performance, suggests that evaluators do not reliably rate student performance using the Michigan practical examination instrument.

      Pubmed     Copy Citation     Plaintext  

      Add institutional full text...

    Notes

     
    Knowledge, pearl, summary or comment to share?
    300 characters remaining
    help        
    You can also include formatting, links, images and footnotes in your notes
    • Simple formatting can be added to notes, such as *italics*, _underline_ or **bold**.
    • Superscript can be denoted by <sup>text</sup> and subscript <sub>text</sub>.
    • Numbered or bulleted lists can be created using either numbered lines 1. 2. 3., hyphens - or asterisks *.
    • Links can be included with: [my link to pubmed](http://pubmed.com)
    • Images can be included with: ![alt text](https://bestmedicaljournal.com/study_graph.jpg "Image Title Text")
    • For footnotes use [^1](This is a footnote.) inline.
    • Or use an inline reference [^1] to refer to a longer footnote elseweher in the document [^1]: This is a long footnote..

    hide…