Easy to pass poor riders, difficult to fail them?

I have the pass-fail data (though not the raw data) from two different years in one state program. Altogether it gives us a sample size of 5,619 which should be large enough to avoid the errors of a small sample size whether looking at both curriculums together or each separately. And when I looked at it after a couple years, I noticed some oddities that I think the rider ed community needs to consider.

What I noticed is particularly helpful for rider educators who want to see how effective one curriculum may be over another—and especially what kind of errors may be more associated with one curriculum over another.

As you see, it includes both MRC: RSS and BRC skills test results. This is not to compare the efficacy or difficulty of either curriculum but to examine oddities in the data—and to examine what it means to pass or fail the end of course evaluations:

Students Tested	2564	100%
Passed Riding	2218	86.51%
Failed Riding	346	13.49%

BRC:

Students Tested	3055	100%
Passed Riding	2722	89.10%
Failed Riding	333	10.90%

When we graph the test scores we see an unusual sawtooth appearance—almost like a drunkard’s walk. More on this later. But we also find that only 2.59% more students passed and failed the BRC students than the RSS:

In this way, it appears the courses—and evaluations—are equivalent. Some could say, as a result, that the BRC hasn’t been “dumbed down”. But it’s wise to look more carefully at the results before doing that:

What is clearly different between the two is that the BRC has more students that almost fail it than the RSS did:, 19. 47% of all BRC students and 13.57% of all RSS students got scores of between 18-20.

So, while it appears that the number of students who passed one curriculum or another is equivalent, there’s a statistically significant difference between the iterations in the percentage of those who just barely squeaked by. Iow, the BRC sent  almost 6% more poor riders

This becomes even more significant when we look at the very last possible point that a student can pass—a score of 20:

106 RSS students or 4.13% and 268 BRC students or 8.77% got an even 20.

Iow, more than twice the percentage of BRC students got the equivalent of a D- than  RSS students. Meaning more than twice the number of very poor riders were sent out into an increasingly complex and dangerous traffic environment with a driver’s license-waiver in their hands.

Is this just an oddity or does it indicate something about the two iterations?

The skills tests

For those who aren’t familiar with the end-of-course evaluations in both iterations, read on. (For those who are, skip to “What a difference one point makes”.)

The RSS skill test involved 5 scored evaluations: Sharp (90-degree) Turns, Cone Weave, Quick Stop, Turning Speed Selection (cornering), and Quick Lane Change (swerving).

The BRC skill test involves 4 evaluations: U-turn, Swerve, Quick Stop and Cornering.

The Sharp Turn and Cone Weave were dropped entirely and replaced with the U-Turn, the configuration of the corner was changed and the timing zone and timing chart were changed in the Quick Stop. Additionally, the U-turn is not a life-critical skill, so 40% fewer critical skills are tested in the BRC.

So let’s keep a running tally: 40% fewer critical skills tested in the BRC and 6% more poor riders passed with the last passing score: Iow, it’s possible that the poor riders may be even poorer than suspected.

A passing score is ≤20 and a failing score is ≥21.

In both iterations:

Evaluations are given separately but consecutively and

Each skill was/is scored separately.

The student had/has two opportunities to take each test and

Points are not added until after the second run (therefore, if they make two errors in the first trial and another in the second, only the second error is counted).

In both iterations, points are assessed for a variety of errors but each evaluation had/has a maximum number of points that can be assessed:

In the RSS the maximum was identical to the Alt-MOST used at the DMV—10 points max in each of the 5 evaluations.

In the BRC, the U-Turn has a maximum of 8 points and the other three evaluations have a maximum of 15 points—or a 50% increase in allowable points for each of the tested skills that comprise 80% of the evaluation.

Both tests used a 1, 3, 5, 10 and 15-point scoring system—but they used that system somewhat differently. In the RSS, 3-points could be assessed in 4 of the 5 evaluations—there were 12 errors that scored 3 points. In the BRC, there’s only one skill (U-Turn) that has two possible errors each worth 3 points each (if done once).

The Quick Stop—a critical life-saving skill—also assesses 1 point for every foot beyond the stopping point that the student travels up to a maximum of 10’—but this is not in addition to the maximum points but includes it.

In 3 out of the four evaluations, students must demonstrate the required skill at  between 12 and 18 mph. In the Quick Stop evaluation, for example, if they go under 12 mph, they can collect 5 points—but if they make the stopping distance at speeds higher than 18 mph, no points are assessed. In Cornering, they are supposed to get up to about 20 mph prior to slowing for the corner but the student will not lose points if they don’t go that fast.

The highest score possible in either iteration would be 53 (15+15+15+8=53)—though the actual number of points accrued can be much higher.

What a difference one point makes

One of the obvious things about the above graph is the huge difference between the last possible passing score of 20 and the exceedingly few students that got the first failing score of 21 in both iterations—but this is more far more evident in the BRC.

In fact, while 8.77% of BRC students got 20 points only 1.08% got just one point more and failed the course while 4.13% RSS students got 20 points only 1.32% got 21 points.

This is another significant change between the two iterations. While the percentage of those who barely scrape by more than doubled, the percentage of those who just failed changed from about 4 to 1 to 8 to 1.

And yet, the overall percentage of those who passed didn’t change in a statistically significant manner from one iteration to another.

Iow, while it appears by overall pass scores that the two curriculums were equivalent, they weren’t—40% fewer critical skills were tested and 6% more got the last possible score that enabled them to pass and the number of people who just failed sank like a stone. And that’s significant any way you look at it.

It would be interesting to see how other states measure up if their data was examined and see if this bulge of barely passed and dearth of just failed also exists–and if this patterns is also found in those states that use TEAM Oregon’s BRT.

Tomorrow’s entry reveals that more than 27% of BRC students who pass the test make the exact same potentially lethal error compared to just over 16% of RSS students—and that’s a dangerous change between the two iterations.

Explore posts in the same categories: Instructors, Motorcycle Safety, Motorcycle Safety Foundation, Motorcycle Training, State Motorcycle Safety Programs, Uncategorized