Easy to pass poor riders, difficult to fail them?

I have the pass-fail data (though not the raw data) from two different years in one state program. Altogether it gives us a sample size of 5,619 which should be large enough to avoid the errors of a small sample size whether looking at both curriculums together or each separately. And when I looked at it after a couple years, I noticed some oddities that I think the rider ed community needs to consider.

What I noticed is particularly helpful for rider educators who want to see how effective one curriculum may be over another—and especially what kind of errors may be more associated with one curriculum over another.

As you see, it includes both MRC: RSS and BRC skills test results. This is not to compare the efficacy or difficulty of either curriculum but to examine oddities in the data—and to examine what it means to pass or fail the end of course evaluations:

Students Tested

2564

100%

Passed Riding

2218

86.51%

Failed Riding

346

13.49%

BRC:

Students Tested

3055

100%

Passed Riding

2722

89.10%

Failed Riding

333

10.90%

When we graph the test scores we see an unusual sawtooth appearance—almost like a drunkard’s walk. More on this later. But we also find that only 2.59% more students passed and failed the BRC students than the RSS:

In this way, it appears the courses—and evaluations—are equivalent. Some could say, as a result, that the BRC hasn’t been “dumbed down”. But it’s wise to look more carefully at the results before doing that:

What is clearly different between the two is that the BRC has more students that almost fail it than the RSS did:, 19. 47% of all BRC students and 13.57% of all RSS students got scores of between 18-20.

So, while it appears that the number of students who passed one curriculum or another is equivalent, there’s a statistically significant difference between the iterations in the percentage of those who just barely squeaked by. Iow, the BRC sent  almost 6% more poor riders

This becomes even more significant when we look at the very last possible point that a student can pass—a score of 20:

106 RSS students or 4.13% and 268 BRC students or 8.77% got an even 20.

Iow, more than twice the percentage of BRC students got the equivalent of a D- than  RSS students. Meaning more than twice the number of very poor riders were sent out into an increasingly complex and dangerous traffic environment with a driver’s license-waiver in their hands.

Is this just an oddity or does it indicate something about the two iterations?

The skills tests

For those who aren’t familiar with the end-of-course evaluations in both iterations, read on. (For those who are, skip to “What a difference one point makes”.)

The RSS skill test involved 5 scored evaluations: Sharp (90-degree) Turns, Cone Weave, Quick Stop, Turning Speed Selection (cornering), and Quick Lane Change (swerving).

The BRC skill test involves 4 evaluations: U-turn, Swerve, Quick Stop and Cornering.

The Sharp Turn and Cone Weave were dropped entirely and replaced with the U-Turn, the configuration of the corner was changed and the timing zone and timing chart were changed in the Quick Stop. Additionally, the U-turn is not a life-critical skill, so 40% fewer critical skills are tested in the BRC.

So let’s keep a running tally: 40% fewer critical skills tested in the BRC and 6% more poor riders passed with the last passing score: Iow, it’s possible that the poor riders may be even poorer than suspected.

A passing score is ≤20 and a failing score is ≥21.

In both iterations:

Evaluations are given separately but consecutively and

Each skill was/is scored separately.

The student had/has two opportunities to take each test and

Points are not added until after the second run (therefore, if they make two errors in the first trial and another in the second, only the second error is counted).

In both iterations, points are assessed for a variety of errors but each evaluation had/has a maximum number of points that can be assessed:

In the RSS the maximum was identical to the Alt-MOST used at the DMV—10 points max in each of the 5 evaluations.

In the BRC, the U-Turn has a maximum of 8 points and the other three evaluations have a maximum of 15 points—or a 50% increase in allowable points for each of the tested skills that comprise 80% of the evaluation.

Both tests used a 1, 3, 5, 10 and 15-point scoring system—but they used that system somewhat differently. In the RSS, 3-points could be assessed in 4 of the 5 evaluations—there were 12 errors that scored 3 points. In the BRC, there’s only one skill (U-Turn) that has two possible errors each worth 3 points each (if done once).

The Quick Stop—a critical life-saving skill—also assesses 1 point for every foot beyond the stopping point that the student travels up to a maximum of 10’—but this is not in addition to the maximum points but includes it.

In 3 out of the four evaluations, students must demonstrate the required skill at  between 12 and 18 mph. In the Quick Stop evaluation, for example, if they go under 12 mph, they can collect 5 points—but if they make the stopping distance at speeds higher than 18 mph, no points are assessed. In Cornering, they are supposed to get up to about 20 mph prior to slowing for the corner but the student will not lose points if they don’t go that fast.

The highest score possible in either iteration would be 53 (15+15+15+8=53)—though the actual number of points accrued can be much higher.

What a difference one point makes

One of the obvious things about the above graph is the huge difference between the last possible passing score of 20 and the exceedingly few students that got the first failing score of 21 in both iterations—but this is more far more evident in the BRC.

In fact, while 8.77% of BRC students got 20 points only 1.08% got just one point more and failed the course while 4.13% RSS students got 20 points only 1.32% got 21 points.

This is another significant change between the two iterations. While the percentage of those who barely scrape by more than doubled, the percentage of those who just failed changed from about 4 to 1 to 8 to 1.

And yet, the overall percentage of those who passed didn’t change in a statistically significant manner from one iteration to another.

Iow, while it appears by overall pass scores that the two curriculums were equivalent, they weren’t—40% fewer critical skills were tested and 6% more got the last possible score that enabled them to pass and the number of people who just failed sank like a stone. And that’s significant any way you look at it.

It would be interesting to see how other states measure up if their data was examined and see if this bulge of barely passed and dearth of just failed also exists–and if this patterns is also found in those states that use TEAM Oregon’s BRT.

Tomorrow’s entry reveals that more than 27% of BRC students who pass the test make the exact same potentially lethal error compared to just over 16% of RSS students—and that’s a dangerous change between the two iterations.

Advertisements
Explore posts in the same categories: Instructors, Motorcycle Safety, Motorcycle Safety Foundation, Motorcycle Training, State Motorcycle Safety Programs, Uncategorized

10 Comments on “Easy to pass poor riders, difficult to fail them?”

  1. Jeff Brenton Says:

    Just a nit or two on scoring the BRC…

    Too slow on the braking (second attempt) is 15 points, as is anticipating the stop point on a second attempt. And there is a third place you can pick up 3 points in the BRC evaluation – cornering too slowly can be 1, 3 or 5 points, depending upon speed.

  2. vstromer Says:

    I’m not being critical of the BRC riding evaluation here, just providing information on what I see. (In my state the U-turn max points is 10, not 8, different from MSF standard.) To pass the evaluation a student can completely blow the U-turn. That’s -10. Typically even weak students complete the swerve with no points off, it really isn’t that hard. So, still at -10. In the quick stop if they can stop within 5 feet of standard and don’t make any other mistakes (use both brakes, down shift to first gear) all they lose is 5 more points. Now they’re at -15. In the cornering eval, if they use both brakes, keep their eyes up, and go slow enough to stay between the lines and not decelerate, they lose 5 more points. That’s -20, which is a pass. This is a very typical score. Or if they are only, say, 3 feet long in the braking, then its an 82. I see a lot of passes in the eval with scores or 80-82 just as I’ve described above. It isn’t unusual in a class of 12 to have 3 or 4 scores in the 80-82 range.

  3. wmoon Says:

    Ah, yes, now I see it tucked down in the corner of the form. I have no idea how common going too slow through a corner is–others want to share their observations on that? However, I would suggest that few if any riders are killed by going too slowly through a corner in real life. But you are correct that it could result in some of those numbers. If going too slowly is a big problem (and considering just how slow they can go and not be too slow), then I’d suggest the instructors have seriously failed to teach those students adequately. I’m also wondering what combination of errors are typical on a test.

  4. Jeff Brenton Says:

    On average, about half the class will pick up at least one point on the corner. A constant 11MPH circle through the timing zone nets zero or one point, depending upon the reflexes of the timer, with a target speed of 12-17 MPH. What more students pick up points on is continuing their deceleration through the curve.

    It’s also possible to do the swerve evaluation (RSS or BRC) without actually doing a swerve; I’ve demonstrated a path of travel that I’ve seen students do that garners zero points, but doesn’t involve actually demonstrating a swerve.

  5. Frank Pennucci Says:

    just sort of wondering aloud……
    could the difference be accounted for by stricter grading? I don’t have a horse in this race…. just that I’ve learned …statistics mean pretty much what the analyst makes of them

  6. wmoon Says:

    I, too, don’t have a horse in this race. The standards were higher in the RSS, true–but that only confirms the notion that the BRC allows poorer students to pass.
    W.

  7. Gary Suarez Says:

    There are no “second chances” for the U-Turn or Cornering evals.
    The student gets a second try in the swerve if (a) the speed is too slow and there was no other infraction, or (b)anticipation, or (c) the speed is too fast and there was an infraction.
    A rerun is allowed on the quick stop if (a)speed is too slow, or (b) anticipation, or (c) speed is too fast and the maximum stopping distance is not achieved.

    For what it’s worth, the great State of South Carolina, as part of the DMV test, adds the following to the evaluation–weave, sharp turn, and regular stop. MSF passing requires only a passing score on the 4 MSF evaluations. License waiver requires a satisfactory score on all 7 evaluations.

    My $.02 (after inflation) worth.

  8. gymnast Says:

    Just a note:
    It appears that ABATE of Indiana has completely taken over the Indiana rider training (MSF) program as well as all motorcycle license testing. The former State Program Coordinator was laid off in what appears to be a “budget crunch”-or something.

  9. wmoon Says:

    Yes, John Bodeker lost his job with no notice at all and given only a few hours to pack up and get out. The reason given was, as you said, budget considerations. He was replaced by someone with absolutely no experience or training or even familiarity with motorcycle training. Indiana ABATE already did the actual student training–and a good job of it, too. I wouldn’t be surprised if there’s further changes in store for Indiana…
    W.

  10. Mark Weiss Says:

    The cornering error is not actually that the riders are moving too slowly, but that they misjudge their entry speed. The evaluation is designed primarily to test judgment. If the rider enters even a tad too quick for their skill level, they’ll record a slow time.

    The key to the cornering evaluation is to set a slow entry speed. Almost painfully slow. If this is done, even very moderate acceleration will get the rider through with zero penalties. This is fairly easy and most students do this quite well during practice.

    If the rider’s entry speed is a bit higher, they’ll have to have sufficient skill to maintain their path of travel, and increase their speed, as they travel the curve. This the common error on the test. Entry speed is a bit high, but not quick enough to travel the path with a zero score. A novice rider is usually not willing to accelerate from what they perceive as a ‘high speed’ and are therefore penalized for ‘too slow’.

    Mark Weiss


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: