Route Ratings Changed to Consensus

Today I changed the site so that the consensus grade is the official grade that shows up in search results, routes lists, etc. The “original” grade (what the submitter rated it) is still available and listed on the route page.

MP's primary purpose is a route guide: help people find and enjoy climbs suited to their ability. Though I expect there to be a few problems with consensus grades, I believe that, on average, the “wisdom of the crowds” will be better than a single opinion.

This idea has been discussed many times in the forum, and two good concerns repeatedly come up:

1. Outlying values: the consensus is calculated as the MEDIAN of the all ratings submitted, including the “original” rating, which is fairly insensitive to outliers. This is different from the AVERAGE of all ratings.

2. People only submit ratings if different from the original. If we have a problem with the new system, I can re-calculate the consensus and overweight the "original" rating so that it biases the result in that direction. I’m not sure this is a problem, so I haven’t done this yet. Over the long-term, I believe this problem is self-correcting as we collect more and more data.

As always, please post any problems you come across.

I think this is a fantastic change. Consensus is way more meaningful for we who are looking more for guidance than a single opinion or the opinion of someone from 1978.

I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.

I agree that the ratings will likely creep up a bit on older routes, but I think this will improve consistency between routes. Thanks Nick!

nkane wrote:I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.

whoah! Jens must be going f*ucking bonkers. please tell me you used slash grades too!?!?!?!?!?!!

Love it, thanks Nick!

Im into it.

Definitely in favor of this approach, though I wonder if it might be best to leave original until there is a minimum number of votes, like 3 or 5? Then your median will reject the outliers. But if there is just one other vote, that is way off, what happens?

Great! I'm going to be a harder climber and I don't even have to work out.
:-)

There's a major flaw in this change Nick.

If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a.

check out the white wall at echo cliffs, ca
mountainproject.com/v/white…
the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d

I like this change. One suggestion I have is to label the grade "Consensus YDS" instead of just "YDS". Right now the "Original" grade is labeled, but a user that did not see this post might not realize what the posted grade means. This could be especially confusing if they are accustomed to the way the site used to be.

Yeitti wrote:There's a major flaw in this change Nick. If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a. check out the white wall at echo cliffs, ca mountainproject.com/v/white… the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d

Nick Wilder wrote: That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow. Thanks for pointing it out.

s.price wrote:One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example. I posted this climb at 5.12 and it reflects this in the Original YDS. In the suggested ratings my rating now shows 5.12a which is the consensus. I never rated the climb .12a, only .12. Looking through other routes I posted it has done the same on many, but not all of them.

Awesome! Thanks Nick. I think you are spot on in your reasoning and I look forward to the results.

So Nick, what is the formula used to reach a consensus on a grade? Shouldn't a 5.11c and a 5.11b suggestion = 5.11b/c or a 3 people suggesting 5.12a and 2 suggesting 5.11d = 5.12a.... majority vote?

I added code to bias the original grade somewhat. It won't make a difference when there are more than ~5 votes, but with routes with only a couple votes, the original grade counts more.

It would be real interesting to introduce a "Sandbagger" tag for mp members who consistently downgrade routes......would be fun to see. Would either be a badge of honour or a tag of ludicracy based on your viewpoint.

Joe Forrester wrote:It would be real interesting to introduce a "Sandbagger" tag for mp members who consistently downgrade routes......would be fun to see.