Route Ratings Changed to Consensus


Original Post
Nick Wilder · · The Bubble · Joined Jan 2005 · Points: 1,536

Today I changed the site so that the consensus grade is the official grade that shows up in search results, routes lists, etc. The “original” grade (what the submitter rated it) is still available and listed on the route page.

MP's primary purpose is a route guide: help people find and enjoy climbs suited to their ability. Though I expect there to be a few problems with consensus grades, I believe that, on average, the “wisdom of the crowds” will be better than a single opinion.

This idea has been discussed many times in the forum, and two good concerns repeatedly come up:

1. Outlying values: the consensus is calculated as the MEDIAN of the all ratings submitted, including the “original” rating, which is fairly insensitive to outliers. This is different from the AVERAGE of all ratings.

2. People only submit ratings if different from the original. If we have a problem with the new system, I can re-calculate the consensus and overweight the "original" rating so that it biases the result in that direction. I’m not sure this is a problem, so I haven’t done this yet. Over the long-term, I believe this problem is self-correcting as we collect more and more data.

As always, please post any problems you come across.


Brian Abram · · Celo, NC · Joined Oct 2007 · Points: 228

I think this is a fantastic change. Consensus is way more meaningful for we who are looking more for guidance than a single opinion or the opinion of someone from 1978.


nkane · · Unknown Hometown · Joined Jun 2013 · Points: 75

I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.


David Hous · · Boulder, Colorado · Joined Nov 2001 · Points: 145

I agree that the ratings will likely creep up a bit on older routes, but I think this will improve consistency between routes. Thanks Nick!


simplyput · · Unknown Hometown · Joined Nov 2013 · Points: 0
nkane wrote:I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.
any way for MP to track this?

Redyns · · Unknown Hometown · Joined Apr 2011 · Points: 0

whoah! Jens must be going f*ucking bonkers. please tell me you used slash grades too!?!?!?!?!?!!


Ken Noyce · · Layton, UT · Joined Aug 2010 · Points: 2,017

Love it, thanks Nick!


Creed A · · Salt Lake City, UT · Joined Apr 2012 · Points: 680

Im into it.


Paul Zander · · Tucson, AZ · Joined Oct 2012 · Points: 667

Definitely in favor of this approach, though I wonder if it might be best to leave original until there is a minimum number of votes, like 3 or 5? Then your median will reject the outliers. But if there is just one other vote, that is way off, what happens?


Brian · · North Kingstown, RI · Joined Sep 2001 · Points: 465

Great! I'm going to be a harder climber and I don't even have to work out.
:-)


Yeitti · · Colorado or sometimes LA · Joined Dec 2015 · Points: 20

There's a major flaw in this change Nick.

If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a.

check out the white wall at echo cliffs, ca
http://www.mountainproject.com/v/white-wall/106045970
the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d


Bob Johnson · · Philadelphia, PA · Joined May 2014 · Points: 53

I like this change. One suggestion I have is to label the grade "Consensus YDS" instead of just "YDS". Right now the "Original" grade is labeled, but a user that did not see this post might not realize what the posted grade means. This could be especially confusing if they are accustomed to the way the site used to be.


Nick Wilder · · The Bubble · Joined Jan 2005 · Points: 1,536
Yeitti wrote:There's a major flaw in this change Nick. If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a. check out the white wall at echo cliffs, ca mountainproject.com/v/white... the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d
That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow.

Thanks for pointing it out.

Yeitti · · Colorado or sometimes LA · Joined Dec 2015 · Points: 20
Nick Wilder wrote: That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow. Thanks for pointing it out.
thank you Nick!

s.price · · PS,CO · Joined Dec 2010 · Points: 1,018

One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example.
I posted this climb at 5.12 and it reflects this in the Original YDS.
In the suggested ratings my rating now shows 5.12a which is the consensus.
I never rated the climb .12a, only .12.
Looking through other routes I posted it has done the same on many, but not all of them.


Nick Wilder · · The Bubble · Joined Jan 2005 · Points: 1,536
s.price wrote:One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example. I posted this climb at 5.12 and it reflects this in the Original YDS. In the suggested ratings my rating now shows 5.12a which is the consensus. I never rated the climb .12a, only .12. Looking through other routes I posted it has done the same on many, but not all of them.
This is fixed. Thanks for pointing this out!

Jon Ruland · · Tucson, AZ · Joined May 2007 · Points: 392

Phil Sakievich · · Tempe, AZ · Joined Aug 2014 · Points: 30

Awesome! Thanks Nick. I think you are spot on in your reasoning and I look forward to the results.


Yeitti · · Colorado or sometimes LA · Joined Dec 2015 · Points: 20

So Nick, what is the formula used to reach a consensus on a grade? Shouldn't a 5.11c and a 5.11b suggestion = 5.11b/c or a 3 people suggesting 5.12a and 2 suggesting 5.11d = 5.12a.... majority vote?


Nick Wilder · · The Bubble · Joined Jan 2005 · Points: 1,536

I added code to bias the original grade somewhat. It won't make a difference when there are more than ~5 votes, but with routes with only a couple votes, the original grade counts more.


Nivel Egres · · New York, NY · Joined Dec 2014 · Points: 0

My suggestion would be to make the rating process anonymous. Right now on many routes/problems you see people turn rating into a subtle spray and also nobody wants to be the guy to upgrade something that's 'within his grade'. Making ratings anonymous would allow people to honestly contribute to the grade estimate.


Guideline #1: Don't be a jerk.

Post a Reply

Log In to Reply