Route Ratings Changed to Consensus


Original Post
Nick Wilder
The Bubble
Joined: Jan 2005
Points: 16,378

Today I changed the site so that the consensus grade is the official grade that shows up in search results, routes lists, etc. The “original” grade (what the submitter rated it) is still available and listed on the route page.

MP's primary purpose is a route guide: help people find and enjoy climbs suited to their ability. Though I expect there to be a few problems with consensus grades, I believe that, on average, the “wisdom of the crowds” will be better than a single opinion.

This idea has been discussed many times in the forum, and two good concerns repeatedly come up:

1. Outlying values: the consensus is calculated as the MEDIAN of the all ratings submitted, including the “original” rating, which is fairly insensitive to outliers. This is different from the AVERAGE of all ratings.

2. People only submit ratings if different from the original. If we have a problem with the new system, I can re-calculate the consensus and overweight the "original" rating so that it biases the result in that direction. I’m not sure this is a problem, so I haven’t done this yet. Over the long-term, I believe this problem is self-correcting as we collect more and more data.

As always, please post any problems you come across. — Apr 8, 2016
Brian Abram
Celo, NC
Joined: Oct 2007
Points: 0
I think this is a fantastic change. Consensus is way more meaningful for we who are looking more for guidance than a single opinion or the opinion of someone from 1978. — Apr 8, 2016
nkane
Unknown Hometown
Joined: Jun 2013
Points: 0

I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward. — Apr 8, 2016
David Hous
Boulder, Colorado
Joined: Nov 2001
Points: 0

I agree that the ratings will likely creep up a bit on older routes, but I think this will improve consistency between routes. Thanks Nick! — Apr 8, 2016
simplyput
Unknown Hometown
Joined: Nov 2013
Points: 0
nkane wrote:I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.
any way for MP to track this? — Apr 8, 2016
Redyns
Unknown Hometown
Joined: Apr 2011
Points: 0
whoah! Jens must be going f*ucking bonkers. please tell me you used slash grades too!?!?!?!?!?!! — Apr 8, 2016
Ken Noyce
Layton, UT
Joined: Aug 2010
Points: 0
Love it, thanks Nick! — Apr 8, 2016
Creed A
Unknown Hometown
Joined: Apr 2012
Points: 0

Im into it. — Apr 8, 2016
Paul Zander
Tucson, AZ
Joined: Oct 2012
Points: 43
Definitely in favor of this approach, though I wonder if it might be best to leave original until there is a minimum number of votes, like 3 or 5? Then your median will reject the outliers. But if there is just one other vote, that is way off, what happens? — Apr 8, 2016
Brian
North Kingstown, RI
Joined: Sep 2001
Points: 0
Great! I'm going to be a harder climber and I don't even have to work out.
:-) — Apr 8, 2016
Yeitti
Colorado or sometimes LA
Joined: Dec 2015
Points: 0
There's a major flaw in this change Nick.

If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a.

check out the white wall at echo cliffs, ca
http://www.mountainproject.com/v/white-wall/106045970
the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d — Apr 8, 2016
Bob Johnson
Philadelphia, PA
Joined: May 2014
Points: 0

I like this change. One suggestion I have is to label the grade "Consensus YDS" instead of just "YDS". Right now the "Original" grade is labeled, but a user that did not see this post might not realize what the posted grade means. This could be especially confusing if they are accustomed to the way the site used to be. — Apr 8, 2016
Nick Wilder
The Bubble
Joined: Jan 2005
Points: 16,378

Yeitti wrote:There's a major flaw in this change Nick. If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a. check out the white wall at echo cliffs, ca mountainproject.com/v/white... the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d
That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow.

Thanks for pointing it out. — Apr 8, 2016
Yeitti
Colorado or sometimes LA
Joined: Dec 2015
Points: 0
Nick Wilder wrote: That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow. Thanks for pointing it out.
thank you Nick! — Apr 8, 2016
s.price
PS,CO
Joined: Dec 2010
Points: 0
One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example.
I posted this climb at 5.12 and it reflects this in the Original YDS.
In the suggested ratings my rating now shows 5.12a which is the consensus.
I never rated the climb .12a, only .12.
Looking through other routes I posted it has done the same on many, but not all of them. — Apr 8, 2016
Nick Wilder
The Bubble
Joined: Jan 2005
Points: 16,378

s.price wrote:One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example. I posted this climb at 5.12 and it reflects this in the Original YDS. In the suggested ratings my rating now shows 5.12a which is the consensus. I never rated the climb .12a, only .12. Looking through other routes I posted it has done the same on many, but not all of them.
This is fixed. Thanks for pointing this out! — Apr 8, 2016
Jon Ruland
Tucson, AZ
Joined: May 2007
Points: 0
Phil Sakievich
Tempe, AZ
Joined: Aug 2014
Points: 0
Awesome! Thanks Nick. I think you are spot on in your reasoning and I look forward to the results. — Apr 8, 2016
Yeitti
Colorado or sometimes LA
Joined: Dec 2015
Points: 0
So Nick, what is the formula used to reach a consensus on a grade? Shouldn't a 5.11c and a 5.11b suggestion = 5.11b/c or a 3 people suggesting 5.12a and 2 suggesting 5.11d = 5.12a.... majority vote? — Apr 9, 2016
Nick Wilder
The Bubble
Joined: Jan 2005
Points: 16,378

I added code to bias the original grade somewhat. It won't make a difference when there are more than ~5 votes, but with routes with only a couple votes, the original grade counts more. — Apr 9, 2016
Nivel Egres
New York, NY
Joined: Dec 2014
Points: 0

My suggestion would be to make the rating process anonymous. Right now on many routes/problems you see people turn rating into a subtle spray and also nobody wants to be the guy to upgrade something that's 'within his grade'. Making ratings anonymous would allow people to honestly contribute to the grade estimate. — Apr 9, 2016
Guideline #1: Don't be a jerk.

Post a Reply

Log In to Reply