Route Guide    Partners    Forum    Photos    What's New    Journal        
Sign Up  |   Log In:Login with Facebook
REI Community
Route Ratings Changed to Consensus
View Latest Posts in This Forum or All Forums
Page 1 of 3.  1  2  3   Next>   Last>>
Follow replies to this topic? Notify me at the top of web site.
1

Email me.
 
Site Landlord
Apr 8, 2016
Rock Climbing Photo: Personal Photo
Today I changed the site so that the consensus grade is the official grade that shows up in search results, routes lists, etc. The “original” grade (what the submitter rated it) is still available and listed on the route page.

MP's primary purpose is a route guide: help people find and enjoy climbs suited to their ability. Though I expect there to be a few problems with consensus grades, I believe that, on average, the “wisdom of the crowds” will be better than a single opinion.

This idea has been discussed many times in the forum, and two good concerns repeatedly come up:

1. Outlying values: the consensus is calculated as the MEDIAN of the all ratings submitted, including the “original” rating, which is fairly insensitive to outliers. This is different from the AVERAGE of all ratings.

2. People only submit ratings if different from the original. If we have a problem with the new system, I can re-calculate the consensus and overweight the "original" rating so that it biases the result in that direction. I’m not sure this is a problem, so I haven’t done this yet. Over the long-term, I believe this problem is self-correcting as we collect more and more data.

As always, please post any problems you come across.
Nick Wilder
From The Bubble
Joined Jan 1, 2005
2,134 points
Apr 8, 2016
Rock Climbing Photo: Hawksbill
I think this is a fantastic change. Consensus is way more meaningful for we who are looking more for guidance than a single opinion or the opinion of someone from 1978. Brian Abram
From Celo, NC
Joined Oct 17, 2007
418 points
Apr 8, 2016
I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward. nkane
Joined Jun 6, 2013
23 points
Apr 8, 2016
Rock Climbing Photo: J-Tree
I agree that the ratings will likely creep up a bit on older routes, but I think this will improve consistency between routes. Thanks Nick! David Hous
From Boulder, Colorado
Joined Nov 18, 2001
191 points
Apr 8, 2016
nkane wrote:
I'll be interested to see which way grades move with this change, both now and over time. I suspect the trajectories will be upward.

any way for MP to track this?
simplyput
Joined Nov 30, 2013
55 points
Apr 8, 2016
whoah! Jens must be going f*ucking bonkers. please tell me you used slash grades too!?!?!?!?!?!! Redyns
Joined Apr 11, 2011
16 points
Apr 8, 2016
Rock Climbing Photo: Climbing at the Gallery in Red Rocks
Love it, thanks Nick! kennoyce
From Layton, UT
Joined Aug 12, 2010
2,061 points
Apr 8, 2016
Rock Climbing Photo: Me redpointing Power Series. This route is amazing...
Im into it. C. Archibald
Joined Apr 15, 2012
781 points
Apr 8, 2016
Rock Climbing Photo: Goliath Summit Shadows
Definitely in favor of this approach, though I wonder if it might be best to leave original until there is a minimum number of votes, like 3 or 5? Then your median will reject the outliers. But if there is just one other vote, that is way off, what happens? Paul Zander
From Tucson, AZ
Joined Oct 15, 2012
670 points
Apr 8, 2016
Rock Climbing Photo: Eiger summit
Great! I'm going to be a harder climber and I don't even have to work out.
:-)
Brian
From North Kingstown, RI
Joined Sep 27, 2001
803 points
Apr 8, 2016
Rock Climbing Photo: me
There's a major flaw in this change Nick.

If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a.

check out the white wall at echo cliffs, ca
mountainproject.com/v/white-wa...
the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d
Yeitti
From Colorado or sometimes LA
Joined Dec 8, 2015
49 points
Apr 8, 2016
Rock Climbing Photo: Half Dome Summit
I like this change. One suggestion I have is to label the grade "Consensus YDS" instead of just "YDS". Right now the "Original" grade is labeled, but a user that did not see this post might not realize what the posted grade means. This could be especially confusing if they are accustomed to the way the site used to be. Bob Johnson
From Philadelphia, PA
Joined May 25, 2014
61 points
Site Landlord
Apr 8, 2016
Rock Climbing Photo: Personal Photo
Yeitti wrote:
There's a major flaw in this change Nick. If only one person rates/grades the route/problem, then if a 2nd person downgrades the route, the consensus automatically changes to the downgrade and changes the originals poster's grade as well. For example, if a route posted as 5.13b in then downgraded by someone to 12a, then the consensus and original posters grade change to 12a. check out the white wall at echo cliffs, ca mountainproject.com/v/white-wa... the grades are out of whack there. The original poster put State of Grace at 5.0 when it should be 5.13b/c and Power of One at 3rd class when it should be 5.13c or d


That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow.

Thanks for pointing it out.
Nick Wilder
From The Bubble
Joined Jan 1, 2005
2,134 points
Apr 8, 2016
Rock Climbing Photo: me
Nick Wilder wrote:
That particular user has purposely entered incorrect values for many routes. I have talked to him, but not changed his mind. I will change the way consensus ratings are calculated to fix this. They will be updated by tomorrow. Thanks for pointing it out.


thank you Nick!
Yeitti
From Colorado or sometimes LA
Joined Dec 8, 2015
49 points
Apr 8, 2016
Rock Climbing Photo: Ringo at Riverside Boulder.
One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example.
I posted this climb at 5.12 and it reflects this in the Original YDS.
In the suggested ratings my rating now shows 5.12a which is the consensus.
I never rated the climb .12a, only .12.
Looking through other routes I posted it has done the same on many, but not all of them.
s.price
From PS,CO
Joined Dec 1, 2010
1,376 points
Site Landlord
Apr 8, 2016
Rock Climbing Photo: Personal Photo
s.price wrote:
One issue I have noticed. I will use Wild Hops in Piedra Canyon as an example. I posted this climb at 5.12 and it reflects this in the Original YDS. In the suggested ratings my rating now shows 5.12a which is the consensus. I never rated the climb .12a, only .12. Looking through other routes I posted it has done the same on many, but not all of them.


This is fixed. Thanks for pointing this out!
Nick Wilder
From The Bubble
Joined Jan 1, 2005
2,134 points
Apr 8, 2016
Rock Climbing Photo: this is begging for a caption
Rock Climbing Photo: I support this post.
I support this post.
Jon Ruland
From Tucson, AZ
Joined May 31, 2007
986 points
Apr 8, 2016
Rock Climbing Photo: A Photo
Awesome! Thanks Nick. I think you are spot on in your reasoning and I look forward to the results. psakievich
From Tempe, AZ
Joined Aug 7, 2014
117 points
Apr 9, 2016
Rock Climbing Photo: me
So Nick, what is the formula used to reach a consensus on a grade? Shouldn't a 5.11c and a 5.11b suggestion = 5.11b/c or a 3 people suggesting 5.12a and 2 suggesting 5.11d = 5.12a.... majority vote? Yeitti
From Colorado or sometimes LA
Joined Dec 8, 2015
49 points
Site Landlord
Apr 9, 2016
Rock Climbing Photo: Personal Photo
I added code to bias the original grade somewhat. It won't make a difference when there are more than ~5 votes, but with routes with only a couple votes, the original grade counts more. Nick Wilder
From The Bubble
Joined Jan 1, 2005
2,134 points
Apr 9, 2016
My suggestion would be to make the rating process anonymous. Right now on many routes/problems you see people turn rating into a subtle spray and also nobody wants to be the guy to upgrade something that's 'within his grade'. Making ratings anonymous would allow people to honestly contribute to the grade estimate. Nivel Egres
From New York, NY
Joined Dec 10, 2014
21 points
Apr 9, 2016
Rock Climbing Photo: My first belay certification.
Yeitti wrote:
So Nick, what is the formula used to reach a consensus on a grade? Shouldn't a 5.11c and a 5.11b suggestion = 5.11b/c or a 3 people suggesting 5.12a and 2 suggesting 5.11d = 5.12a.... majority vote?


Nick Wilder wrote:
the consensus is calculated as the MEDIAN of the all ratings submitted, including the “original” rating, which is fairly insensitive to outliers. This is different from the AVERAGE of all ratings.



Median = the middle value in a list of ordered values. Thus:

5.11d
5.11d
5.12a <==
5.12a
5.12a

But:

5.11c
5.11c
5.11d <==
5.12a
5.12a

Even though there is only one vote for 11d and two for 12a and two for 11c, the median is the 11.d, reflecting a closer consensus than the majority (strictly speaking, a plurality) that voted for the higher grade
Dylan B.
Joined Mar 31, 2006
971 points
Apr 9, 2016
Rock Climbing Photo: On Durin's Bane
It would be real interesting to introduce a "Sandbagger" tag for mp members who consistently downgrade routes......would be fun to see. Would either be a badge of honour or a tag of ludicracy based on your viewpoint. Joe Forrester
From Palo Alto
Joined Aug 7, 2005
1,086 points
Apr 9, 2016
Rock Climbing Photo: At the BRC
Joe Forrester wrote:
It would be real interesting to introduce a "Sandbagger" tag for mp members who consistently downgrade routes......would be fun to see.


We could call it the Tony B tag. :-)
He's consistent, so I always just add two letters to what ever he grades a route. Then I know what to expect.
Mark E Dixon
From Sprezzatura, Someday
Joined Nov 29, 2007
517 points
Apr 9, 2016
I have no complaints with this change.
I always got a kick out of Moby Dick Ahab - a nasty 10b offwidth/squeeze being downrated to 10a by the original route submitter when Supertopo says 99 out of 100 5.13 sport climbers would be completely shut down by this climb.
Lou Hibbard
From Eagan, MN
Joined Mar 12, 2006
246 points
Apr 9, 2016
Mark E Dixon wrote:
We could call it the Tony B tag. :-)

Or the Tank Evans tag for harder routes. Though he doesn't perform the same downgrade on boulder problems...

Back on topic: the consensus does still have the issue of people grading the 1st pitch of a multi-pitch route or different variations of a single route/boulder problem. Here, the intention of the OP is better defined than the mob consensus.
reboot
From Westminster, CO
Joined Jul 17, 2006
163 points


Follow replies to this topic? Notify me at the top of web site.
1

Email me.
Page 1 of 3.  1  2  3   Next>   Last>>