Route Ratings Changed to Consensus

grog m wrote:The gunks...the route desciptions say things such as "anywhere else this would be 5.10". Okay, why not just grade it 5.10? Ego? Douchism? Whats the point of grades if not to provide the climber a universal idea of how difficult the terrain he will encounter may be. Hopefully this new consensus grading system will "fix" the gunks.

I don't think the original ratings had anything to do with ego and douchism. Those were just the prevailing grades at the time. When the early Gunks climbers graded a 5.6 that was considered a hard climb in those days. If you look at the early grades as being the standard then everything else has been grade creep since then. Although as I mentioned earlier I like grade creep. It makes me a harder climber without working out.

rgold wrote:I think Bill has a point, but it works both ways. Climbers who have specialized in one genre of climbing naturally tend to overrate less familiar types of climbing. And when it comes to trad climbing, there is the prospect of undergrading by people who have not "really" led the route. If we are going for precision, then reporting the standard deviation would be nice too. In principle, half the responders think the climb is as difficult or harder than the median grade, and half the responders think it is as difficult or easier than the median grade. I think there is quite a difference between nearly full agreement and a spread of ratings as much as, say, two full grades between the hardest estimate and the easiest. The standard deviation would at least alert the reader about the precision of the consensus grade.

Rich - take a look at redriverclimbing.com That's got a pretty good example of what you're referring to in terms of the standard deviation in a good visual depiction

I imagine this affect will change the grades of all the classic climbing areas. Yosemite, Jtree, Index, the Gunk's etc will all see grade inflation. I have a feeling that this will cause a lot of internal bickering. But seeing how everyone on this forum is probably just wasting time at work anyways, I don't really see a problem with it.

I highly suggest trying to climb pitches without knowledge of the grades/beta. Usimg your eyes to judge a route, or the suggestion from a friend that such and such is a good route has always pushed me to climb better. This technique makes every climb much more enjoyable in my opinion.

Loganator wrote:I highly suggest trying to climb pitches without knowledge of the grades/beta. Using your eyes to judge a route, or the suggestion from a friend that such and such is a good route has always pushed me to climb better. This technique makes every climb much more enjoyable in my opinion.

+1 on that, Loganator. Your statement captures more of the spirit of climbing than grade chasing.

It looks like the consensus grade shows up in search results, but that the search is still retrieving matches based on the original grade. For example, if I search for .10a routes in the Estes Valley, I end up with results for 5.8-5.10b routes (consensus grades): mountainproject.com/scripts…

Is this the intended functionality? It's a bit confusing to see 5.8 or 5.10b routes if I'm looking for 5.10a specifically (just as an example).

Ryan Nevius wrote:It looks like the consensus grade shows up in search results, but that the search is still retrieving matches based on the original grade. For example, if I search for .10a routes in the Estes Valley, I end up with results for 5.8-5.10b routes (consensus grades): mountainproject.com/scripts… Is this the intended functionality? It's a bit confusing to see 5.8 or 5.10b routes if I'm looking for 5.10a specifically (just as an example).

Oops - fixed! Thanks for the report.

Here's a perfect example of why listing the "median" grade suggestion fails when there are few route grades: mountainproject.com/v/omaha…. The route is a consensus 14a but one guy with zero site contributions dropped a 5.12 rating last year and instantly the route gets downgraded to 5.12 and shows up in searches as 5.12 because the only other rating on MP is from the original posting. I'd rather find a route by its original grade and find out later that it's soft/hard by looking at the consensus and seeing who ticked it and their reasoning.

I totally agree with MattH.

The route grade should be listed as what the locals or guidebook authors believe it to be (or its original grade by the poster). Then the consensus on MP listed so if you want info on what other people on MP think you can find it.

His example of Omaha Beach is exactly why...lots of routes like this on MP.

Plus the consensus grades on MP are always jacked up by a sandbagger or someone who thinks routes are sandbagged...everyone I know takes them with a grain of salt.

I doubt many old and sandbagged climbs will stray too far from their original grade due to anchoring bias.
In the thread "Lies, Damned Lies, and Statistics" teece303 makes a pretty good point worth repeating.

teece303 wrote:Anchoring bias play a big part in reducing deviation. Look it up if you've never thought about it. Once a route is assigned a grade, people will, in general, have a very hard time suggesting a grade very far from the assigned grade, for fundamental psychological reasons. And that's true even if they think the assigned grade is just plain wrong. I've seen this in action. Climber think she is warming up on a 5.9+. The route is VERY hard. Half way up her and the belayer think this is more like 10+ or 11-. Pretty stiff for a 9. When they figure out what route they were on, it's a 12a. This climber climbs 12a, so she knows what that feels like. But because she was anchored to the belief that the route was a 9+, she was unwilling to de-anchor and stray that far from the assigned rating. It's a fascinating psychological problem. So not only are grades somewhat subjective (although, they are nowhere near as subjective as many climbers imply), and a grade will vary by climber (some routes exploit our weakness or play to our strengths, and thus are legitimately different to different climbers), but ALSO, the very act of assigning a number to them makes it difficult to get an "accurate" grade of the route, if that initial number is somehow "wrong."

The "Wisdom of the Crowd" may work surprisingly well with groups guessing a value that's objective when individuals aren't influenced by the opinions of others. It's likely less effective when dealing with something like climbing grades which are somewhat subjective and individual opinions are heavily influenced by the existing grade.

I'd love to see a normalization of grades across areas so I know what to expect when I get on a climb anywhere in the world. But I'll bet localization in grading will remain a fixture of the climbing world for many years to come. Crack climbs in sport climbing areas like Shelf Road will still be rated much harder than the same climb if it existed in a crack climbing area like Indian Creek. Many people recognize this fact, but the problem still remains. See the comments for I, Claudius at Shelf Road for an example.

MattH wrote:Here's a perfect example of why listing the "median" grade suggestion fails when there are few route grades: mountainproject.com/v/omaha…. The route is a consensus 14a but one guy with zero site contributions dropped a 5.12 rating last year and instantly the route gets downgraded to 5.12 and shows up in searches as 5.12 because the only other rating on MP is from the original posting. I'd rather find a route by its original grade and find out later that it's soft/hard by looking at the consensus and seeing who ticked it and their reasoning.

This is either wrong, or someone JUST added another grade to it. Both the consensus and original grade are 14a, and the 12 is still in there. And it shouldn't work that way - I added code a couple days ago that biases a route towards it's original grade in the cases where there are very few votes.

Nick Wilder wrote: This is either wrong, or someone JUST added another grade to it. Both the consensus and original grade are 14a, and the 12 is still in there. And it shouldn't work that way - I added code a couple days ago that biases a route towards it's original grade in the cases where there are very few votes.

Thanks for fixing this Nick, my only 5.14 and Mountain Project had it downgraded to a 5.12 :)

Nick Wilder wrote: This is either wrong, or someone JUST added another grade to it. Both the consensus and original grade are 14a, and the 12 is still in there. And it shouldn't work that way - I added code a couple days ago that biases a route towards it's original grade in the cases where there are very few votes.

It's definitely the latter case - when I posted, there were 2 grades, the original 14a post and a second 5.12 post. The consensus was listed on both the page and the sidebar as 5.12. Now there's a 3rd grade, of 14a, which pushes the median back to 5.14a. Every route with 2 grades that I've seen has this problem - it takes the 2nd grade suggestion as the median instead of the first. An example from my ticklist that I noticed: mountainproject.com/v/try-o… was v8+ when submitted but is now v7 after just one v7 suggestion.

As I mentioned earlier, it might be useful to have a "silent grade suggestion" of sorts for people who tick a route and don't give a grade suggestion because they tend to do this only when the listed grade is seen as accurate and this listed grade can now change. If you gave them a half-weighted silent grade suggestion of whatever the listed grade was at the time they ticked it, you'd likely have more accurate consensus grades. For all ticks before the switch, it'd be easy to figure out the listed grade at the time of the tick, as it'd just be the submitted grade. Now it might take a bit more work obviously.

MattH wrote: It's definitely the latter case - when I posted, there were 2 grades, the original 14a post and a second 5.12 post. The consensus was listed on both the page and the sidebar as 5.12. Now there's a 3rd grade, of 14a, which pushes the median back to 5.14a. Every route with 2 grades that I've seen has this problem - it takes the 2nd grade suggestion as the median instead of the first. An example from my ticklist that I noticed: mountainproject.com/v/try-o… was v8+ when submitted but is now v7 after just one v7 suggestion. As I mentioned earlier, it might be useful to have a "silent grade suggestion" of sorts for people who tick a route and don't give a grade suggestion because they tend to do this only when the listed grade is seen as accurate and this listed grade can now change. If you gave them a half-weighted silent grade suggestion of whatever the listed grade was at the time they ticked it, you'd likely have more accurate consensus grades. For all ticks before the switch, it'd be easy to figure out the listed grade at the time of the tick, as it'd just be the submitted grade. Now it might take a bit more work obviously.

My prior claim was wrong - apologies! I wrote the code to bias the original grade, but forgot to run every existing route through it and get them updated. I've done so now. Hope I got it right this time. Please let me know if you see any more problems like this one.

Thanks!

mountainproject.com/v/regul…

A slight problem is people logging aid routes as a free routes. For example the RNWF should show up as either 5.12 (actual free grade) or as 5.9 C1 (hardest mandatory free climbing), but people logging it as just "5.9" means the consensus grade is now 5.9, and it's showing up as the best 5.9 free climb in the Valley. [obviously this ignores the recent changes to the routes, a different discussion]

Even w cutting out the outlier 13b and weighting the original rating more when there's less than 5 votes, how do the ratings below work out to a "median" of 5.8?

It seems like in these cases, the effect of the algorithm is not "overweight the original", but more like "take the original, period"? Perhaps a a case where a mean would be better?

Don't remember the bolt spacing, but from other comments, I'm not sure 5.8 leaders being super confident about getting on this route is necessarily a good thing.

Jump for Cholla

Christian wrote:Even w cutting out the outlier 13b and weighting the original rating more when there's less than 5 votes, how do the ratings below work out to a "median" of 5.8? It seems like in these cases, the effect of the algorithm is not "overweight the original", but more like "take the original, period"? Perhaps a a case where a mean would be better? Don't remember the bolt spacing, but from other comments, I'm not sure 5.8 leaders being super confident about getting on this route is necessarily a good thing.

Just to make sure everyone understands the algorithm: 5.8 was the original submission, and we count it extra times. So the median of:

5.8
5.8
5.8
5.9
5.10a
5.13b

is either 5.8 or 5.9 depending on how you round.

I have deleted the 5.13b, which appears to be an attempt to move the consensus up. Bottom line: I am sure you are right that there are MANY cases where, at a given point in time, the MEAN would be better than the MEDIAN, but out of 130,000+ routes, we can't manually pick and choose which routes get which system. This will fix itself as more people weigh in and move the consensus.

Thanks for the explanation, Nick.

Just out of habit, I always think of terms of weighted averages, so I was having trouble picturing how the extra weighting works with a median. Even the fact that the original rating counts as if 3 people gave it that rating was news to me. Could have been 2 or 4 or 5 for all I knew.

Christian wrote:It seems like in these cases, the effect of the algorithm is not "overweight the original", but more like "take the original, period"?

Yes and no. The effect of the algorithm is that the original grade sticks until at least 3 other people have weighed in. If the weighting is in effect only when there are less than five people suggesting grades, then the algorithm is really "original grade or unweighted median" except when there are four suggestions ... and then it depends on whether the median rounds up or down.

The additional rating suggestions eventually have an effect, but only once enough people have weighed in.

Nick, can you query the data to find out what the overall trend has been? I'm interested to see if the trend was up or down, and how substantially it went up/down over the last month.

I thought the 3x weighting for the original dropped out entirely after 5 people weighed in, but it appears it's in effect regardless of how many people weigh in, and if there's an even number of ratings, it always rounds towards the original (could also be "always rounds down", I suppose?)

For example this route had 10 ratings, not including my 11- rating added last. It was showing 10+ as consensus before my rating, which implies Brigitte's rating was still counting for 3 and that w those 12 ratings and the median between 10+ and 11-, it was rounding to 10+.

After my 11-, there were 13 ratings (w Brigette's counting for 3) and a clear median of 11-, and indeed the consensus changed to 11-.

"How do you like them apples"

How do you like them apples