Climbing Heatmap

Hey guys,

I just finished working on this heatmap feature for my climbing recommendation system. I took down some of the other stuff so it's easier to use. I was wondering if anyone wanted to take a look and maybe tell me what you think? I'm trying to do what I can to make it better. It's working well for my fiancee and I so far.

https://climbingsuggestions.com/

I took a look at if for my local crag (index) and the first thing I noticed was (our grade) was completely fucked up. Don't know how you get that, but calling iron horse, model worker, bwanna dik, keiths crack, natural log cabin,  amongst other already stiff 11's 5.10 is comical. There were also some 5.9c's in there? lol

This is really cool.  What is the difference between "Grade" and "Our Grade" columns?

Good question! 

So the first thing I did was I converted the yosemite decimal scale to just a decimal scale... so 5.0=0, 5.0a=1, 5.0b=2...etc.

Next I used a maximum likelihood algorithm to figure out every climber's probability of climbing a climb at a particular grade. I fit it to a beta distribution. That gave me a probability density.

After that I sought to determine the grade of a route, given some set of users has climbed it. 

By evolving the posterior distribution given each climber has climbed the climb; I got 92% accuracy predicting the grade of 45,000 climbs in the US on mountain project. 

Those that are far off from the "true grade" on mountain project tend to have comments like "this one is soft" or "this one is sandbagged".

Yeah, it's weird to think of a 5.8a vs a 5.8d... but that's how the computer has to think. The attempt is to bring all the crags in the country to a consistent grading... so nomads (like me) don't have to be surprised jumping on a "5.5" in the Gunks vs a "5.10" at tensleep canyon. I've been to a few dozen crags, and it seems right on (in terms of consistency nationally, NOT locally).

There's nuance to it, though. It's not "this is a 5.9c in terms of technical difficulty". It's "this is a climb for a 5.9c CLIMBER". The difference being it takes into account things like runout, adventure, trickiness with the gear, etc. 

The "our grade" system is probably not for everyone. It works well for me.

Andrew Kelleherwrote:

Good question! 

So the first thing I did was I converted the yosemite decimal scale to just a decimal scale... so 5.0=0, 5.0a=1, 5.0b=2...etc.

Next I used a maximum likelihood algorithm to figure out every climber's probability of climbing a climb at a particular grade. I fit it to a beta distribution. That gave me a probability density.

After that I sought to determine the grade of a route, given some set of users has climbed it. 

By evolving the posterior distribution given each climber has climbed the climb; I got 92% accuracy predicting the grade of 45,000 climbs in the US on mountain project. 

Those that are far off from the "true grade" on mountain project tend to have comments like "this one is soft" or "this one is sandbagged".

Yeah, it's weird to think of a 5.8a vs a 5.8d... but that's how the computer has to think. The attempt is to bring all the crags in the country to a consistent grading... so nomads (like me) don't have to be surprised jumping on a "5.5" in the Gunks vs a "5.10" at tensleep canyon. I've been to a few dozen crags, and it seems right on (in terms of consistency nationally, NOT locally).

There's nuance to it, though. It's not "this is a 5.9c in terms of technical difficulty". It's "this is a climb for a 5.9c CLIMBER". The difference being it takes into account things like runout, adventure, trickiness with the gear, etc. 

The "our grade" system is probably not for everyone. It works well for me.

You got way way way more brains than needed for climbing. ;-)

EDIT to add: wow! Local crap is interesting. I've always thought it hard, compared to everywhere else, and, that's how your second grade came up. 

Andrew Kelleherwrote:

Good question! 

So the first thing I did was I converted the yosemite decimal scale to just a decimal scale... so 5.0=0, 5.0a=1, 5.0b=2...etc.

Next I used a maximum likelihood algorithm to figure out every climber's probability of climbing a climb at a particular grade. I fit it to a beta distribution. That gave me a probability density.

After that I sought to determine the grade of a route, given some set of users has climbed it. 

By evolving the posterior distribution given each climber has climbed the climb; I got 92% accuracy predicting the grade of 45,000 climbs in the US on mountain project. 

Those that are far off from the "true grade" on mountain project tend to have comments like "this one is soft" or "this one is sandbagged".

Yeah, it's weird to think of a 5.8a vs a 5.8d... but that's how the computer has to think. The attempt is to bring all the crags in the country to a consistent grading... so nomads (like me) don't have to be surprised jumping on a "5.5" in the Gunks vs a "5.10" at tensleep canyon. I've been to a few dozen crags, and it seems right on (in terms of consistency nationally, NOT locally).

There's nuance to it, though. It's not "this is a 5.9c in terms of technical difficulty". It's "this is a climb for a 5.9c CLIMBER". The difference being it takes into account things like runout, adventure, trickiness with the gear, etc. 

The "our grade" system is probably not for everyone. It works well for me.

Sometimes your "Our Grade" seems to provide neat insight, but other times it just seems stupidly innacurate. At the Gunks Modern Times and Moonlight (5.8 and 5.6) are both acknowledged sandbags and get higher "Our Grades" that make sense (5.9b and 5.8c). On the other hand, Erect Direction is another sandbag (5.10c on MP but 5.10d in the guidebook). It only gets 5.9c. How did that happen? At Ragged Mountain in CT pretty much every climb no matter the actual grade got some version of 5.8 (maybe CT climbers are all 5.8 climbers)? 

Other things seem broken too. When I go to Rumney and tell it to only care about higher star rating, the 4th most recommended route is "Lichen Dike" a climb that has a bomb rating from 2 users.

Very strange. A local route consensus 5.12 you have as 10c. How does that work?

Bryce Adamsonwrote:

 At Ragged Mountain in CT pretty much every climb no matter the actual grade got some version of 5.8 (maybe CT climbers are all 5.8 climbers)? 

Yeah, weissner's slab is definitely 5.8, even if you think it's snadbagged you couldn't reasonably argue that it's harder than 5.5. Same with golden book at cathole, that can't reasonably be argued to  harder than 5.6. 

Also three pines in the gunks a 5.8? That makes no sense, it really is graded right at 5.3

First, this is rad.

Second, wow, my home crag is sofffffft

Andrew Kelleherwrote:

Good question! 

So the first thing I did was I converted the yosemite decimal scale to just a decimal scale... so 5.0=0, 5.0a=1, 5.0b=2...etc.

Next I used a maximum likelihood algorithm to figure out every climber's probability of climbing a climb at a particular grade. I fit it to a beta distribution. That gave me a probability density.

After that I sought to determine the grade of a route, given some set of users has climbed it. 

By evolving the posterior distribution given each climber has climbed the climb; I got 92% accuracy predicting the grade of 45,000 climbs in the US on mountain project. 

Those that are far off from the "true grade" on mountain project tend to have comments like "this one is soft" or "this one is sandbagged".

Yeah, it's weird to think of a 5.8a vs a 5.8d... but that's how the computer has to think. The attempt is to bring all the crags in the country to a consistent grading... so nomads (like me) don't have to be surprised jumping on a "5.5" in the Gunks vs a "5.10" at tensleep canyon. I've been to a few dozen crags, and it seems right on (in terms of consistency nationally, NOT locally).

There's nuance to it, though. It's not "this is a 5.9c in terms of technical difficulty". It's "this is a climb for a 5.9c CLIMBER". The difference being it takes into account things like runout, adventure, trickiness with the gear, etc. 

The "our grade" system is probably not for everyone. It works well for me.

Lol, good luck getting on those 5.10 (our grade) routes in index

saign charlesteinwrote:

I agree. I’m not gonna complain that this thing is calling me soft but there is definitely something messed up when routes where I’m the only tick on mountain project are not down graded. Then objectively harder climbs I have done have a few ticks are downgraded into oblivion. I feel like the system assumes that routes that don’t get many repeats are somehow harder since their success rate is lower in the applications mind however these climbs are often worse or a pain to get to. It seems the more popular the larger the downgrade.

I feel like the best way to do this mathematically is to maxdiff groups of routes that are classics at a crag compared to another crag with the same subset of climbers though this would be computationally difficult.

Yeah, the algorithm needs some work. It has a clear tendency to bump easy routes and downgrade harder ones. Horseman is a reference 5.5 at the Gunks and it's given 8c, while Carbs and Caffeine (11a) is given 9c. Or the Yellow Wall, 11c PG13 given 5.10. Now that's how to sandbag someone!

Thanks so much, everyone! These are fantastic observations.

@Cole Crawford; I think you're right, and it makes total sense that easy routes and hard routes would trend toward the average climber's ability. I'm almost certain this is due to sampling bias. I'll make some tweaks to the algorithm to account for this. There are some pretty well-standardized way to handle it.

And @Trevor Taylor I definitely shouldn't be reporting a projected grade on routes with fewer than ~30 climbers. That's my bad! I'll make sure to tweak that too.

If you like it, please feel free to share! If more people use it maybe some people will probably donate to keep the servers up and running for the rest of us.

@saign challenge yeah, those look pretty tough! The area looks awesome, I'll be sure to pop in when I make my way back west this year.

seems like two things could use doing:

- Are you sure that a gaussian distribution model is accurate to the nature of consensus grades? It seems like just putting the consensus grades scraped from MP would be far simpler or applying a different type of distribution function if you must make a model.

- It seems like you could make the '8a/b' behavior thing into something more in the language of the general community by making an if statement that basically delineates + or - based on if your script spits out something on either side of the b/c line (i.e. if grade.letter == c | grade.letter == d: grade = [str(grade.number) "+"])

Weird. I looked at Goliath (13a) which has 40 opinions, and of those only two downgrade the route to 12+. Yet, somehow your algorithm decided that it is 12a? How does that work?

Still, I’m glad to see that Andrew Bisharat is right. Rifle IS as soft as a baby’s bottom. Bite the Bullet and Apocalypse get the downgrade from .13c to 12a!  :-)

Andrew Kelleherwrote:

So the first thing I did was I converted the yosemite decimal scale to just a decimal scale... so 5.0=0, 5.0a=1, 5.0b=2...etc.

Just commenting to point out: It sounds like your algorithm "thinks" that the difference between 5.8 and 5.9 is the same as the difference between 5.11a and 5.12a. (It's not - the difference between 11a and 12a is 4 times bigger.) This is likely part (only part) of why its assigned grades for popular harder routes are wildly off, because when it downgrades a 12a to 11a, it sees it as equivalent to downgrading a 5.9 to a 5.8.

The encoding that would make more sense is:

5.0 = 0, 5.1 = 1, ..., 5.9=9

5.10a = 10, 5.10b = 11, 5.10c = 12, 5.10d = 13

5.11a = 14, 5.11b = 15...

^ Yep, if the algorithm kept assuming a linear scale from 5.0 to 5.16a or whatever that’ll skew the results.

Finding the proper YDS grading curve fit is an exercise in itself.   

Applying logic to the grading system? Well I see where you went wrong, right there. 

Grading inconsistencies are to be celebrated, not neutered by an algorithm. 

What's next - smoothing brushstroke irregularities in a Rembrandt?

Cherokee Nuneswrote:

Applying logic to the grading system? Well I see where you went wrong, right there. 

Grading inconsistencies are to be celebrated, not neutered by an algorithm. 

What's next - smoothing brushstroke irregularities in a Rembrandt?

Aha!!! Then you are missing the beauty of the heatmap. Some classics are as much as seven grades removed from consensus! Chaos and confusion reigns!!!!

I think some of the locations are just plain wrong....or some cliff line jumped a whole lot closer to me.