If you have access to movement gym's data, I am curious about the comparison across movement gym (e.g., whether movement belmont is easier than movement sunnyvale for each grade.)
Nice piece of work! You may be interested to know that there is some prior art in the scientific literature that uses very similar methodology. This approach can also be used to grade the climbers and how their ability changes through time.
I am a professor in biomechanics (sports science) and a hobby climber who is recently trying to combine hobby with profession. At my university, I run a project on teaching mechanics in biomechanics via bouldering. Your works on route grade estimations are valuable and have potential in my opinion. If you are interested in generating a publication in a scientific journal together, contact me.
If you are keen to collaborate on these methods I would be happy to chat. I am an expert in Bayesian statistics, but my day job is to apply Bayesian methods to genetic data, not climbing :) The climbing paper was a hobby project, but I am keen to see how far it can be taken.
I'd be curious to see what happens if you use the requirement that V5 = V5 +/- 1 to calibrate some of your parameters. I.e. say that all "V5s" are actually V5s, and use that to determine some constants in your formula for how many votes are cast based on the number of attempts.
Another interesting thing to look at is the percentile breakdown of how many climb V8 vs V9 vs V10 etc. In other words, how hard is it to increase 1 grade of difficulty? Maybe there are 100 V8 climbers, 10 V9 climbers, and only 1 V10 crusher.
Hi superfun and intesting! Wondering a bit about the regrading, is this what you are doing: you set the benchmark for say V4 as the median (from the tournament ranking) of all the boulders graded as V4, and a climbs new grade is set via its relation to these benchmarks? I.e. a V4 that is above the median of the V5 boulders, you now grade it V5?
Totally makes sense! We keep the quantity of grades from the original dataset - we just rerank them. E.g if the setters thought that the 15th hardest climb was a V7, then our 15th hardest climb is also a V7. We just have a different opinion from the setters as to which climb is 15th hardest.
There's also a slight nuance where we use decimal grades such that the hardest V7 is a V7.5 and the easiest V7 is a V6.5.
Ah, I see. How would you know which climb the setters thought was the 15th hardest? What more data do you have other than the grade that it was given by the setters?
No worries, I am new to data science, so just trying to understand:)
So in your example above, the reranked V3.5 is the V3 (according to original setters grade) that had the most points from the tournament? Or why did you rerank it as V3.5?
Hi Nathaniel - fellow data scientist + climber here! Very interesting analysis! Would love to run some analysis on the data as well. One thought I had - would the results be sensitive to the mix of climber profiles in Kaya was different? i.e. if Kaya had a larger proportion of intermediate to advanced climbers, rather than a larger proportion of beginner climbers?
It could be, but I'm hopeful that the results aren't be too sensitive to that. The question is - at each grade, are the climbers who logs their climbs in Kaya different from other climbers who don't log in Kaya? It could be, but I'm guessing it's not statistically significant at gyms like Dogpatch where we have quite a bit of data.
Was skill level progression of the climber taken into consideration? i.e. within one year one climber can go from sending a v3 in 5 tries to sending a v6 in 5 tries?
Great question - we did not! We tried to make this less of a confounding issue by looking at a relatively short time period (3 months) - but I'm sure that some of the data is less accurate by virtue of a climber getting stronger.
Let me know if you have ideas for taking this into consideration - especially without making the analysis too complex.
This is great! Would it be possible to expand this analysis to use the entire indoor gym dataset against the entire outdoor gym dataset to get distributions for how each grade compares indoor-outdoor?
Unfortunately Kaya currently has a closed API - I was fortunate enough to get their blessing to do this work. I tried doing a very large comparison between indoor and outdoor grades but found that there's not yet enough outdoor data on Kaya to make that accurate. If you saw the Geek Climber video, you'll notice that only a handful of Bishop climbs have been ascended by a significant amount of Bay Area climbers - and that's a pretty local crag.
One idea would be to focus on benchmark climbs - e.g Midnight Lightning (V8) - and do slightly more bespoke analysis.
If you have access to movement gym's data, I am curious about the comparison across movement gym (e.g., whether movement belmont is easier than movement sunnyvale for each grade.)
Nice piece of work! You may be interested to know that there is some prior art in the scientific literature that uses very similar methodology. This approach can also be used to grade the climbers and how their ability changes through time.
2020: Estimation of Climbing Route Difficulty using Whole-History Rating https://arxiv.org/abs/2001.05388
2021: Bayesian inference of the climbing grade scale https://arxiv.org/abs/2111.08140
The second paper is my own.
Super cool! I'll read these and comment back later!
@Nathaniel and @Alexei
I am a professor in biomechanics (sports science) and a hobby climber who is recently trying to combine hobby with profession. At my university, I run a project on teaching mechanics in biomechanics via bouldering. Your works on route grade estimations are valuable and have potential in my opinion. If you are interested in generating a publication in a scientific journal together, contact me.
www.philipxfuchs.com
If you are keen to collaborate on these methods I would be happy to chat. I am an expert in Bayesian statistics, but my day job is to apply Bayesian methods to genetic data, not climbing :) The climbing paper was a hobby project, but I am keen to see how far it can be taken.
Really cool project!
I'd be curious to see what happens if you use the requirement that V5 = V5 +/- 1 to calibrate some of your parameters. I.e. say that all "V5s" are actually V5s, and use that to determine some constants in your formula for how many votes are cast based on the number of attempts.
Another interesting thing to look at is the percentile breakdown of how many climb V8 vs V9 vs V10 etc. In other words, how hard is it to increase 1 grade of difficulty? Maybe there are 100 V8 climbers, 10 V9 climbers, and only 1 V10 crusher.
As a fellow climber at Dogpatch, very interesting analysis! Thank you for taking the time to do this
Hi superfun and intesting! Wondering a bit about the regrading, is this what you are doing: you set the benchmark for say V4 as the median (from the tournament ranking) of all the boulders graded as V4, and a climbs new grade is set via its relation to these benchmarks? I.e. a V4 that is above the median of the V5 boulders, you now grade it V5?
Hope that makes sense:)
Totally makes sense! We keep the quantity of grades from the original dataset - we just rerank them. E.g if the setters thought that the 15th hardest climb was a V7, then our 15th hardest climb is also a V7. We just have a different opinion from the setters as to which climb is 15th hardest.
There's also a slight nuance where we use decimal grades such that the hardest V7 is a V7.5 and the easiest V7 is a V6.5.
Ah, I see. How would you know which climb the setters thought was the 15th hardest? What more data do you have other than the grade that it was given by the setters?
I think I described that in a slightly confusing way. Imagine 3 climbs of each grade V1 - V3. The reranked climbs will get the following grades.
1. V3.5
2. V3.125
3. V2.75
4. V2.375
5. V2
6. V1.625
7. V1.25
8. V0.875
9. V0.5
No worries, I am new to data science, so just trying to understand:)
So in your example above, the reranked V3.5 is the V3 (according to original setters grade) that had the most points from the tournament? Or why did you rerank it as V3.5?
Hi Nathaniel - fellow data scientist + climber here! Very interesting analysis! Would love to run some analysis on the data as well. One thought I had - would the results be sensitive to the mix of climber profiles in Kaya was different? i.e. if Kaya had a larger proportion of intermediate to advanced climbers, rather than a larger proportion of beginner climbers?
It could be, but I'm hopeful that the results aren't be too sensitive to that. The question is - at each grade, are the climbers who logs their climbs in Kaya different from other climbers who don't log in Kaya? It could be, but I'm guessing it's not statistically significant at gyms like Dogpatch where we have quite a bit of data.
Was skill level progression of the climber taken into consideration? i.e. within one year one climber can go from sending a v3 in 5 tries to sending a v6 in 5 tries?
Great question - we did not! We tried to make this less of a confounding issue by looking at a relatively short time period (3 months) - but I'm sure that some of the data is less accurate by virtue of a climber getting stronger.
Let me know if you have ideas for taking this into consideration - especially without making the analysis too complex.
This is great! Would it be possible to expand this analysis to use the entire indoor gym dataset against the entire outdoor gym dataset to get distributions for how each grade compares indoor-outdoor?
Would you be willing to share your code for generating these results? I would love to get my hands on the Kaya API and do some of my own work.
Unfortunately Kaya currently has a closed API - I was fortunate enough to get their blessing to do this work. I tried doing a very large comparison between indoor and outdoor grades but found that there's not yet enough outdoor data on Kaya to make that accurate. If you saw the Geek Climber video, you'll notice that only a handful of Bishop climbs have been ascended by a significant amount of Bay Area climbers - and that's a pretty local crag.
One idea would be to focus on benchmark climbs - e.g Midnight Lightning (V8) - and do slightly more bespoke analysis.
Love it. Would be fun to see a list of the outliers at each grade dogpatch. (The video was a great teaser!)