return to table of content

Show HN: AI climbing coach – visualize how to climb any route based on your body

prions
23 replies
1d3h

It looks really interesting, but as an experienced climber I'm not sure if just watching a video of my avatar climbing would really help with skill acquisition.

Also, this claims that the wall type or video quality doesn't matter, but I have a hard time understanding how the model would be able to understand that a small crimp could possibly be dual textured and therefore has only a few specific ways of approaching it.

So it seems that this is more for visualizing a climb (which is a skill most climbers should develop) and not really for dialing in some sort of microbeta for a problem.

etrautmann
9 replies
1d2h

Agreed - so much about the detail of how you would climb something comes down to details that would be hard to measure with a camera, like textures, your estimate of friction, etc. Very cool idea though, looks fun to test.

sandworm101
8 replies
1d

And this is deep into "sport climbing", borderline gym rat territory imho. It doesn't model all the other core aspects such as protection, rope management, exposure and rest stops. I imaging if you pointed this at a real cliff and recorded several assents it would quickly become a blurry-twitchy mess as all the movements not touching the rock spoiled the data. Maybe for bouldering, but not for real climbing.

soperj
7 replies
1d

What makes rope climbing "real"?

lambdaone
6 replies
20h54m

There are lots more variables to consider, particularly in lead climbing, even when you have a bolted route. And trad climbing is even more complex than that.

sandworm101
5 replies
18h38m

Trad lead is climbing. Basically everything else is some sort of simulation.

et-al
4 replies
18h22m

It's 2024 and some people are gonna gatekeep that if you're not shoving cams in some bigwall granite, it's not climbing?

lambdaone
1 replies
10h42m

This one-upmanship is very much part of climbing culture. From the top down, the hierarchy is: free soloists (climbing at height without any protection), trad climbers (carry the rope up with them, set their own protection), sports lead climbers (carry the rope up with them, bolts are set into the wall), top rope climbers (rope already there dangling from the top). Right down the bottom, you have aid climbing, when you use equipment to haul yourself up on a rope.

And then there's indoors vs. outdoors, with some dedicated outdoor climbers regarding anything done in a climbing gym as "not real climbing".

Most people don't take this literally, and it's generally considered to be a standing joke in climbing. Sadly, though, some people take it very seriously.

sandworm101
0 replies
3h49m

Aid climbing isn't respected on smaller or well-established cliffs, but most understand that it has a place. All the great routes started as aid routes, only being climbed "free" years later. Aid is also a very useful skill in the rain or rescue situations when friction disappears.

sandworm101
0 replies
18h4m

Such differences are actually at the heart of climbing. Is taking a helicopter up to the top climbing? Of course not. So from day one it is not about getting to the top but about following invented rules governing how you get to the top. How about a bolted-on ladder? Or pre-placed protection (bolts)? Clean trad, leaving the rock as you found it, is generally seen as the highest form.

(Climbing rock without ropes may be more "pure" but is so dangerous that it should never be idolized.)

Intermernet
0 replies
11h40m

Cams? Real climbers use stones wrapped with hawser!

logtempo
7 replies
1d1h

I don't see the shape of the holds being a big problem. With some help from indoor companies and hold makers, figuring out which hold model is on the wall should be possible.

As for the usefullness of the software, I'm sceptical too as it don't really solve a problem. But maybe I'm not seeing it and it could be good for beginners :) A good improvement would be adding a comparison between you and the model in term of body position and fluidity of movements.

party_possum
5 replies
1d

The idea of incorporating actual hold data and "recognizing" specific holds is interesting, but I'm not sure it completely solves the problem.

The "Boss" from Pusher is arguably the most famous climbing hold ever made. For a decade or more, every gym had one, but they were all unique. Lots of them had micro chips that became critical to usage of the hold. Some had decent texture and some were glassy smooth from years and years and years of use. A lot of the accidental variation in new holds has gone away as the industry has standardized around a handful of industrial fabricators like Aragon, but even over the course of a single indoor boulder problem's life, the accumulation of chalk, sweat, and shoe rubber can have a significant impact on how a hold climbs.

I guess the real question is, do these changes just make routes harder or do they make them fundamentally different? Do they actually change the set of moves that constitutes the easiest way to the top? To be honest, I'm not entirely sure. But it's something interesting to think about.

alexcp_
3 replies
1d

Exactly, holds will evolve as they get used and more polished, even indoors. Climbing a Moonboard with a new set of holds is quite different than climbing on one with older more polished holds, even if it's the exact same problem and the same holds.

It's an interesting project and it could be fun to watch, but it's completely useless.

ethbr1
2 replies
23h27m

Couldn't you reverse-reason about that?

To me, the customer here would be climbing gyms, offering a service to climbers.

   1. Set up camera on routes
   2. Record all climbs
   3. Reason through hold details
   4. Generate potential movements
   5. Show climbs vs ghost movements
   6. Feedback to tune model
3 being accomplished by reasoning "if a movement should be possible using the identified hold, but no one successfully does it, the hold must be misidentified or have different properties."

alexcp_
1 replies
22h17m

But what is the point? Finding the optimal movements that are needed to complete a climb is not useful if you are not strong enough to execute it.

ethbr1
0 replies
22h7m

The point in this thread seemed to be "real world holds have different properties, and that defines possible approaches to holds."

To which I pointed out that, with enough data, you could reason backwards to figure out their properties.

Assuming that's solved, if the question is "What is the point?" then I'd answer the same point as golf swing analysis -- structured comparison feedback for continual improvement.

"Have you thought about trying X move at Y point?" or "You're trying X move at Y point, but here's how you differ from someone successfully doing it" both seem useful feedback.

And essentially what's manually generated now, from someone watching and then providing feedback.

With regards to strength, hell, if you wanted to get fancy you could also deduce a specific user's strength, comparing their moves against others' moves on the same features.

jfengel
0 replies
17h59m

Huh. I recognize it but didn't know its name. (I don't know any of the names.) Route setting sounds fascinating.

Zanfa
0 replies
23h36m

I don't see the shape of the holds being a big problem. With some help from indoor companies and hold makers, figuring out which hold model is on the wall should be possible.

Even if you know the exact hold model and it’s in pristine condition, it’s basically impossible to tell how it’s gonna work from a single angle on video at a distance. Even tiny variations in angle of the wall and rotation of the hold on the wall can completely change how you use it.

petsfed
2 replies
22h34m

I suspect that, absent that information about the exact right way to grab a hold, or the exact way to put a foot on a hold, you'll be limited to beta suggestions, which is fine, I think. It'd be like having a group of other climbers nearby to suggest different beta, even if you don't have any friends.

So, in terms of solving complicated beta faster, I see real utility to this.

It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.

smandava
0 replies
15h17m

Correct

bmj
0 replies
6h30m

(Long time climber here)

It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.

I would think this is actually a Bad Thing. It's very easy to get stuck trying to make a sequence fit your style of climbing. The better approach (especially for long term skill acquisition) is a willingness to learn new styles. That's to say that every sequence is only solvable via one particular style, but I think long term development is hindered if you approach every crux with the one thing you are good at.

So, in terms of solving complicated beta faster, I see real utility to this.

I can agree with this. But, to the point that others have made, I do wonder what this and the availability of beta videos for many, many routes and blocs does to climbing skill overall. Perhaps I'm just a grumpy old man, but, particularly when bouldering, sorting out the beta should be part of the journey toward eventually sending. Last fall, I visited Hueco Tanks after a six year absence. I suppose I was a bit disappointed to see so many people watching YouTube beta videos of nearly every problem they tried.

smandava
0 replies
15h18m

To clarify:

Of the full distribution of possible video qualities one can take on a modern phone camera, the vast majority of video qualities will be fine for the AI to understand fine details. Obviously, if you somehow or for some reason, take a video with really bad quality, it will not give you what you want.

Same explanation goes for the walls. If you take a video of just a really dark wall with really bad holds, it is probably won't give you what you want either.

Snackchez
0 replies
1d1h

What could be interesting is if you could compare your attempts to the avatar climbing and receiving feedback afterwards. This would effectively be a step up from simply recording your send attempts.

party_possum
15 replies
1d

Climbing route setting (when done at a high level) is constantly in conversation with itself. Many climbers don't realize that there are trends or fads in movement styles that sweep across the industry. This has only picked up steam in an era of social media.

Just like movie dorks will happily spend hours explaining how an individual shot in a movie is actually an insider reference to another movie, and as a result a statement of intent for the movie as a whole, professional route setters will talk your ear off about the way one of their problems embraces or rejects specific kinds of movement trends of the last 6 months.

That intentional rejection is interesting. Many route setters, especially for competitions, are in constant search of novelty. One kind of perfect problem is something that looks confusing and impossible, up until you see it done, at which point it seems almost obvious. It's the feeling of solving a sudoku. But critically, they want climbers to be initially confused.

I wonder if AI might actually be better than humans at sequencing these kinds of problems. Humans bring so much context and experience and expectation to the process that we are easily tricked. AI just looks through a few terabytes of video and says "What about this?".

chefandy
13 replies
22h59m

I wonder if AI might actually be better than humans at sequencing these kinds of problems

Probably a deeply unpopular take here, but without knowing anything about climbing routes, I'm gonna say no. I'm not saying that they won't have excellent quality output that might even solve problems that human output can't, but the process of creating something is meaningful, even commercially. Surely this will be useful in some respects, but I just don't buy the idea that humanity is destined to passively consume automated algorithm-generated utility products-- especially creative ones-- no matter how smooth, cheap, and clever they might be.

petsfed
6 replies
22h26m

Well, the tricky bit here is that the route setter, a human, is the one actually solving the problem. So the problem as set is (and must be) a human creation first. This is especially true in outdoor climbing, where the first ascent process might involve the installation of anchor fixtures, or the removal of poorly-secured features for safety. You'd need some pretty wild sensor suites to correctly differentiate between a really good hold, and a dangerous flake that will peel off the wall if the slightest force is applied to it. The AI just generates potential solutions to the problem once the holds are found/placed. Certainly, there's some interesting conversations about how satisfying it is to solve a rubick's cube using somebody's algorithm vs. just figuring it out, but its not like the computer is inventing a rubick's cube.

Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall. I expect that for a long time, AI generated climbing routes would rarely be good, but would consistently be physiologically impossible, feature uninteresting movement, or be too easy.

Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.

rockfishroll
2 replies
21h40m

Spotting a refugee from rockclimbing.com on hacker news was not on my bingo card for the day. But I guess if I'm here (writing novels about route setting) then I shouldn't be surprised other people are too.

petsfed
1 replies
21h22m

Outside of a short-lived usurper on instagram selling pet food dishes, I am still the only petsfed on the internet, since 1997.

What's really wild to me is how somebody would recognize a mid-tier poster from a website I thought effectively defunct for nearly 10 years now.

rockfishroll
0 replies
20h24m

It just goes to show how impactful online communities are capable of being. rockclimbing.com was in it's heyday right as I was discovering climbing. I was a bored kid constructing my entire identity around climbing and there was no other place to do that outside of the gym. No mountain project. No youtube. No social media. I spent a lot of hours lurking those forums. There are only a handful of users I could still name, but I bet I would recognize a lot of them.

ethbr1
1 replies
22h0m

Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall.

This would follow the exact path image GenAI evolved through.

Step 1: Teach a model to recognize objects from noisy data.

Step 2: Reverse-feed that model random noise and force it to hallucinate that noise back into likely objects.

As there's probably physics simulation at some point in this particular scenario, there'd probably also be step 3 of simulating a climb through the generated path to validate feasibility / specific qualities.

It doesn't sound impossible.

petsfed
0 replies
21h20m

Its certainly not impossible, but that physics simulation is the biggest obstacle that I can see.

chefandy
0 replies
22h6m

You'd need some pretty wild sensor suites to correctly differentiate between a really good hold

Ah, I see.

The AI just generates potential solutions to the problem once the holds are found/placed

Yeah and I think that's really going to be the sweet spot for generative tools for the forseeable future.

Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.

Right right. I have a feeling that making a more convincing substitute is primarily a matter of having less access to data than say, paintings and photography which are certainly not less nuanced than this creative task. But as I said, a lot of people care about how something was made, too. I'll bet that's going to be a much bigger factor, at least in marketing, than many realize in the near future.

JadeNB
5 replies
22h53m

the process of creating something is meaningful, even commercially.

That's true, but why does it mean that the answer to the more or less objective question "will AI actually be better than humans at sequencing these kinds of problems?" (As stated it's not really objective, but one could easily come up with metrics like, say, total time to a correct solution, or time spent observing the route or other climbers, or ….) One can imagine other, related questions that are less objective (like "will it be a good idea to integrate this AI assistance into climbing competitions?"), but, to me, the answer to the (implicit) original question has nothing to do with whether or not the activity is meaningful, or with humans' destiny one way or the other.

chefandy
4 replies
22h45m

Sure, they may well be quantitatively better. If you were to create metrics to measure the number of mistakes or weird spots or overly annoying things, it's quite possible that the output from the algorithm could score better than human output, and the throughput would obviously be incomparable. But, whether or not something is qualitatively better is far more subjective-- it's influenced by our culture, our experiences, and everything else that creates the lens through which we see the world. Something's origin absolutely affects the way people experience it, be it a physical object, story, experience, etc. Don't get me wrong-- I realize there's real value in affordable quantity with with mediocre quality-- how many restaurants in the world are McDonald's? But then again, how many restaurants in the world aren't McDonald's? If McDonald's could sell you a ribeye comparable to Capital Grille, I'd be astonished if it put a dent in Capital Grille's bottom line. Applebees, however...

travisjungroth
1 replies
22h21m

I think your arguments against AI being better at creating routes reduce to:

1. You don’t like what that would mean about the destiny for humanity.

2. A human making it makes it inherently better.

3. If it’s lower quality, it’s lower quality.

I get why this would lead to strong beliefs. But these arguments aren’t very convincing.

chefandy
0 replies
22h15m

I think you probably need to re-read what I wrote. I'm not against AI at all-- I use it all the time. I just don't think it's going to be qualitatively better than human creative output because how something was made matters to people. I also think the tech world thinks they're a lot better at judging creative output than they are.

JadeNB
1 replies
22h38m

Right, those are important considerations, but I don't think that they're the same consideration. If you ask whether a celebrity chef is better at cooking a certain dish than I am, then the answer (no matter the dish, for I'm not much of a chef) is almost certainly "yes." If I answered instead no, that the provenance of a dish matters, that the celebrity-chef culture is bad for fine dining, and that my wife would rather have the dish made by me, then I think I'd be regarded as missing the point of the question, even if my objections were true on their own.

chefandy
0 replies
22h17m

Well, I think that celebrity vs husband is not a good analog for human vs generated. I also think that in this case, origin has broader connotations than provenance, in that it includes context. What if the celebrity chef was your wife's brother? What if you had a degenerative neurological disease? What if it was the dish you cooked for her on your second date? Anything can seem very cut and dried if you impose artificial restrictions and examine it in a vacuum. And, because you can't easily define and quantify things like context, the engineering and tech worlds tend disregard them because they don't really factor into the engineering equation... and that's why designers exist.

pierrec
0 replies
18h52m

You mean AI might be better at setting or at reading? I feel like your question is getting misinterpreted.

Temporary_31337
10 replies
21h13m

I feel like I’m the right audience for this - a not very good climber with unusual distribution of limbs. But I’m not interested - part of the fun in climbing is on sight climbs and solving problems yourself.

johnfn
4 replies
16h5m

Man, where is the hacking spirit? OP posts a very cool, impressive hack and all everyone can talk about in the comments is how it actually isn't interesting to them.

swores
1 replies
10h34m

It's great if someone wants to use the "hacking spirit" to help themself climb, it doesn't obligate everyone else to want to use the same technique; and frankly it's (slightly) more interesting hearing people say why they would or wouldn't use it than to have a bunch of comments just saying "awesome that you're doing this" and ignoring whether they'd find it interesting themselves or not.

SOLAR_FIELDS
0 replies
10h22m

I would say (opinion) that it’s vastly more interesting to hear whether other people would use Thing I Built. I can nerd snipe myself all day on projects. By that I mean I can easily endlessly entertain myself building things, both useless and useful. Whether they are interesting AND useful for other people is quite a large driver for me both for personal and business projects

jajko
0 replies
3h34m

If somebody solves the 'hacking' part for you, a lot of fun disappears. Yes they probably had tons of fun, but you lose some of it.

Figuring a route for the first time is by far the most rewarding part of climbing (although I love all of it). Same with kids - if they grok something on their own, what a reaction and reward compared to being told how to do it.

anvil-on-my-toe
0 replies
2h40m

It's cool as an academic exercise, but there are things in life where the struggle is kind of the point. The messy learning process, asking other humans for help, is better for your growth.

perfect_wave
1 replies
15h47m

Came here hoping to see something like this. Glad to see it’s the top comment. Part of the beauty of climbing is solving a puzzle and figuring things out.

Edit: Unsurprised to see the demo on a gym route instead of actual rock.

smandava
0 replies
15h45m

Will post more videos on outdoor rocks soon! I just chose that video for the demo because it was easy to explain the route

cloudripper
0 replies
2h23m

As an experienced, very poor climber myself, I enjoy the process of problem solving. However, knowing many experienced, very good climbers - studying beta is how many of them excel. As an aside, the term beta also came from the use of beta tapes by climbers to record themselves climbing so they could study the minutiae of their movement to find improvements. In higher levels of bouldering especially, a nuanced and firm understanding of beta is everything - so careful analysis of your movement can help identify areas of improvement.

I believe this is consistent with most elite (or elite-aspiring) athletes from many sports.

While I personally enjoy the problem-solving aspect of climbing (when I rarely do get out), I absolutely see the value in this project (and other climbing apps that thrive on beta sharing)

carabiner
0 replies
11h59m

If you ever Moonboard, you'll see the app gives easy access to beta videos from Instagram. That's because many people want help on really hard problems. OP app is like those videos.

If you never feel like you need help from watching someone else climb, then I think you are not trying hard enough problems.

31337Logic
0 replies
18h19m

Thank you for posting this. :-)

smandava
6 replies
2d16h

Note:

Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!)

I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app.

If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it.

Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).

ethbr1
2 replies
23h34m

High-level details about how you tackled the various sub-problems would be useful.

I assume, at minimum, there's:

   - Segmentation of moving elements
   - 3d reconstruction from 2d video
   - Reverse kinematics
   - Identification of holds
   - Search for potential climbing path
   - Generation of kinematics for path

hirako2000
1 replies
23h27m

Likely looking to see a 100k emails list before really thinking about all these problems to solve and well.

Worst case able to build this, scenario, author builds something else entirely.

ethbr1
0 replies
22h13m

Afaict, author already built a proof of concept.

crabmusket
1 replies
16h17m

*temper expectations!

smandava
0 replies
15h44m

typo :) thanks

icpmacdo
0 replies
21h17m

I've been working on the same thing, there is an interesting quote from back in the day and its either make your program faster or wait 6 months for the computers to get faster. I look at it the same way now with Key Point Analysis and Pose Estimation libraries.

I think OSS is best for this category right now.

kettunen
6 replies
1d2h

One of the most fun part of climbing is the problem-solving aspect of it. Figuring your way to get up the wall/boulder. Why would you want to get rid of that?

crazygringo
1 replies
22h38m

What about when you try to solve the problem and you get stuck and can't? Or solve it but in a bad way that requires way too much exertion?

The point is to try to solve it yourself, then compare with an expert solution, and therefore learn how to improve.

If you're just blindly trying to problem-solve through trial and error without ever comparing against expert feedback, you're going to learn climbling extremely slowly...

Tronno
0 replies
20h24m

Getting stuck is part of the game. It could take several sessions or it could take several years. It's what makes a climb truly hard, a "project".

It's an essential part of the sport - the satisfaction of using one's own body and mind to overcome the seemingly impossible.

If you never get stuck, how could you possibly experience it?

rokkamokka
0 replies
1d2h

As with any tutor, this could (possibly) help beginners.

hacksoi
0 replies
1d1h

You can use it after you do the climb to see if there is a more optimal solution.

_fs
0 replies
1d1h

Do you enjoy climbing with others? Personally, I enjoy seeing the different beta on a climb, especially between different body types. It is interesting seeing unique approaches to the same problem.

NegativeK
0 replies
23h27m

Where the fun is is an opinion.

hangonhn
6 replies
1d1h

This is really cool from a technology point of view but the main attraction of climbing for me is figuring out how to "solve" the climb. Trying to solve a crux with a buddy is half of the fun for me.

crazygringo
4 replies
22h39m

Right, but for people in their first couple of years of climbing, it can be really helpful to try to solve it, then see what the AI coach says, examine the diff, and thereby improve your solving and climbing skills.

I have definitely discovered in my life that I learn probably 10x faster in situations where I can try something, and then immediately compare it with an expert solution. Rather than just trying and trying and trying on my own.

And once you're experienced in climbing (like you probably are), then of course you don't need to compare anymore. The joy is in the solving. But expert solutions are really helpful to get to the point where you can consider yourself experienced, so much more quickly.

et-al
3 replies
22h17m

With bouldering, the set of moves is fairly limited and what actually prevents someone from pulling harder comes down to flexibility, core strength, and contact strength.

crazygringo
2 replies
20h52m

OK, but the submission and conversation here isn't about bouldering.

This is about wall climbing where there's lots of choice for moves.

et-al
0 replies
19h20m

Well if we're talking about "wall climbing", the set of moves don't expand that much as one progresses. Your technique needs to be more efficient and your transitions and balance on point, but it's not like there's a esoteric 13a move that a 10d climber hasn't seen. Sure the first time someone dyno'd on a roped climb was a sea change.

Bouldering arguably has a larger set of moves than "wall climbing", but again the physicality of this sport is the crux for most people. One can study and maybe drill all the "advanced" moves, but without the flexibility and strength to execute on microholds, that knowledge is useless.

IanCal
0 replies
19h39m

I'm not sure what definition you're using here but the example video on the main page is bouldering.

eigelc
0 replies
23h12m

And a quarter of the other half is asking people in the gym how to do it.

goodmattg
5 replies
1d3h

A few thoughts as I've done academic research & built products in this area:

- if you're using SMPL body parameters this will have to stay research / open-source - is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing.

Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.

syntaxing
1 replies
1d2h

Not OP, but why does it have to be open sourced? Copy left license?

goodmattg
0 replies
1d1h

commercial license - the research group formed a corporate entity that licenses the body model and all derived work (SMPL-X, etc.): https://meshcapade.com/SMPL

smandava
1 replies
15h27m

Yes, this will be open source.

This is an end-end system that just takes in video frames. Camera parameters are one of the things that is predicted. It gives promising results for a wide variety of environments (cliffs, diff types of bouldering walls, diff outdoor walls, etc.), though not always accurate. Path planning is also part of the end-end system. Will share more details in the paper.

goodmattg
0 replies
3h52m

sweet can't wait to read it! I'll also be at CVPR this year if you're presenting

ethbr1
0 replies
23h25m

the tricky part that I'm impressed you have a solution to is path planning up the wall.

I'm assuming this is evolutionary / brute force of some nature, given OP's comments about it being expensive to run.

latchkey
4 replies
1d

Of course everyone on HN is focused on the climbing / beta / technique aspect of this.

It is ok that it is a solution looking for a problem. There is obviously no 'business' or 'product' here. It isn't like there is a payment link on the page or anything.

What I'd like to see the comments focus on is that we should just be happy that someone is making the effort to learn more about AI and building tooling around it. Experimentation is king.

They've put their work out into the open (soon to be open sourced even!), not to be criticized over whether it is useful or not, but just that they created something that could spawn other interesting things that solve real world use cases.

Huge kudos for doing this work.

iosguyryan
2 replies
18h31m

If I could “yes and” this… route setters could benefit from this tool to simulate layouts against a number of body types and mobility.

ungreased0675
0 replies
18h16m

Yes, and help setters create more interesting routes. Maybe a layout with multiple ways to solve it, but all the approximate same difficulty?

latchkey
0 replies
18h7m

I spent 8 years lead climbing at Mission Cliffs in SF 2-5 days a week (5.12 range was my top). They had excellent route setters there that I spent many hours just watching between climbs.

The route setters would effectively do what you're saying, while they were setting the route. They knew exactly what would work and what wouldn't. The holds/wall are really what dictate things and if you're not able to climb it... it isn't really their problem.

As far as I can tell, while this project is cool, it really has nothing to do with climbing or route setting. It is an AI project where the developer just kind of made up some task and had AI follow it.

iosguyryan
0 replies
18h34m

really good example of the culture of climbing too!

takk309
3 replies
1d

Could this be used to grade routes? It would be cool if a photo or short video of a route could be used to better assist in grading climbing routes. Grading is a massively subjective thing, to some extent, so this could be super tricky.

et-al
2 replies
23h45m

I only watched the video, but without taking in account the texture of the holds, the grading aspect would be limited.

One also learns to just recalibrate according to regions. E.g. climbing 11d at one gym, versus 12a at another.

mordechai9000
1 replies
22h56m

Also, how will the AI know when it is appropriate to inflate the grade to make the climbing seem harder, and when it is appropriate to sandbag so I can pretend a route is easier for me than it actually is?

takk309
0 replies
22h5m

Easy, it'll just grade everything 5.9+!

gus_massa
3 replies
3d3h

Does the AI create the 3D model from a static camera?

Can I choose the color of the avatar? (It's not an important feature, but people will love it.)

Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)

smandava
2 replies
2d16h

- it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks.

- don't have that functionality but its really easy to implement.

- ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.

gus_massa
1 replies
1d23h

My guess is that for the computer it's easier in an artificial wall becasue features are standarized.

(Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))

smandava
0 replies
1d16h

Yeah it performs best with indoor bouldering videos with spread out holds and great lighting. I will post examples of it climbing natural walls in the paper. Will send soon.

surume
1 replies
21h13m

The AI analysed my body and said coldly, "Just take the elevator."

amelius
0 replies
5h0m

Here is an interesting post about how body mass affects climbing ability.

Even if you have a great BMI and muscular build, but if you are heavy, then at some point physics starts working against you.

https://www.reddit.com/r/bouldering/comments/595a2m/how_much...

The basic physics of strength/weight is that strength is reliant on the cross sectional area of muscles (number of fibers) and therefore increases quadratically, whereas weight is a function of volume and increases cubically. So the amount of strength you can gain as your mass increases will always lag behind the amount of mass gained. This is why insects are so strong compared to their bodyweight, and why the creatures with the highest strength/weight ratio's are all small.

If we analzye male pro climbers we'll see that the heaviest of them max out at around 170-180 lbs (and there are VERY few of those, the majority being between 130 and 150), and that's not a fluke. In order to be a great rock climber you need strong forearms, and strong lats, and that's about it. Muscle mass, or fat mass basically anywhere else is mostly just pulling you off the wall (a strong core helps too but it doesn't need to weigh a lot).

(...)
silentsea90
1 replies
1d2h

Very cool!! I am unfortunately very far from an AI expert. Can you share how you built this? Did you have to train your own model, or is it an open source model that you finetuned for climbing?

smandava
0 replies
17h58m

It is a new type of biomechanics AI model. More details in our paper which is coming soon!

raptorraver
1 replies
5h14m

Cool project but please delete it. You’re taking the fun part out from bouldering and just leaving us the labour ;)

Edit: apprently I wasnt the first to make this joke. I emphasise that this is Really a cool project. Great job!

party_possum
1 replies
1d

It would be really interesting to build an instance of this model trained on world cup footage for a few reasons.

- Like all of these things, your training data matters and the internet is awash with videos of people climbing badly. A lot of people specifically post "I can't climb this, what am I doing wrong?" videos. World cup climbers are, by the nature of the competition, extremely talented and technically proficient climbers. Even when they fail, they fail in smart interesting ways.

- There's lots of high quality video footage out there. Heck, the problems are even set with visual clarity in mind which would help when parsing that footage. There's potentially enough video to train instances on individual climbers. You could run side by sides like "How would Tamoa climb this and how would Janja climb this?".

- World cup problems are stylistically distinct. They involve lots of moves "typical" climbers will never ever encounter. Many climbers will look at a typical gym problem and think "I have an idea of how to climb this" but will look at a world cup problem and just think "????????". An app that told you how a problem like that should be climbed might be useful.

There are drawbacks too.

- World cup climbers are outliers, whose physical ability (strength, flexibility, etc.) give them access to kinds of movement that other climbers just don't have. No amount of "knowing the sequence" will get me up a climb that requires a full bat hang (look it up) because I just don't have the ankle strength to do the movement.

- World cup "style" is only commonly used at high level comps and in very large commercial gyms. It's probably not extremely relevant to a typical climbing session.

- World cup problems are very hard. Mostly v10 and up? It would be hilarious to watch a model trained on genetic monsters crushing the world's hardest boulder problems try to tell a doughy office worker (me) how to climb v2.

nl
0 replies
17h51m

This was exactly my thought. See https://www.instagram.com/p/Cvg3bJkp30a/ on a super-hard slab problem at Innsbruck 2023 (Sorato Anraku couldn't do it, so you know it's hard...)

nsagent
1 replies
18h54m

Great job! A lot of folks from my CS PhD program are really into climbing (including me), but none of us has conducted research that explored climbing. Rather climbing is our respite from research.

Anyway, I think this could work really well as an app. Good luck.

smandava
0 replies
17h52m

Thanks! Still unsure if we will commercially pursue this specific idea or not. This was moreso meant to be a public progress marker from our broader work pursuing virtual avatar simulations. If enough interest is there, we will think about it. But, for now, this (will be) a paper and some open sourced code.

jfengel
1 replies
2d1h

I look forward to playing with this. It seems astonishing.

smandava
0 replies
1d17h

I will be open sourcing the codebase and model. Due to compute and training dataset size constraints, the model is not exactly "production ready". I have (and am) cherry picking rare examples of it where it is decent. You can play around with it when I open source it but truly good results require multiple millions of dollars (but it really is a linear scaling of compute and dataset).

eps
1 replies
19h52m

Your demo video is monetized and showing 30 seconds of ads. Might not be a bad idea to switch the ads off.

dathos
0 replies
5h58m

Pff, they're sharing their work for free and you complain about a 30s ad?

colehud
1 replies
18h49m

Saw tennis on your LinkedIn - is SABR a nod to the Sneak Attack By Roger?

smandava
0 replies
18h2m

Ha! That was unintended. "You can't connect the dots looking forward; you can only connect them looking backwards." - someone somewhere (likely Steve Jobs)

alanzhuly
1 replies
19h40m

This is really neat. Do you plan to expand this to other sports or activities as well?

smandava
0 replies
19h39m

Maybe! Depends on the reception.

tuatoru
0 replies
14h30m

But does it do Hammersmith Broadway?

teapourer
0 replies
17h34m

Looks cool! It definitely looks technically challenging, especially for expert climbers who have a variety of creative moves at their disposal. But I can definitely see this being useful for novices.

southernplaces7
0 replies
41m

Considering the notoriety of AI models in making confident but blatant mistakes or simply lying about things they don't have real answers to I don't know how smart it is to trust one with something in which bad advice can kill you very quickly and immediately.

some_random
0 replies
2h24m

This is rad and I can't wait to try it out for myself

ropable
0 replies
18h43m

I'm genuinely curious how the model classifies holds. I could (hypothetically) see this doing something like assigning more utility to certain holds (e.g. a steep sloper that is impossible to use as a handhold but is can be used as a smearing foothold) and thereby giving bad/implausible beta. That being said, it seems like an obvious shortcoming and I'm very interested in learning more!

programmarchy
0 replies
1d

Not a climber myself but this is super impressive. Curious about what techniques you’re using. Is this based on a GAN model?

piterrro
0 replies
1d3h

Nice! Cant wait to try it!

nl
0 replies
17h54m

I have a side project in a related space. For mine I'm syncing climbing between different videos automatically. I find this pretty useful because it highlights difference.

Some examples:

https://www.instagram.com/p/Cvg3bJkp30a/

https://www.instagram.com/p/Cpuv2zvptdW/

It's actually syncing in 3D so is fairly independent of camera angle (as the second video shows)

nachteilig
0 replies
3h50m

Very cool project. I will try this out when you release it

monroewalker
0 replies
12h30m

This is cool and all but I can't help but wonder if it'd be even cooler to instead show the beta for climbing out of the negativity found in the comments here. /s

But really though, awesome project and congrats on getting it to this point! I'm excited for the day when something like this can be used to critique form

lambdaone
0 replies
21h0m

I'm sceptical about the usefulness of this for a number of reasons, including the points raised by others about hold recognition etc.

It's also difficult to see how the system would work out the 3D arrangement of holds. It can clearly try to infer 3D relationships from video of an existing climb, but it would be hard for it to work out the 3D position or orientation of any holds not used by the climber in the video.

These two put together make solving the climbing problem even more difficult, because appropriate body positions and moves are often very sensitive to even tiny differences in shape, relative position, and orientation of holds; and occluding volumes, arêtes, cracks and so can make the problem even harder.

But as a simplified demo, this is cool, and I salute it.

kriro
0 replies
9h16m

Very nice. Please do open it up as much as possible. I'm curious about the data pipeline. I love bouldering and have thought about different ML ideas in that domain. I'm also an avid beta breaker. Finding alternative solutions somehow tickles my hacker-brain. So my first idea was intended beta vs. alternatives but I was only thinking in 2d and images :) Another idea I've played around with is more objective grading but labeling is kind of hard especially with gyms rotating routes so I thought about using Kilterboatd data and transfer learning it or maybe unsupervised approaches.

Congrats on a cool idea and most importantly the execution.

justinnhli
0 replies
1d1h

I have had a similar idea for the past few years: less an AI model that takes in a photo and generates a video, more a movement planning tool given a 3D model of the wall.

Echoing some of the other comments, I think it would be more interesting to not just see a single avatar climbing the problem, but to see many possible approaches to the same problem, even for climbers with the same body type. This way, even skilled climbers that are stuck on a problem can consider alternate beta/techniques that they may find easier to execute.

i_am_a_squirrel
0 replies
23h33m

If anyone would actually use this (I don't think they would, because the puzzle, and talking about beta is half the fun of climbing)...

...then this would be very valuable data.

Eventually you could train AI to generate routes, and then you could fire all of your master setters!

Climb setters should unionize and copyright their art before it's too late (like for painters and software developers and musicians)

greenie_beans
0 replies
1d2h

cool technical project, but as a climber, part of the fun is solving the problem with the help of my own brain and the brains of the other climbers around me.

giantg2
0 replies
19h55m

Probably going to show me the best route to the couch.

gazelle21
0 replies
1d1h

Wow, this is great!

I do BJJ and wonder how it could AI models could help. I find watching tape to help, not only myself but everyone who I know has recorded their matches has found it helpful.

ericmcer
0 replies
1h36m

This has always been a dream, because there is a "perfect" way for your body to climb a route but it would be insanely complex to calculate, especially when factoring in every tiny physiological aspect like ankle strength and hamstring flexibility.

This is a really cool step though.

Maybe a more useful variant with the current tech would be isolating a single move and seeing every variation the AI can think up.

copypasterepeat
0 replies
2h23m

Very cool idea but I wonder how well this can possibly work in practice. So many bouldering problems hinge on small things, such as how a little crimp hold is oriented or how deep the groove in it is. It can be hard to determine this even in person until you get up close or try to grab it.

Additionally, this model seems to take into account the body type but not other relevant factors, such as climber's strength and flexibility/mobility. Even in the demo, that suggested move of putting the right foot high and pushing off the jug up and to the right could not be done by a lot of new climbers. This parallels what I've personally seen when watching others climb. For example, some climbers are exceptionally strong and can basically campus their way up a route. Watching their beta is irrelevant to me. I don't have the same strength to pull those moves.

bryanrasmussen
0 replies
11h58m

I guess it's just how my mind works but I thought this will make a great part of some stories I'm working on, only adapted for cat burglars to figure out the optimal route on a building beforehand.

Maybe also good thing for the next Mission Impossible movie if they ever make one.

blobbers
0 replies
1d1h

Fun! How does it know what route you're climbing, if you can't climb it? >_> Sorry if that's an obvious question.

Does it just tell you how it would climb things you can already climb?

alexcaza
0 replies
1d

Cool idea! My friends and I have chatted about building something similar, though with a coaching focus. Specifically for use with spray walls/kilterboards/etc.

It could be interesting to combine something like this with an analysis tool. Analyse a climbers attempts or successes and compare it to the beta figured out by the model. Then offer tips on body positioning or technique based on your weight/height/strength.

aconz2
0 replies
1d2h

Very interesting, do wish we could play with it since it is a show hn, maybe repost when it is open sourced/usable. Looking forward to reading about how it works. How does it know which holds are on? How does it know what the top is? Are those inputs or inferred or is it just end to end completing likely body positions one after the other?

DanBC
0 replies
1d2h

Be interesting to see if climbers like it - I'd love to see Wide Boys testing it out.