How did we not have this 10-15 years ago? It just feels like not enough people work on, or care enough about assistive tech till it impacts them personally.
I can imagine this helping anyone that has struggled with computer-related overuse injuries. I had a bad case of tendinitis that made using a mouse or trackpad very painful. I would have loved to have this kind of device as another option.
I'm really wondering why nobody's selling a foot operated mouse yet, it would be pretty practical even for normal people when typing with both hands.
I did experiment with using a trackball taped to the floor. It's hard to get precise positioning moving your foot. I think using the tongue along the roof of your mouth would be more precise and less fatiguing. What was more useful was a set of 9 foot switches that could be programmed to send arbitrary keystrokes. I could off load from my hands the most common keypresses I do all day, for example: pageup, pagedown, tab, enter, backspace, mouse click, passwords... along with speech recognition software you can get at least non-coding tasks done without using your hands too much.
It's hard to get precise positioning moving your foot.
How long did you do your experiment? Obviously most of us are not typically used to point and move precisely with our feet. So to be efficient we would need to train for months but that doesn't mean we can't be precise with our feet if we were to dedicate enough time for it.
I once saw a documentary about a german woman who was born without arms. She would do everything with her feet, writing, drawing, smoking a cigarette or mounting a horse and holding the reins with one foot. We are much more adaptive than we think we are but we don't realize it when we have the luxury to not needing it.
I went to grad school with a guy w/o arms. Sure, he could write with his feet. It was extraordinarily hard and difficult. Nothing changes the physical facts - we neither have the nerves or fine control in our legs/feet that are present in our hands.
We also build fine motor control in very young childhood; children can't color within the lines (for example) because the nerves just aren't there yet. I would expect someone born without arms to have better fine motor control of their legs than an adult that tries to develop the ability. And it takes them years - 7 or 8 years to get pretty decent control. Even if I had the neuroplasticity of a young child (I surely don't) I wouldn't want to spend years trying to develop that control.
I face this every day on the piano. You distribute motion between fingers, wrist, forearm, elbow, and shoulder based on the amount of fine control needed. Sure, you can use your shoulder to trill, but it is always going to be very slow and clumsy. finges are best, but you face fatigue, so you usually use wrist rotation to get muscles with more endurance suppling the gross motions, and then fingers for the fine control of dynamics. You can't change this physical reality, at the most you can compensate (e.g. someone with a fused wrist would have to find alternatives).
Can you link the hardware you're using? I have one foot pedal for push-to-talk and I think more could be useful, but I don't want to tie up 9 USB ports.
Yes, push to toggle microphone is another great use for these! I was using the Kinesis Programmable Foot Switch 26 years ago with ps/2 connectors (later they switched to USB). You would get 3 foot switches in each unit and chain them together along with your keyboard so they only used one host port. The most similar thing I see now would be https://www.amazon.com/X-keys-Foot-Pedal-Playback-Control/dp... It's frustrating that I can't reprogram my old foot switches since the manufacturer stopped supporting them. If buying today, I'd look for something based on open source QMK firmware like my keyboard uses. This project looks cool: https://github.com/christrotter/qmk_firmware/tree/arcboard-s...
Best I personally managed was push-to-talk via a foot pedal, to help with mic discipline in cooperative games without reducing my reaction time.
I'm not sure my feet have the accuracy to work as a mouse.
I think assistive devices do better as a class when some fraction of them are also usable or even attractive to able people as well. It gets us thinking about other modes of interaction and of course it brings the unit price for those items down out of the stratosphere.
Foot mouse will unfortunately stay niche, but when I saw the device in this article, I thought, "Why not for everybody?"
I'm not sure my feet have the accuracy to work as a mouse.
You should be thinking more of like a trackpoint for your feet/foot.
Foot mouse will unfortunately stay niche, but when I saw the device in this article, I thought, "Why not for everybody?"
I can feel my tongue having a muscle ache just by looking at this.
I don't get the tongue idea either, any kind of device you'd put in would make it hard to talk and who wants to eat a battery? Plus the movement area is pretty tiny.
Accurate foot movements are just training I bet, just like with anything else. Arm amputees have shown again and again that it's just a matter of practice. Plus, were any of us really that proficient with a mouse when we first started? We take it for granted but we've been honing that for decades.
trackpoint
Looking at how that one works, it's just four load cells. That would be exceedingly simple to reproduce with a microcontroller that supports HID. I might actually try to make one.
You can try some rapid prototyping with microbit and this library below that has ble hid support
Demo showing cursor accuracy and voice control/talking when the MouthPad is in the mouth:
* Only tongue tracking: https://www.youtube.com/watch?v=gH-z1KnIthM&t=1s * Head tracking mode (& wink to Neuralink demos :P) : https://youtu.be/6TTIsE4GMEU?si=TbS07BdjUaZDSwx4
Regarding fatigue: We constantly use our tongues for talking and eating without feeling tired and so far, current users haven’t reported tongue fatigue issues.
FWIW, I've considered mapping a midi pedal board to meta keys and macros before. That alone could be useful - you'd go from a typist to something like an organist.
Lift your foot up and try to balance in your seat and see how it feels. Try using a standing desk with it. Put a pen between your toes and try to draw a square and feel major muscles in your upper thighs struggling to make these fine control motions. Put on high heels and try to do anything that isn't gross movement. Sit in a train/plane/car and think of trying to use a laptop with a foot controller. Heck, put your mouse on the ground right now, turn pointer speed down as slow as possible to accommodate the gross movements of your legs, and try to use it.
Assistive technology for the disabled is the higher use for mouth-input, but it'd be great to have a simple bluetooth mouthguard with media playback controls (play, pause, next previous) for action sports.
Could you provide some examples of this?
While snowboarding or skiing, it is difficult to control music playback because bulky gloves are being worn. Cycling might be another sport where you want to control music without taking your hands off the handlebars. A single button (like tapping an earbud) that could be gently bit down on could control song playback. Prior art for this type of device would be something like this: https://www.airturn.com/products/airturn-bite-switch-remote-...
It's very strange to me to listen to music while snowboarding or cycling. Especially while cycling you need a lot of situational awareness of what's going on around you. I think it's very dangerous not to be able to hear cars coming up from behind.
in other parts of the world there are dedicated bike trails in nature where there is no danger from cars
Some headphones (eg airpods) have pass-through audio to allow situational awareness. There are also trails in my area that allow bikes but that are very remote and are unlikely to have much bike and pedestrian traffic. It wouldn’t be unsafe to have headphones in there.
You’re absolutely right, but a truly startling amount of people on the trails give absolutely zero care about other trail users and are just reckless antisocial parasites.
Even without the need for situational awareness, it feels weird to me to feel the need to constantly hear music.
When I do a bike ride, for me, the silence is a feature.
Replying to the reply- even in dedicated car-free recreation areas, there are other people doing the same thing as you but going faster.
voice control is pretty convenient for those environments
Yup, I can see it now:
- Single tap on the roof of your mouth: play/pause.
- Tap on the inside of your upper left/right teeth: previous/next.
- Drag your tongue along the inside of your teeth: playback scroll.
- Drag your tongue on the roof of your mouth from front to back and back to front: volume up/down.
It seems very intuitive, and like it would be much more precise than any of the finger gestures we use today. My only concerns would be comfort, mistaken inputs, and impact on speech. But then again, some people wear dental jewelry and it doesn't seem so bad. :)
I just want this for extra control options in gaming. I bet I could get good at aiming with my tongue. Shorter distance brain to tongue than brain to hands too? Faster response time? Tongue control is going to dominate in esports ;) RGB mouth guards incoming.
What is the average gamer's tongue stamina or tongue dexterity like? How long do you have to practice to improve for this to actually be useful?
I feel like it's probably pretty hard to get good with this.
Some people have much stronger and more nimble tongues than others due to physical differences (e.g., tongue size, the length of the tongue frenulum, etc.). But this is the case for the hands as well, which is why people have different input preferences and different gameplay preferences.
Personally, I think I would prefer a mouth controller over a traditional controller. I am very prone to tendinitis in my hands, plus I have small hands and short fingers, and can't play twitchy games well because of it.
The picture of a tongue mouse from article (maybe it's a render I don't know) seems to have maybe 16 dpi so it is probably fits for secondary mouse just for rotating the world, but for tiny pixel-hunting I would like to have something with a multiplicator.
There is a streamer named RockyNoHands[0] He plays Warzone entirely with his mouth. His controller has tubes that he blows and sucks to control the character.
It is really impressive watching him live on such a competitive game.
I always thought this would be the best way for anyone to operate a mouse. Not just the disabled.
Better than eye-tracking? It's all though-experiment land for me, but when I think about the precision and speed with which I can move and "click" something, nothing beats the eyes.
Maybe. I just think of all the other things that my eyes want to be doing. I figure that would interfere too much in the usefulness. But i agree. We should just test out all the wacky ideas and see what works best.
Great point, maybe that is the weakness in eye tracking. What about a small switch in the mouth you press with your talk, similar to a push-to-talk that activates the eye tracking. When you aren't pressing, you can look around and do other things with your eyes without affecting the computer?
Wish I had the bandwidth to take this on right now, because it sounds super fun and something I could be very passionate about.
Aren't there already products that use eye tracking to locate pointer clicks?
Yes, Tobii sells cheap cameras and SDKs that have done exactly this for years now, to the point that several random games have Eye Tracking integration for no reason.
Something like Eye Gaze perhaps
I would use this before any voice assistants our neurolinks.
We're adding silent speech input and bone conduction audio feedback to the next version of the MouthPad^.
Also consider perforations in the inert parts of the MouthPad. Especially considering its intended users, it's a choking hazard and sooner or later somebody will get one of these stuck in their throat. You'd rather it not kill them when that happens.
Forget code smells. "This code tastes bad."
This was written by someone with bad taste.
This definitely left a sour taste in my mouth.
Fascinating. I started to instinctively "scroll" with my tongue while reading the article - felt really intuitive.
Tickles a bit. But the pad might help.
Exactly! I also started to pretend scroll with my tongue and it actually felt quite intuitive. It is a very clever solution! But they will probably need a well written driver which rejects accidental touches since at any point in time my tongue touches the roof of my mouth.
What an incredible piece of technology. I'm very impressed.
Thank you! We're just getting started :)
Why am I being downvoted for saying this?
People on HN complain how a substantial amount of the comments are negative. I give my honest opinion, as a doctor, knowing full well how hard it is to not only clear the hurdles to get to where they are at, but I also appreciate that it is a worthwhile endeavor. In an era where half of the people in tech are wasting their lives optimizing SEO and to sell more ads for useless crap, these people are out there making tangible improvements in people's lives.
Oh hey, I remember this, signed up to beta test a year or so ago and 6 months later got an email that said I was rejected for being outside of California. I think at the time I found it weird that it took them 6 months.
Hi Logan, thanks for your comment, and apologies for the delay in our response. We're a small company with limited resources, so we've prioritized waitlist applications from those who really need our solution and are close to us. We're working on scaling our operations and hope to serve everyone in the US soon. Stay tuned!
It's fine. I'm not where the money and news coverage is. Go help those quadriplegics.
The recently launched Vision Pro and its eye tracking also offers a relatively niche and very accessible input method. It is already being used in this context:
“I was having trouble getting around that day,” he said, so he took the day off and did work for his consulting agency, Equal Accessibility, from bed, wearing the Vision Pro, surrounded by screens he controlled with his eyes and a series of custom mouth-sounds that triggered selections. “As I get older, and this happens more often for me,” he said, “I envision myself working virtually with the AVP even more.”
https://nymag.com/intelligencer/article/apple-vision-pro-dis...
Eye tracking is coming to iPhone and iPad too. I’m sure it won’t work quite as well as the Vision Pro’s hardware, but for someone who can’t take a headset on and off by themselves it’s nice to have options.
https://9to5mac.com/2024/06/13/turn-on-iphone-eye-tracking-i...
People living with such extreme disabilities are so much stronger than me.
I put a lot of pride in my independence. If I needed someone else to feed, clothe, and bathe me, I think I'd rather die. I'd feel like I was no longer living, but merely surviving.
Probably most people would think like that, but it all depends on what you can still do. If you become a good writer or even if you stick to software, but make valuable contributions, that improve the world a bit, or become a great orator, or give people good counsel, or any other great activity, you might find, that that is a goal worth still continuing to live.
Really cool tech!
Does anyone here know how the sensor technology actually works? Is it capacitive touch, a low voltage electrical connection, or something else? Also, how would something like this differentiate mouse use from speech?
I think in the early Iron-man comic books, it's said that he had some switches controlled by his tongue in the helmet
Is there a site where such tools are listed? Some of these may help people without disability too.
Wish brolylegs lived to see this :( He was using regular controllers
On a side note MS has some neat stuff for disabled gamers, the Adaptive Controller and they worked with Byowave on the upcoming Proteus Controller
https://www.xbox.com/en-US/accessories/controllers/xbox-adap...
https://www.byowave.com/product/proteus-controller-early-bir...
Interesting that he built this after getting closely familiar with Neuralink
Kind of tangential, but I don't understand why keyboard/mouse layers haven't caught on more. I kickstarted the UHK[1] years ago, and haven't had a mouse ever since. It does take a little bit of getting used to, but controlling the mouse with a keyboard is really amazing.
[1] Ultimate Hacking Keyboard
Nice to get this before The Jackpot arrives!
Can we talk about the absolute perfection of the name? MouthPad sounds like someone trying to say mouse pad with a MouthPad in their mouth. It's beautiful.
Not exactly related to disabled people, but a though crossed my mind on how cluttered F1 pilot wheels are with buttons, I imagined some F1 Engineer already thinking on how feature dense these pads can get and how to integrate them into races.
Not related, I know.
wholesome content :') x.com/augmentaltech/status/1798701297748902349
Sounds like a pretty good solution for those who can't wait for neural interfaces to become available (and affordable).
where are you building them? How much does it cost to build?
I read somewhere about a similar tech a long long time ago. It must have been this one:
https://www.researchgate.net/publication/224648269_Tongue_dr...
Exactly. Its unfortunate that this is the way the world works...but we hope that the MouthPad^ can change that tired script! Universal design benefits us all, not just those at the extremes -- we should all care. See the history of curb cuts on our sidewalks as a small example.
For a short time in DC (District of Columbia, USA) government, I worked with Age-Friendly DC. They were all about universal design, obviously. I learned a ton - like how curb cuts are good for folks using canes, walkers, and wheelchairs, and those who push things like strollers. However, the Age-Friendly DC staff could have done much more to work with non-aged policy advocates and organizations, to show them the benefits of things like curb cuts.
Personally, I'm using a lot of what I learned from the Age-Friendly folks; my 83yo mom has moved in and the learning curve, while steep, got a head start with my government experience.
As someone who's not a quadriplegic, I want one of these.
I went to grad school for architecture at the end of the last century.
Tim was a classmate. He was and remains paralyzed from the neck down. At the computer Tim manipulated a trackball with a tool — rod connected to a flat mouth piece.
This is different, but I am not sure it is better. The trackball and stick were mostly mechanical, reliable, and straightforward to repair.
No special software. No need to charge. Tim’s tools were as reliable as a mouse. Not much of a choking hazard because of the stick either.
I’m not saying the mouth-pad is a toaster fridge. Only that it has technical complexity rather than mechanical simplicity.
I would think fewer moving parts is less complex. Considering it’s basically a trackpad, I doubt the hardware and software are bleeding edge.
It’s a radio and computer that goes inside a person’s mouth. That person can’t use their hands. How do they take it out and put it in?
Tim had a red solo cup size container next to his keyboard — he also typed with the tool in his mouth. [1] When he wanted to talk to someone, he stuck the stick in the cup and stood the tool up where he could pick it up again.
[1] I hadn’t remembered that because it’s been 25 years. Tim could also out sketch most people with a pencil in his mouth as well. He had a bachelor in fine art before the motorcycle wreck.
I work in a lab at a university where I'm currently working on building assistive music controls for disabled musicians. From what I've seen in my short time here is it seems like a lot of these kinds of projects don't often translate from the research space into viable businesses. Also, due to the highly variable physical and neurological characteristics, often assistive devices will need specific customization, tuning, and support. So these things also tend to be expensive.
This is exactly correct (I worked in AAC and BCI for years, and still collaborate with the group that I used to work closely with). To elaborate a little bit on your excelelnt answer: In addition to being enormously expensive and requiring heavy customization and personalization, assistive technology has other issues that go well beyond the core technical functionality. It also often requires significant up-front investment in configuration and training for the user, which needs to be done by a professional in the field, and then --- crucially --- also requires ongoing maintenance and re-training by skilled practitioners, which means there are ongoing operational expenses.
One of the reasons for this last bit is that many users of assistive technologies have very dynamic situations. For example, if they have a progressive condition like ALS, their mobility and other capabilities and needs will change over time; even if the user doesn't have a progressive condition, their living situation very likely will change over time, which may involve needing to re-work their setup. This might involve something as major as moving from one care facility to another, or having to adjust to the presence of a new pump or something that causes EM interference with your device.
Even for somebody with a very static medical condition and very stable living situation, their computing setup won't be static, so will need ongoing maintenance; or, alternatively, their computing setup will become static if they end up dependent on a particular piece of assistive technology, which can lead to a whole other set of issues (which is how you get people running ancient versions of Windows for years and years and years, because it's what their eye-tracker works on, and the vendor is long out of business and so never got it working on Windows 10).
The other piece to the puzzle is that, speaking as a gross generalization, users of assistive technologies tend not to be wealthy people (many speech and motor disabilities make it challenging to work). So, you've got an expensive product, often involving ongoing operational costs, with zero economy of scale, for a small and relatively poor customer base; this all adds up to what may fairly be described as "a very challenging business". Adding another layer of complexity, the economics of commercial assistive tech in the US have a lot to do with insurance Medicaid rules; the only reason there is any commercial activity at all in assistive technology and AAC is because under some circumstances Medicaid will pay for a device, within certain very rigid and complicated boundaries. So now your business also is going to have to juggle that problem in order to keep the lights on, which will add another layer of constraint to what is already a complicated situation.
On the other hand, when the right technology is paired with the right user and the right support, the impact on the quality of life for the user and their family can be truly enormous. Which is what keeps people working in the space...
At least one company I know of (though this was 15 years ago) had ridiculously high prices for their products. But they had extensive after-sales support, and the warranty was such that you could basically run over the unit with a car, and they'd still repair / replace it under warranty for free.
we did, there were at least 3 kickstarters (of which i ran one) but not enough people were interested.
that's so cool that you tried. how much did you spend on advertising for your Kickstarter? I totally would have kicked in if I'd heard about it.
We tried about 9 years ago :) we even open sourced the solution. At the time we felt there was not a big enough market so we decided to make it open for anyone to build upon our research.
http://pallette.io/instructions.html
Seems to be the case for so many assistive technologies. It is even more frustrating considering how high impact some of these technologies are.
Do you have any insights on how this space could grow other than State Funding or Insurance policy changes?
I suspect it’s a couple things. First, the skill gap - maybe someone begins needing assistive tech and has no idea how to start making. Second, perspective - as an able-bodied techy, I lack the understanding of the problem to build the right thing. Third, communication - can the one in need describe their need? This really requires reflection, and maybe skill. And lastly: money - how many of is can afford to work for months or years on a project that doesn’t (and likely won’t) pay?
MITs MO is to try to create a breakthrough tech and rapidly spin off startups from the tech. But I think the main barrier is that to operationalize hardware requires massive capital investment upfront for testing and then regulatory approval, this is mostly not possible for VC. So the students/profs who make the breakthrough do the next best thing: Patent and then sell it to a company that has the capital to do it.
But someone can correct me if I am wrong about this.
Typing with eye movement. https://www.optikey.org/ Typing with head pointer. https://www.alimed.com/adjustable-head-pointer.html One handed typing. Various keyboard layouts. https://en.wikipedia.org/wiki/Dvorak_keyboard_layout#Variant... Speed recognition, Text to Speech, Windows accessibility features.
Products do get worked on and have been released. But yes, more work needs to be done and this product is a welcome addition.