Overview/General Discussion

[ATMakersBill]

Here's a place to discuss what we need, start discussing plans, etc. Anyone can post to the discussions if they have an account. Committing to the code/files section needs teem access - ask Bill Binko.

[nelsonii]

Testing comments to confirm access.

[ImproveAbility]

Here is the problem statement and scope for this project:

I have a number of clients with advanced ALS - no physical movement except eye movement and atypical eyelid movement (cannot do a quick blink).
These clients need a way to trigger an alarm when the eye gaze communication device is not behaving.
So, I am looking for a way to trigger an alarm that only uses eye abilities. If this makes a contact closure, I would be able to make this all work. We just need the trigger.
The system needs to be 100% separate from his eye gaze system for reliability.
I am saying eye gaze, but it has been noted that maybe we could use an eye open/close trigger?

ImproveAbility is willing to purchase parts to create prototypes as well as test the device with our clients to find a solution.

[ATMakersBill]

Let's set some boundaries for the first generation of this.

First, I'd like to focus on the input side of this: that is extracting the user's intention, not worrying about what we are going to activate. If we have the trigger in a Raspberry Pi, we can toggle a pin w/a relay or SSR and get an AT switch style output... OR we can play a sound... OR we can send a SMS message... but we shouldn't focus on what the output is for the first prototype.

Second, I've seen in the email comments about different scenarios like low-light situations and folks who can't open their eyes without help. While we want to keep those in mind, I think our first solution should focus on a "known solved" situation: using eye-detection (not eye gaze or directional input - just eye detection) to detect intentional slow open and closes (with good feedback to the end-user) to trigger an alert. One nice feature of this is that it doesn't require careful positioning of the device or the user to be able to look in a particular direction: just that the camera can see the eyes well enough to detect them (which is commonly available in open-source computer vision systems).

I believe this is something that will help a great many people (far beyond those w/ALS) given that many of ATMakers recipients are eye-gaze users w/CP, SMA, etc. or they are on an inflated trach at night and have no way to get attention.

It's true that this may not help those who can't control eye closure (and opening), and I'm all for finding another solution for them (EMG is probably a good one, and not as cost-prohibitive as it seems), but I think we should start with this as a first-pass (or Minimal Viable Product - MVP) and expand from there. One pro for this approach is that low-light should actually be an easy modification w/a camera replacement but it's not necessary for many of the users -- it's a great addition.

It's possible that we can use the same approach with a higher resolution camera and machine learning to detect eye-motion under lids, but that seems like a goal that shouldn't be in phase 1.

Thoughts?

[ImproveAbility]

I 100% agree with this thought process Bill!

[LisieRae]

Have you seen https://github.com/charliegerard/gaze-detection?

[ATMakersBill]

@ImproveAbility On the kickoff meeting - would you like me to host it on StreamYard? It allows up to 10 participants and streams to Facebook/YouTube natively. That would give the rest of ATMakers and Assistive Technology FB groups a chance to watch & provide input.

[ImproveAbility]

Yes, I think that would be great! Just waiting for everyone to respond with their availability - so far looking like first week of May.

[ATMakersBill]

Here's a thought for hardware for the first prototype. It's a braincraft hat + RPi4.
https://www.youtube.com/watch?v=zNfgfn1_dyA

Noe & Pedro made a case for housing them plus a camera and speaker.

I have a braincraft hat + camera + Pi4 here and can use it for prototyping (I haven't printed the housing). Obviously, raspberry pi's of all stripes are hard to come by with the chip shortage, but I think within the ATMakers community (and my friends @ adafruit), I can come with a few sets of these.

I don't think that TensorFlow Light (or any type of machine learning) is needed for this project at all: simple OpenCV running on the Pi should be completely capable of doing eye detection. I just like this hardware because it has the preview screen, the 5 position input joystick, feedback LEDs, audio output, and expansion for other sensors via STEMMA/QWIIC connectors.

[nelsonii]

As so often seems to happen, the Vizy was reviewed yesterday.

https://hackaday.com/2022/04/11/review-vizy-linux-powered-ai-camera/
https://vizycam.com/
$300.

No screen or joystick, which I think are great things to have for the setup/adjustment process.

[ImproveAbility]

I like the idea of the video for caregiver positioning as well as the audio output that could be used for the client to perform the needed action - this could give feedback to them to help with timing. Uugg - so much I do not know about this world, but I am loving seeing all of these things! I am waiting on one more feedback on availability to schedule first meeting.

[ImproveAbility]

So Bill, you are thinking that this could be used to detect eyes open/eyes closed?

[ImproveAbility]

Project Activate Feedback-Raspberry Pi question

I wanted to respond to the concept that Android Project Activate "does this already." I have been playing with it - having an old cell phone mounted in my office for a week now. It is a great option for what it is. But, twice now over the last week, the app quit. The phone stayed on, but when I went to the device, it was on the home screen instead of the project activate app. This is why having a dedicated device is so important for this problem. The users for this device need it to ALWAYS be available - it cannot go to sleep or shut down at any time.

This leads me to a question about Raspberry Pi - I have never used one myself, so I am a bit ignorant about them. I know it is a computer, so how reliable are they? Do they need to be shut down on a regular basis or are they ok with being on for weeks or months at a time?

[NachumLehman]

I think we really need to do some comprehensive testing on the Project Activate app because it's available and does most of what we're looking for. There are some nifty features built in and some even better ones in the works. I was more involved during the development until I got insulted when it couldn't detect my mouth opening because of my beard:) Rebooting your primary eye gaze device from that app is really cool and there are a number of people across the country who have used it for that purpose. I will do some preliminary testing on my end and will hopefully have some results by the time we meet.

[ImproveAbility]

That does sound like a great idea. I will tell you I had the app running on a device in my office. I checked it a couple times a day. After about 3 days of running, it just stopped the app without warning - went to the home screen. So, I am concerned that it might not always be available. I would not call this "extensive testing" but it is information for the group to have.

[JayToltTech]

Nelson, you've been more successful than me. My longest running Pi was a "Magic Mirror" project for my kitchen like this:

But it would only stay up for a few months at a time and I got tired of my wife asking me to "restart the mirror" after a while and took it down. Generally the reliability problem with Pi's is the SDCard getting corrupted and at least a few years ago the "read only filesystem ram overlay" approach wasn't stable enough to last more than a few months at a time.

I spent a while fiddling with USB bootable external flash drives to get a reliable Pi server for a few months after the Pi 4 came out with beta support for USB drive boot, but the boot bios was still experimental and I still had reliability problems back a year ago.

If you are shooting for "always available", that brings in a new level of engineering & design. I'm not sure facial recognition libraries can meet this bar today, because the general definition is "no dynamic memory allocation/deallocation", as this is generally where "things go wrong over sufficient time". It also means 'no consumer OS' which includes Linux along with Android, iOS, Windows, etc. as they all dynamically start and stop things which is required to be able to do User Interfaces more complicated than say your standard/simple looking UIs like you see on your car dashboard or your medical device (e.g. wheelchair display, ventilator display).

A 'cheat code' for this is to take an unreliable system and "whack the unreliability out of it", at least down to 0.1% downtime (e.g. unavailable 10 minutes per week). The three ways to do this are:

• Add a watchdog (e.g. a second thing) that monitors the first thing and then whack it if the first thing isn't responding. My Chevy car dashboard does this. It turns on 'most of the time' and then about once a month I turn on my car and my dashboard fails to boot and stays black. After about 5 minutes, a second computer 'watchdog' notices that my car dashboard isn't working and power cycles the whole thing. Nasty, and a slow watchdog too. Hate it.

• Proactively restart to clear out the memory leaks and other cruft. For a long time, to get servers up to 99.9% reliability (down just 10 minutes / week), they would be set to automatically restart themselves every Sunday morning at 3 AM. As long as you weren't trying to browse that website from Sunday 3:00 AM to 3:10 AM, it was pretty much always up!

• Run redundant systems (e.g. multiple servers or multiple eye switches), have them observe each other, and have them 'fail over' to a working system when one goes down. This is one of the primary techniques used today, but for this case it would mean multiple switches that cross-checked each other.

[JayToltTech]

@ImproveAbility it might be worthwhile to repeat your experiment and try and leverage 'Android Kiosk Mode'. This both limits the phone to 'just one thing' and it should auto-restart the app if it crashes. See:

https://snow.dog/blog/kiosk-mode-android

Reading through that article, it looks like the app may need to be modified to support running as a kiosk app, so you may not be able to do this with the app that's published to the Google Play Store.

I've pointed the people at Google I know who work on Project Activate to this discussion and I've asked them if Kiosk Mode is something they have considered.

• Jay

[JayToltTech]

Looks like Kiosk doesn't include auto-restart, but it's possible to add this to the app as well:

https://ssaurel.medium.com/how-to-auto-restart-an-android-application-after-a-crash-or-a-force-close-error-1a361677c0ce

[ATMakersBill]

Pi’s are very reliable and yes, they can run for months and years. If the device remains simple and not connected to the internet then there’s no reason for updates that cause issues etc.
OUR code will have to be careful. It’s pretty easy to leak memory and crash so we’ll have to be careful and probably detect “sleep” times to restart our monitor if our usage is high. But it should be manageable. We should also consider adding a watchdog to ask the user to respond if we aren’t seeing what we expect. But this is all down the line

[ImproveAbility]

Ok, good to know. I would sacrifice features to make it as reliable as possible.

[NachumLehman]

I wanted to throw out some factors/concerns for our first conversation. Although I completely agree with Bill's comment, there may be some factors which take away the simplicity of that solution. I am repeating some of them just to have them all in one place.

1. People like to sleep in low or no light
2. Using just open/close detection will put a bigger cognitive load on the user to manage the alert without getting false positives. Eye gestures seem to be a better balance
3. Eye gestures create tremendous opportunities for much more than just an alert system if necessary.
4. Addressing users who cannot open or close their eyes.
5. Ptosis is something we fight with a lot. Bill's method would probably be better for these situations. Eye gestures are harder to detect with any significant ptosis.
6. Slow blinkers - they really give us a workout when it comes to eye tracking. Using blinking as a confirmation can be challenging with many diagnoses because of a slow or incomplete blink.
7. Cost? I think this is a big factor here because of solutions that are available even now. For instance, we can purchase old Tobii eye tracking devices for around $800-$1000. It is possible to put together an inexpensive (or used) tablet and a Tobii gaming camera for <$400. These options don't address Antoinette's situation where someone's eye tracker goes down but it does address the thousands of people who could potentially benefit from this technology. Mounting would be an additional and more significant expense with a full fledged 10 pound eye tracker. So if the cost to produce/support this tech starts reaching the $250 range, we have to reassess.
8. For the eye tracking users that Antoinette's worried about, we probably can't use a low cost night vision camera which shoots out infrared. That would likely interfere with the primary device's eye tracking just like head mice interfere with eye trackers. I would imagine that a digital night vision camera would be more expensive.

[JayToltTech]

If I may suggest, this list is going to set up a difficult project to accomplish. In particular:

• Working at night with room lights off without providing other visible light source would require the system to use non-visible light sensitive cameras, meaning either IR or UV.

• Working side by side while an eye tracker is operating will mean avoiding conflicting light spectrum. I was able to find an older reference that Tobii eye trackers use 850 nm ± 30 nm spectrum, so you would want to avoid this range.

• Discussing this as an 'alert' or 'call system' implies that you are going to need a system that is high reliability which has the issues discussed above.

It's easy to get caught up in 'perfect is the enemy of the good'. I'd suggest you might want to start with an easier initial scenario and tackle that, learn from it, and build from there.

[ImproveAbility]

Jay, I totally appreciate your feedback. The problem is finding a way for someone to deal with their eye gaze system not working that only has eye gaze to get assistance. So, I appreciate that a simpler problem would be easier to solve, but there are other options for those with alternate switch sites. But, I do not want to squash any creative ideas.

[NachumLehman]

"Project Activate" probably satisfies the initial stage already.

Also, good to mention that Jeffrey Ebin's site https://communicationboard.io/ takes this to another level on smartphones if we're not concerned about lighting, reliability, etc.

[ATMakersBill]

I'd like to reset this to what Antoinette originally asked for and what we need to focus on: she repeated it in her last comment:

The problem is finding a way for someone to deal with their eye gaze system not working that only has eye gaze to get assistance.

This is not about all of the other things that we'd like to do for people with ALS/SMA/TBI/etc. It's about solving a specific problem: when a user cannot access their primary communication device, they need to alert their caregiver. Specifically we're talking about folks who, when their primary device is not available, do not have access to switches or other input devices--they have to use their eyes in some way.

This doesn't cover everything that Project Activate does: and for some folks Project Activate will provide much more functionality. But for others, having that Android device watch their eyes and face may interfere with their primary communication device (which may also track their eyes or use a head mouse, etc.).

Antoinette originally asked for a device that watched for a user to "look in a specific location" long enough to trigger an alert. I asked if that could be replaced by deliberate blink (and wink) patterns that had a low probability of occurring randomly. That's how this started.

These discussions of alternatives is great, but there are few things that should be critical:

1. The solution should not run on the device that hosts their primary communication tool. That is, if it runs on the SGD and the SGD's power fails, their backup fails as well--that may still be valuable, but it's not adequate for this project's goals.
2. The device should ideally be affordably built or purchased with new parts. The chip shortage may cause issues in the near term, but relying on the fact that you can buy bare Tobii 5c sensors from China for $35 shouldn't be necessary. (Yes, I've bought some.) That supply could dry up at any time and we have no idea of the reliability of those parts.
3. Edge cases shouldn't drive the design. Jay's comment of "don't let the perfect be the enemy of the good" was exactly where my mind went in this discussion. If we need alerting at night, we can add an infrared light that is turned on at night. If we need alerting at night without interfering w/the primary eye gaze device, we can detect the IR output from that device and react accordingly (turning our light on if needed or using theirs). And if we can't find solutions to edge cases about working at night without IR, we have still improved the lives of every user during the day and all the users who don't have those issues at night.

I'm also fine with this discuss spawning five separate projects that all try to solve problems in this space. Wouldn't that be great? However, for our initial kickoff, let's try to solve the one Antoinette originally asked.

[NachumLehman]

To clarify, Project Activate is already used by pALS together with their eye tracking devices and there's no interference. Antoinette's main concern was reliability when the eye tracker fails and project activate has crashed. Statistics may determine that's still considered highly reliable - especially considering that most eye tracking software allow the user to mirror their androids on their desktop so they can reboot the device or just restart the project activate app.

[ImproveAbility]

That is a good point - may need to have some clients actually test this theory. What are you using to mirror Android on Desktop - I am familiar with Vysor - is there something else?

[ImproveAbility]

Agree with Bill 100%

[ImproveAbility]

I have an idea that I wanted to record here. Let's think of this as an eye switch - some sort of eye activation of switch closure. We sell Smartbox devices, which have a remote power switch that is switch adapted. I do think that other AAC devices have a port that can be set to turn on/off the device with switch activation. So, the point is there are already ways to restart the computer - the issue is using only eye abilities to create a contact closure. At least that is the way I see it.

[JayBeavers]

You are also welcome to take advantage of Punt37, which is the reboot agent I wrote to interoperate with Project Activate:

https://github.com/teamgleason/punt37

Punt37 is a Windows app that enables cross platform remote rebooting of a computer.

It has been designed to enable a Windows-based speech device to be restarted if something, anything, has gone wrong with the device.

The key scenario is that a person with disabilities that prevents verbal speech is using a Windows tablet with specialty software to speak for them, perhaps with an Eye Tracking camera, and something has gone wrong. Perhaps a driver has crashed or the software they use to speak or emulate a mouse has stopped working.

Punt37 allows some external device -- such as a phone that can recognize facial gestures, a microcontroller with a physical switch, etc -- to send a 'force restart' command to the computer.

[JayBeavers]

FYI, this approach has the advantage that it does an "OS initiated reboot" which gives the operating system the chance to shut down properly, as opposed to just kicking off the power which can cause file system and data file corruption.

[willwade]

Not adding much here.. But felt the need add this:

Along with Punt37 you could pair it with https://github.com/acecentre/Eyecommander pretty simply (you dont need to hack much - you could just use autohotkey to map a space from EyeCommander to Punt37 - Granted its on a Windows device again but you get the point). Or take EyeCommander and use a switch output device such as a BJToyBox to send the switch press to the remote. NB: There are builds for EyeCommander for Linux and MacOS