IR Watchdog : Tobii Dynavox First Impressions (IR Signals)

[nelsonii]

If you this scene from the movie Airplane -- well, that's how it's going right now.

1. The Tobii is all over the place with IR emissions. The usual frequency and decoding code (ex. IRRemoteESP8266, IRLib2, etc) can't lock on to much. Frequency (with the TSAP learner) is "sometimes".

2. IR photodiode (my original solution) is not picking up consistent IR from the Tobii. Based on the fact that I can see some of the "red" with my naked eye, I have a feeling they are using an IR that's lower than the usual 940 range. Are there sensors out there for that? I'll have to check.

3. I built two of the QT Py hats, but having issues running the hat w/o a Pi. Likely brownouts on startup. I'll try external power, or solder in the headers and try an older Pi for power. (I did cut the traces and install the LED drop resistors -- 47 ohm.)

4. I'm putting together an "IR Buddy" which will include the photodiode (along with TSMP and TSOP). Similar to the ATM QT Py hat. Debugging tool which may be helpful for others in the future.

[nelsonii]

Okay, due to the fast and fluctuating values from the Tobii, I've stripped down the code to run faster. By doing this, I'm getting better values from the IR photodiode. (Yet to test the TSMP/TSOP.) So, I'm feeling a bit better now.

Just calling NeoPixel code (ex) is enough to slow it down too much. So, I have to refactor all the light/alarm/tone code. Oh, well. At least I'm getting good values again.

[nelsonii]

[NachumLehman]

Sounds like you're making wonderful progress! As soon as you're comfortable with the results you're getting with that device, I would send your way another device or two to see how they fair. The Eyegaze Edge device has only one IR emitter and is officially the camera that shoots out the least ir at users' eyes. We probably need to test a Gridpad and a newer Tobii too.

[nelsonii]

Yes, we're going to have to try on multiple devices. I'm going to be building a couple of these prototypes so that they can be field tested with multiple brands. That way, I only need to get my hands on devices which aren't currently supported. (Hope that makes sense, I woke up with a headache.)

[nelsonii]

Chris Young
Moderator
Group expert
I'm surprised TSMM 58000 doesn't work. It shouldn't have any filtering whatsoever.
ReplyShare18h
John Jordan
Chris Young Edit: If not TSMM, TSMP says "AC coupled response from 20 kHz to 60 kHz, all data formats". https://www.vishay.com/docs/82485/tsmp58000.pdf

[nelsonii]

John Jordan
After googling TSMP 🙄 I glanced at the Vishay TSMP4138 which employs a bandpass filter suitable for modulated/keyed signals. I would expect the raw IRPD to work better with a diode used for illumination. Can you connect the output to an interrupt pin to capture those pulses better? Do you have an image of the signal's signature?

[nelsonii]

Jay Beavers
A couple of comments on your work:
You probably want to discriminate on the wavelength of the IR signal you're processing. From a previous conversation on a Tobii message board, Tobii sensors used to use the 850 nm IR wavelength for their eye illumination, perhaps they still do. I believe this bandwidth is fully invisible to the human eye. In earlier generation Tobii devices you could not see any lights on the bar when it was active, but sometimes digital cameras (e.g. phones & webcameras) would show the illuminators by shifting the wavelengths (unintentionally I'm sure). Some times it's easier to understand the eye tracker behavior over a Facetime call because of this 🙂
There was another comment that they have added 'near-IR' illuminators to more recent cameras to give people visual feedback on the eye tracker activity, so there may be multiple illuminators at multiple wavelengths perhaps operating on different cycles/frequencies.
Here is where you can get a 850 nm bandpass filter to

[nelsonii]

Great stuff, everyone. I've pasted all these comments into the Github discussion so they don't get lost. 🙂

The newest code is working with the IR photodiode (BPV23FL). The Tobii alters its duration of pulses quite a lot (not "always on"), so the code has to wait for a certain amount of lost before triggering alarm mode. Fortunately, this is pretty basic -- log the "last saw IR" and if it's more than X seconds, go to alarm mode.

It also cycles between left and right emitters/cameras, likely so that it can focus on one eye at a time, and compare. Pretty tricky stuff.

The good news is that, for this, I just need "some kind of IR" -- and the photodiode gives me that.

[nelsonii]

Bill Binko
Admin
Group expert in Science & Tech
I’m not surprised by what you’re seeing. The tobii is using IR for illumination. It’s not trying to send packets at a known frequency.
Its possible that it’s just turning on the leds in-line with a resistor. If there are any frequencies or pulses perhaps it uses PWM to dim the IR LEDs.
I’d look for just the intensity of the light. Perhaps the photodiode can do that if biased correctly or perhaps use a lux sensor that includes ir like this one.
https://www.adafruit.com/product/1980...

[nelsonii]

Jay Beavers
Also, it's my informed guess that eye trackers used a combination of two techniques -- the first is facial feature tracking to determine the position of the eyes in the camera's field of view. This is used to help discriminate false signals (e.g. light bulbs, sunlight, camera frame reflections). This will likely be using 'off the shelf' algorithms and visible light. This would explain why your IR chips turn off during 'eye acquisition' mode. Running these algorithms is quite computationally expensive, so they are only going to be running this when absolutely necessary (e.g. eyes are present, but eye tracking is not aquired).
I suspect there is an 'not currently tracking, are eyes present?' mode in which the device doesn't see eyes and wants to know if it should run the visible light 'eye acquisition' code. This would explain the 2-3 fps steady chirping you are seeing. In this mode, the IR is pulsed on and I suspect they are looking for very bright pairs of point lights, as retinas reflect IR signals really well. Some eye trackers use two sets (L/R) of illuminators arranged in a triangle, so this mode might be looking for one to four sets of triangle point light reflections.
And then there will be a third mode which is 'tracking mode'. Given that Tobii eye trackers are spec'd for 30+ fps, this is likely a rapid pulse at 30 fps or some multiple of 30 fps.

[nelsonii]

My rough experiments match your informed theory. 🙂 The good thing (at least for V1) is that I'm looking for any kind of IR -- don't have to decode.

The fact that the decoders aren't working is actually a plus -- I can put a decoder in, and ignore "normal" IR commands (ex. TV). More of an edge-case / nice to have.

[nelsonii]

Jay Beavers
Ron Nelson That is very insightful to know about the L/R emitter cycling. I didn't know that before!
Using pulsed emitters and differentiating image capture between emitter on/emitter off is going to be a critical key to disambiguating between interference sources (e.g. the sun doesn't blink).
Using two emitters and cycling them independently allows you to get a differential recognition pattern (e.g. Off/L/R/Both) which allows you to detect eyes with a lower BOM cost than say using triangle emitters.
And using two offset emitters that you can read independently would let you use simple triangulation logic to do depth discrimination between camera and reflection targets (e.g. only recognize eyes in a 50 cm to 2 m range which is determined because the point light centers should be within xx pixels and xx+dd pixels apart)...
ReplyShare10m
Ron Nelson
Author

Jay Beavers It's going to be interesting to see how other devices behave/work. I'm guessing some techniques still under patent, everybody does their own thing, etc.
I was trying to get more tech info from the FCC database, but that's mostly for the WiFi stuff -- I've yet to see anything about the IR emitters.
Fortunately, Bridging Voice is loaning me different devices to test with. And this knowledge of "how they do it" will help with the ATMakers camera solution...which I have yet to start on. 🙁