Using Computer Vision to Help the Blind Appreciate Scenery

Cyborg

System architecture for the prototype. Drawn by me.

For EPD’s big 30.007 project, my team and I created a product to “translate” visual scenery into physical sensations the blind can feel. This is to help the visually-impaired appreciate scenery, allowing them to experience a dimension of travel that was previously not possible for them.

To do this, we implemented a scene classification model on a portable single board computer (SBC), then created a wearable camera that streams the scenery to the scene classification model. The identified scenery is then transmitted over to a wearable arm sleeve with an array of vibrator motors, which then sends a pattern of vibrations down the arm of the user. With a library of different patterns, users are able to identify and associate different sceneries.

The machine learning model is an implementation of the open-source Places2 dataset and models by MIT. I have also created a simple demo of this implementation that you can try out for yourself here.

Wearable camera packaged in a pendant. Photo by Steph.

The wearable camera is built using an ESP32-CAM, with a lithium-ion battery attached and a wireless charging coil to keep everything charged. Combined, these components measure just 27 x 40 x 30 mm, and can be easily fit into different pendant designs.

Internal circuitry of the arm sleeve showing array of vibration motors. Photo by Prateeti.

Sensation control circuitry in the arm sleeve. Photo by Steph.

Sleeve when fully sealed up. Photo by Steph.

The sensation patterns are created by an array of 4x7 individually controllable vibration motors, spread across a sleeve worn on the forearm. The unit is controlled by the ESP32-S3-based TinyS3, and uses two 16-channel PCA9685 PWM controller ICs. In the pictured prototype, we used off-the-shelf breakout boards for the PCS9685, and motor control circuits. To reduce the wire mass and bulk of the electronics, we have designed our own PCBs for the PCA9685 as well as the motor circuit. Test samples manufactured at SUTD’s Fab Lab are shown below. The motor circuitry will be manufactured on a flexible PCB, while the 2x PCA9685 and TinyS3 are planned to be integrated into one small PCB.

Shrunken PCA9685 board. Credit to Jarell.

Motor driver circuit, 28 of these in a single sleeve!

This demonstration of our first prototype was praised by the EPD professors and we made improvements before submitting it for the James Dyson Award by July 2023.