This is an old revision of the document!
Smart Glove for interpreting ASL (Sign Language) alphabet
Author: Andrei-Alexandru Georgescu
Group: 332-CD
Introduction
The intent is to make a hand-worn device that uses several sensors to internally store the position of each finger, as well as orientation measured with a gyroscope, and then using this data to interpret, based on set of logistic regressions, a corresponding letter according to the ASL standard alphabet.
The ML based interpretation of the gestures makes the device highly personalizable, as it could be trained for a specific individual's range of motion and manner of making the gestures.
Its primary and intended purpose is to interpret gestures, but it can also be used, once trained, in an educational fashion as it uses an LED to signal if the gesture is recognized.
General description
\\An important part of the project's logic is based on the 6 IMUs used. One is the “Reference Sensor” and it is mounted on the wrist such that it doesn't much experience the rotational movements of the hand. The other 5 are each mounted on a fingertip and all funnel their outputs towards the controller through an I²C multiplexer. The microcontroller is going to in turn request data from each of the sensors over the I²C line, and use the values to determine the position of each fingertip relative to the wrist.
The intended function of the project has 2 possible running modes, which determine how the positional data is used:
- Learning Mode - In Learning Mode, the expectation is that the controller is connected to a companion computer that deals with the mathematical computation of the regression weights. It is also in this mode that the connected button is intended to be used, serving as a snapshot “trigger”, recording the data at the pressing moment and sending it over USART to the companion for processing.
- Interpreting Mode - In Interpreting Mode, the expectation is that the controller has been pre-loaded with the necessary weights from a prior learning session. Here the positional data is used to sequentially calculate the “probability” of the gesture being associated to known symbol, interpreting the gesture to be the symbol with the highest “probability”, but only if it passes a minimum threshold of 80%. While a value is interpreted as known, a green led is active and internally a clock is started. If the guessed position is still the same after a half a seconds has passed, the interpretation is considered valid and written over I²C to the LCD. In Interpreting Mode, the connected button is instead used to clear the LCD screen of prior written content.
Hardware Design
