Author: Andrei-Alexandru Georgescu
Group: 332-CD

The intent is to make a hand-worn device that uses several sensors to internally store the position of each finger, as well as orientation measured with a gyroscope, and then using this data to interpret, based on set of logistic regressions, a corresponding letter according to the ASL standard alphabet.
The ML based interpretation of the gestures makes the device highly personalizable, as it could be trained for a specific individual's range of motion and manner of making the gestures.
Its primary and intended purpose is to interpret gestures, but it can also be used, once trained, in an educational fashion as it uses an LED to signal if the gesture is recognized.

An important part of the project's logic is based on the 6 IMUs used. One is the “Reference Sensor” and it is mounted on the wrist such that it doesn't much experience the rotational movements of the hand. The other 5 are each mounted on a fingertip and all funnel their outputs towards the controller through an I²C multiplexer. The microcontroller is going to in turn request data from each of the sensors over the I²C line, and use the values to determine the position of each fingertip relative to the wrist.

The intended function of the project has 2 possible running modes, which determine how the positional data is used:

• Learning Mode - In Learning Mode, the expectation is that the controller is connected to a companion computer that deals with the mathematical computation of the regression weights. It is also in this mode that the connected button is intended to be used, serving as a snapshot “trigger”, recording the data at the pressing moment and sending it over USART to the companion for processing.
• Interpreting Mode - In Interpreting Mode, the expectation is that the controller has been pre-loaded with the necessary weights from a prior learning session. Here the positional data is used to sequentially calculate the “probability” of the gesture being associated to known symbol, interpreting the gesture to be the symbol with the highest “probability”, but only if it passes a minimum threshold of 80%. While a value is interpreted as known, a green led is active and internally a clock is started. If the guessed position is still the same after a half a seconds has passed, the interpretation is considered valid and written over I²C to the LCD. In Interpreting Mode, the connected button is instead used to clear the LCD screen of prior written content.

Components:

• ESP-WROOM-32
• TCA9548A I2C Multiplexer
• 1602 LCD + I2C Interface
• MPU 6050
• Button
• Green LED

Conexions: In my project I used the following color coding:

Development Medium: Arduino IDE Libraries:

• Wire.h - For I²C communication.
• LiquidCrystal_I2C.h - For controlling the LCD over I²C.
• math.h - For the mathematical functions required for inference.
• MPU6500_WE.h - For controlling the IMUs used and polling them for data.

Algorithms and Datastructures:

• Multinomial Logistic Regression/Softmax Regression - Using sklearn in Python on the companion / math operation locally on the ESP32
• Wrapper Class over MPU6500
• Wrapper functions over Wire for using TCA9548
• Wrapper functions over Wire and LiquidCrystal_I2C for communicating with the LCD1602

Implemented Functions:
TCA9548 Functions:

• init - to be called during setup, activates the I²C connection

void tca9548_init() {
  Wire.begin(SDA_PIN, SCL_PIN);
}

• tca9548_select - sets the TCA9548 multiplexor to channel i, provided that i is a valid channel (0 to 7)

void tca9548_select(uint8_t i) {
  if (i > 7) return;
  Wire.beginTransmission(0x70);
  Wire.write(1 << i);
  Wire.endTransmission();
}

MPU6500 Functions:

• Constructor - makes the MPU6500 library object and memorizes the TCA9548 channel it is on

MPU6500::MPU6500(uint8_t mux_channel) {
  _mux_channel = mux_channel;
  _mpu = new MPU6500_WE(MPU6500_ADDRESS);
}

• init - to be called by setup, launches the IMU into autocalibration and configures it for maximum precision

void MPU6500::init() {
  tca9548_select(_mux_channel);
  _mpu->init();
  _mpu->autoOffsets();
  _mpu->enableGyrDLPF();
  _mpu->setGyrDLPF(MPU6500_DLPF_6);
  _mpu->setGyrRange(MPU6500_GYRO_RANGE_250);
  _mpu->enableAccDLPF(true);
  _mpu->setAccDLPF(MPU6500_DLPF_6);
  _mpu->setAccRange(MPU6500_ACC_RANGE_2G);
}

• get_data - to be called inside the loops, makes sure the communication is over the correct TCA9548 channel and fetches current angular data

xyzFloat MPU6500::get_data() {
  tca9548_select(_mux_channel);
  return _mpu->getAngles();
}

LCD1602 Functions:

• lcd1602_init - to be called during setup, initializes the LCD and turns on the backlight

void lcd1602_init() {
  lcd.init();
  lcd.backlight();
  lcd.clear();
  lcd.setCursor(0, 0);
  delay(25);
}

• lcd1602_write_char - writes a character at the specified position, provided that position is valid

void lcd1602_write_char(uint8_t pos_col, uint8_t pos_row, char c) {
  if (pos_col >= LCD_COLS || pos_row >= LCD_ROWS) {
    return;
  } 
  lcd.setCursor(pos_col, pos_row);
  lcd.print(c);
}

• lcd1602_write_char - writes a character at the current cursor position

void lcd1602_write_char(char c) {
  lcd.write(c);
}

• lcd1602_write_string - writes a string at the specified positon

void lcd1602_write_string(uint8_t pos_col, uint8_t pos_row, const char* s) {
  lcd.setCursor(pos_col, pos_row);
  lcd.print(s);
}

• lcd1602_set_first_row - moves the cursor to point at the begining of the first row

void lcd1602_set_first_row() {
  lcd.setCursor(0, 0);
}

• lcd1602_set_second_row - moves the cursor to point at the begining of the second row

void lcd1602_set_second_row() {
  lcd.setCursor(0, 1);
}

• lcd1602_test - displays a basic text for testing purposes

void lcd1602_test() {
  lcd.setCursor(2, 0);
  lcd.print("Booting up!");
}

• lcd1602_clear - clears the text from the LCD and sets the cursor to point to 0, 0

void lcd1602_clear() {
  lcd.clear();
  lcd.setCursor(0, 0);
}

Interpreter Functions:

• make_score - for a given row, calculates the score as the bias of said row + the sum of all input values with the coresponding weights

float make_score(float values[INPUTS], int row) {
  float score = bias[row];
  for (int j = 0; j < INPUTS; j++) {
    score += weights[row][j] * values[j];
  }
  return score;
}

• softmax - determines the class with the highest probability and yields both it and the probability, after calculating using the Log-Sum-Exp Trick.

void softmax(float scores[CLASSES], int* guess, float* prob) {
  float max_score = -INFINITY;
  for (int i = 0; i < CLASSES; ++i) {
    if (scores[i] > max_score) { 
      max_score = scores[i];
      *guess = i;
    }
  }
  float sum = 0;
  for(int i = 0; i < CLASSES; i++) {
    scores[i] = expf(scores[i] - max_score);
    sum += scores[i];
  }
  *prob = 100.0f / sum;
}

• classify - yields the class the data was classified to if the probability of it being corect is above 80%, otherwise yielding -1 (gesture not recognised)

int classify(float values[INPUTS]) {
  float scores[CLASSES];
  for (int i = 0; i < CLASSES; i++) {
    scores[i] = make_score(values, i);
  }
  int guess;
  float prob;
  softmax(scores, &guess, &prob);
  if (prob > 80) {
    return guess;
  } else {
    return -1;
  }
}

• translate - converts the input x into an apropriate letter

char translate(int x) {
  return 'A' + x;
}

Main Functions:

• handleButtonInterrupt - the interrupt function that toggles when a button is pressed. Debounces digitialy

void IRAM_ATTR handleButtonInterrupt() {
  uint32_t current = millis();
  if (current - last_interrupt > DEBOUNCE_THRESHOLD) {
    buttonPressed = true;
    last_interrupt = current;
  }
}

* perpare_lcd - to be called during setup, initialises and tests the lcd

void perpare_lcd() {
  lcd1602_init();
  delay(1000);
  lcd1602_test();
}

* prepare_glove - to be called during setup, warns the wearer to sit still before launching into sensor initailisatian and callibratian

void prepare_glove() {
  lcd1602_write_string(1, 0, "Get ready for");
  lcd1602_write_string(3, 1, "calibration");
  delay(2000);
  lcd1602_clear();
  lcd1602_write_char(7, 0, '5');
  delay(1000);
  lcd1602_write_char(7, 0, '4');
  delay(1000);
  lcd1602_write_char(7, 0, '3');
  delay(1000);
  lcd1602_write_char(7, 0, '2');
  delay(1000);
  lcd1602_write_char(7, 0, '1');
  delay(1000);
  lcd1602_write_string(1, 0, "Remain still!");
  lcd1602_write_string(1, 1, "Calibrating...");
  prepare_sensor(ref_unit);
  ref_unit.init();
  thumb_unit.init();
  index_unit.init();
  middle_unit.init();
  ring_unit.init();
  pinky_unit.init();
  lcd1602_clear();
  lcd1602_write_string(6, 0, "Done!");
}

* seek_i2c_connections - unused dev function, used for verifying I²C connectivity through the TCA9548 multiplexor

void seek_i2c_connections() {
  for (uint8_t mux = 0; mux < 8; mux++) {
    tca9548_select(mux);
    for (uint8_t address = 1; address < 127; address++) {
      Wire.beginTransmission(address);
      if (Wire.endTransmission() == 0) {
        Serial.print("Found device at 0x");
        Serial.print(address, HEX);
        Serial.print(" on ");
        Serial.println(mux, HEX);
        delay(10);
      }
    }
  }
}

* setup - initialises the serial communication stream, the multiplexor and lcd, then the senors. Configures pins for the connected led and button. Finally, if in recording mode, prints the csv header to the serial line 

void setup() {
  // put your setup code here, to run once:
  delay(1000);
  Serial.begin(115200);
  delay(1000);
  tca9548_init();
  delay(1000);
  perpare_lcd();
  delay(2000);
  prepare_glove();

  pinMode(LED_PIN, OUTPUT);

  pinMode(BUTTON_PIN, INPUT_PULLUP);
  attachInterrupt(digitalPinToInterrupt(BUTTON_PIN), handleButtonInterrupt, FALLING);

  Serial.println("Done.");
  if (FUNCTION_MODE == 1) {
    Serial.println("x1,y1,z1,x2,y2,z2,x3,y3,z3,x4,y4,z4,x5,y5,z5,x6,y6,z6,code");
  }
}

* print_angles - to be used in recording/learning mode, prints to the serial the data from a sensor

void print_angles(MPU6500 &unit) {
  xyzFloat angle = unit.get_data();
  Serial.print(angle.x);
  Serial.print(",");
  Serial.print(angle.y);
  Serial.print(",");
  Serial.print(angle.z);
  Serial.print(",");
}

* record_loop - waits for the button to be pressed, printing a snapshot of the position the serial line in csv data format

void record_loop() {
  if (buttonPressed) {
    buttonPressed = false;
    print_angles(ref_unit);
    print_angles(thumb_unit);
    print_angles(index_unit);
    print_angles(middle_unit);
    print_angles(ring_unit);
    print_angles(pinky_unit);
    Serial.println();
  }
}

* read_angles - to be used in interpreting mode, reads the data from a sensor to a float array buffer

void read_angles(float* data, MPU6500 &unit) {
  xyzFloat angle = unit.get_data();
  data[0] = angle.x;
  data[1] = angle.y;
  data[2] = angle.z;
}

* interpret_loop - reads data from the sensors, then classifies it, interpreting gestures held for at least 2 seconds depending on their value: -1 as a non-recognised position, 0 to 25 as letters 'A' to 'Z', 26 as "move cursor to first line of LCD", 27 as "move cursor to second line of LCD" and 28 as "clear lcd". While detecting a gesture it doesnt know, the LED is of, while it is detecting a gesture it knows, the LED is on

void interpret_loop() { 
  float data[INPUTS];
  read_angles(data, ref_unit);
  read_angles(data + 3, thumb_unit);
  read_angles(data + 6, index_unit);
  read_angles(data + 9, middle_unit);
  read_angles(data + 12, ring_unit);
  read_angles(data + 15, pinky_unit);
  int cls = classify(data);
  if (cls != active_class) {
    active_class = cls;
    active_start = millis();
    already_used_class = false;
  } else {
    uint32_t now = millis();
    if (now - active_start > 2000 && !already_used_class) {
      if (cls == -1) {
        digitalWrite(LED_PIN, LOW);
      } else {
        digitalWrite(LED_PIN, HIGH);
      }
      if (cls >= 0 && cls < 26) {
        lcd1602_write_char(translate(cls));
      }
      if (cls == 26) {
        lcd1602_set_first_row();
      }
      if (cls == 27) {
        lcd1602_set_second_row();
      }
      if (cls == 28) {
        lcd1602_clear();
      }
    }
  }
}

* loop - runs either record_loop or interpret_loop, based on the hardcoded FUNCTION_MODE currently set

void loop() {
  if (FUNCTION_MODE == 1) {
    record_loop();
  } else {
    interpret_loop();
  }
}