Show page

Differences

This shows you the differences between two versions of the page.

--- pm:prj2025:iotelea:ageorgescu2407 [2025/05/26 21:03]
ageorgescu2407 [Software Design]
+++ pm:prj2025:iotelea:ageorgescu2407 [2025/05/28 10:50] (current)
ageorgescu2407
@@ Line 23: / Line 23: @@
   * [[https://www.optimusdigital.ro/ro/optoelectronice-led-uri/697-led-verde-de-3-mm-cu-lentile-difuze.html?search_query=led&results=779|Green LED]]
 {{ :pm:prj2025:iotelea:asl_smart_glove_electrical_scheme.png?700 |}}
+Conexions:
+In my project I used the following color coding:
+  * For Power & GND (twisted together) I used the pairs Red/Black and Orange/Grey
+  * For Data/Clk (also twisted together) I used the pairs Yellow/Brown, Blue/White and Green/Purple
+  * This format allowed me to get 6 distinct pairing of Power/GND/Data/Clk, one for each of the sensors, for easier distinction!
+The Pins used are as such:
+  * The whole system is powered directly or indirectly from the 5v supplied by the Micro-USB
+  * SDA - D21
+  * SCL - D22
+  * LED Control - D26
+  * Button Control - 14
+  * 3v3 to power the IMUs
+  * VIN as a 5v output to power the LCD and TCA9548
+  * RST on the TCA9548 is connected to 5v (VIN) over a 4.7k resistor
+  * SDA and SCL are connected to 3v3 over 4.7k resistor each
+  * The SDn and SCn (outputs of the multiplexors) do not require pull-up resistors as those exist internally on the IMU breakout boards
+  * The LED is connected to its Control Pin through a 200 resistor.
+  * Thumb sensor is connected to line 7 (SD7, SC7)
+  * Index sensor is connected to line 6 (SD7, SC6)
+  * Middle sensor is connected to line 5 (SD7, SC5)
+  * Ring sensor is connected to line 4 (SD7, SC4)
+  * Pinky sensor is connected to line 3 (SD7, SC3)
+  * Reference sensor is connected to line 2 (SD2, SC2)
+Physical Wiring:
+{{ :pm:prj2025:iotelea:manusa_fizica_ageorgescu2407.jpg?700 |}}
 ====Software Design====
 Development Medium: Arduino IDE
@@ Line 39: / Line 66: @@
 TCA9548 Functions:
   * init - to be called during setup, activates the I²C connection
-  void tca9548_init() {
-    Wire.begin(SDA_PIN, SCL_PIN);
-  }
   * tca9548_select - sets the TCA9548 multiplexor to channel i, provided that i is a valid channel (0 to 7)
-  void tca9548_select(uint8_t i) {
-    if (i > 7) return;
-    Wire.beginTransmission(0x70);
-    Wire.write(1 << i);
-    Wire.endTransmission();
-  }
 ----
@@ Line 57: / Line 73: @@
 MPU6500 Functions:
   * Constructor - makes the MPU6500 library object and memorizes the TCA9548 channel it is on
-  MPU6500::MPU6500(uint8_t mux_channel) {
-    _mux_channel = mux_channel;
-    _mpu = new MPU6500_WE(MPU6500_ADDRESS);
-  }
   * init - to be called by setup, launches the IMU into autocalibration and configures it for maximum precision
-  void MPU6500::init() {
-    tca9548_select(_mux_channel);
-    _mpu->init();
-    _mpu->autoOffsets();
-    _mpu->enableGyrDLPF();
-    _mpu->setGyrDLPF(MPU6500_DLPF_6);
-    _mpu->setGyrRange(MPU6500_GYRO_RANGE_250);
-    _mpu->enableAccDLPF(true);
-    _mpu->setAccDLPF(MPU6500_DLPF_6);
-    _mpu->setAccRange(MPU6500_ACC_RANGE_2G);
-  }
   * get_data - to be called inside the loops, makes sure the communication is over the correct TCA9548 channel and fetches current angular data
-  xyzFloat MPU6500::get_data() {
-    tca9548_select(_mux_channel);
-    return _mpu->getAngles();
-  }
 ----
 LCD1602 Functions:
   * lcd1602_init - to be called during setup, initializes the LCD and turns on the backlight
-  void lcd1602_init() {
-    lcd.init();
-    lcd.backlight();
-    lcd.clear();
-    lcd.setCursor(0, 0);
-    delay(25);
-  }
   * lcd1602_write_char - writes a character at the specified position, provided that position is valid
-  void lcd1602_write_char(uint8_t pos_col, uint8_t pos_row, char c) {
-    if (pos_col >= LCD_COLS || pos_row >= LCD_ROWS) {
-      return;
-    }
-    lcd.setCursor(pos_col, pos_row);
-    lcd.print(c);
-  }
   * lcd1602_write_char - writes a character at the current cursor position
-  void lcd1602_write_char(char c) {
-    lcd.write(c);
-  }
   * lcd1602_write_string - writes a string at the specified positon
-  void lcd1602_write_string(uint8_t pos_col, uint8_t pos_row, const char* s) {
-    lcd.setCursor(pos_col, pos_row);
-    lcd.print(s);
-  }
   * lcd1602_set_first_row - moves the cursor to point at the begining of the first row
-  void lcd1602_set_first_row() {
-    lcd.setCursor(0, 0);
-  }
   * lcd1602_set_second_row - moves the cursor to point at the begining of the second row
-  void lcd1602_set_second_row() {
-    lcd.setCursor(0, 1);
-  }
   * lcd1602_test - displays a basic text for testing purposes
-  void lcd1602_test() {
-    lcd.setCursor(2, 0);
-    lcd.print("Booting up!");
-  }
   * lcd1602_clear - clears the text from the LCD and sets the cursor to point to 0, 0
-  void lcd1602_clear() {
+----
-    lcd.clear();
+Interpreter Functions:
-    lcd.setCursor(0, 0);
-  }
+  *make_score - for a given row, calculates the score as the bias of said row + the sum of all input values with the coresponding weights
+  *softmax - determines the class with the highest probability and yields both it and the probability, after calculating using the Log-Sum-Exp Trick.
+  *classify - yields the class the data was classified to if the probability of it being corect is above 80%, otherwise yielding -1 (gesture not recognised)
+  * translate - converts the input x into an apropriate letter
 ----
@@ Line 151: / Line 111: @@
   * handleButtonInterrupt - the interrupt function that toggles when a button is pressed. Debounces digitialy
-  void IRAM_ATTR handleButtonInterrupt() {
-    uint32_t current = millis();
-    if (current - last_interrupt > DEBOUNCE_THRESHOLD) {
-      buttonPressed = true;
-      last_interrupt = current;
-    }
-  }
   * perpare_lcd - to be called during setup, initialises and tests the lcd
-  void perpare_lcd() {
-    lcd1602_init();
-    delay(1000);
-    lcd1602_test();
-  }
   * prepare_glove - to be called during setup, warns the wearer to sit still before launching into sensor initailisatian and callibratian
-  void prepare_glove() {
-    lcd1602_write_string(1, 0, "Get ready for");
-    lcd1602_write_string(3, 1, "calibration");
-    delay(2000);
-    lcd1602_clear();
-    lcd1602_write_char(7, 0, '5');
-    delay(1000);
-    lcd1602_write_char(7, 0, '4');
-    delay(1000);
-    lcd1602_write_char(7, 0, '3');
-    delay(1000);
-    lcd1602_write_char(7, 0, '2');
-    delay(1000);
-    lcd1602_write_char(7, 0, '1');
-    delay(1000);
-    lcd1602_write_string(1, 0, "Remain still!");
-    lcd1602_write_string(1, 1, "Calibrating...");
-    prepare_sensor(ref_unit);
-    ref_unit.init();
-    thumb_unit.init();
-    index_unit.init();
-    middle_unit.init();
-    ring_unit.init();
-    pinky_unit.init();
-    lcd1602_clear();
-    lcd1602_write_string(6, 0, "Done!");
-  }
   * seek_i2c_connections - unused dev function, used for verifying I²C connectivity through the TCA9548 multiplexor
-  void seek_i2c_connections() {
-    for (uint8_t mux = 0; mux < 8; mux++) {
-      tca9548_select(mux);
-      for (uint8_t address = 1; address < 127; address++) {
-        Wire.beginTransmission(address);
-        if (Wire.endTransmission() == 0) {
-          Serial.print("Found device at 0x");
-          Serial.print(address, HEX);
-          Serial.print(" on ");
-          Serial.println(mux, HEX);
-          delay(10);
-        }
-      }
-    }
-  }
   * setup - initialises the serial communication stream, the multiplexor and lcd, then the senors. Configures pins for the connected led and button. Finally, if in recording mode, prints the csv header to the serial line
+  *
-  void setup() {
-    // put your setup code here, to run once:
-    delay(1000);
-    Serial.begin(115200);
-    delay(1000);
-    tca9548_init();
-    delay(1000);
-    perpare_lcd();
-    delay(2000);
-    prepare_glove();
-    pinMode(LED_PIN, OUTPUT);
-    pinMode(BUTTON_PIN, INPUT_PULLUP);
-    attachInterrupt(digitalPinToInterrupt(BUTTON_PIN), handleButtonInterrupt, FALLING);
-    Serial.println("Done.");
-    if (FUNCTION_MODE == 1) {
-      Serial.println("x1,y1,z1,x2,y2,z2,x3,y3,z3,x4,y4,z4,x5,y5,z5,x6,y6,z6,code");
-    }
-  }
   * print_angles - to be used in recording/learning mode, prints to the serial the data from a sensor
-  void print_angles(MPU6500 &unit) {
-    xyzFloat angle = unit.get_data();
-    Serial.print(angle.x);
-    Serial.print(",");
-    Serial.print(angle.y);
-    Serial.print(",");
-    Serial.print(angle.z);
-    Serial.print(",");
-  }
   * record_loop - waits for the button to be pressed, printing a snapshot of the position the serial line in csv data format
-  void record_loop() {
-    if (buttonPressed) {
-      buttonPressed = false;
-      print_angles(ref_unit);
-      print_angles(thumb_unit);
-      print_angles(index_unit);
-      print_angles(middle_unit);
-      print_angles(ring_unit);
-      print_angles(pinky_unit);
-      Serial.println();
-    }
-  }
   * read_angles - to be used in interpreting mode, reads the data from a sensor to a float array buffer
-  void read_angles(float* data, MPU6500 &unit) {
-    xyzFloat angle = unit.get_data();
-    data[0] = angle.x;
-    data[1] = angle.y;
-    data[2] = angle.z;
-  }
   * interpret_loop - reads data from the sensors, then classifies it, interpreting gestures held for at least 2 seconds depending on their value: -1 as a non-recognised position, 0 to 25 as letters 'A' to 'Z', 26 as "move cursor to first line of LCD", 27 as "move cursor to second line of LCD" and 28 as "clear lcd". While detecting a gesture it doesnt know, the LED is of, while it is detecting a gesture it knows, the LED is on
-  void interpret_loop() {
-    float data[INPUTS];
-    read_angles(data, ref_unit);
-    read_angles(data + 3, thumb_unit);
-    read_angles(data + 6, index_unit);
-    read_angles(data + 9, middle_unit);
-    read_angles(data + 12, ring_unit);
-    read_angles(data + 15, pinky_unit);
-    int cls = classify(data);
-    if (cls != active_class) {
-      active_class = cls;
-      active_start = millis();
-      already_used_class = false;
-    } else {
-      uint32_t now = millis();
-      if (now - active_start > 2000 && !already_used_class) {
-        if (cls == -1) {
-          digitalWrite(LED_PIN, LOW);
-        } else {
-          digitalWrite(LED_PIN, HIGH);
-        }
-        if (cls >= 0 && cls < 26) {
-          lcd1602_write_char(translate(cls));
-        }
-        if (cls == 26) {
-          lcd1602_set_first_row();
-        }
-        if (cls == 27) {
-          lcd1602_set_second_row();
-        }
-        if (cls == 28) {
-          lcd1602_clear();
-        }
-      }
-    }
-  }
   * loop - runs either record_loop or interpret_loop, based on the hardcoded FUNCTION_MODE currently set
+====Results====
+The overall result is a system that can be configured for either gesture interpretation or gesture learning. Functionally the ESP32 used could learn even more complex or larger matrices to continue using multionomial regression for more potential classes, learning new gestures that don't strinctly have to me mapped onto a character. The ESP32 has bluetooth/wifi functionality so it is completely plausible to modify a system like this to interact with smart devices by issuing commands. The choice to interpret sign-language is arbitrary. For my current learning set (30 entries for each gesture) inference analysis yields great results:
+  * Average precision - 0.98
+  * Average recall - 0.98
+  * F1-score - 0.98
+  * Accuracy - 0.98
+  * (PS. I know it looks odd that they are the same, but this is the result as calculated my sklearn's script)
+{{ :pm:prj2025:iotelea:conf_matrix_ageorgescu2407.png?800 |}}
-  void loop() {
+====Conclusions and Lessons====
-    if (FUNCTION_MODE == 1) {
+  * Not a final conclusion really, but I realised half-way through development that I bought MPU6500 instead of MPU6050s like originally intended, which is a lesson to read more clearly when buying parts, I suppose.
-      record_loop();
+  * Sometimes, the breakout schemes of the various components have diagrams that just actually lie, the TCA9548A for example was supposed to have its own pull up resistors, but I needed to at my own.
-    } else {
+  * Positional tracking is difficult to realise with 6DOF systems, I was able to salvage my idea only because hand gestures have limited freedom of movement, but Z-rotation instability ruins any attempts at keeping a valid virtual Oxyz coordinate system.
-      interpret_loop();
+  * I know it is minimal in effect, but I think solidly color coding my wiring by function may be the single best decision I made during the project.
-    }
+  * LCD factory settings set contrast to 0, took a while of code/verify/upload cycles before I realised that one.
-  }
+  * Always check if cables you buy are data-capable.
+  * Although it worked for this project, sensors with pins going out like the ones I am using are cumbersome due to their volume occupied blocking natural finger positions. Same issue with the wiring being to long. I think using hard wires might've possibly been more beneficial in retrospect.
+  * Avoid using textiles as a base support for projects, I had to sew my sensors in with thread and use a zip tie to position the breadboard because nothing was sticking.
+====Code Repository====
+Available [[https://github.com/AnduG/SmartGlove|here]]!