Show page

Differences

This shows you the differences between two versions of the page.

--- iothings:proiecte:2021:doorsmartlock [2022/01/24 15:47]
mihai_florin.neacsu [Android]
+++ iothings:proiecte:2021:doorsmartlock [2022/01/27 20:06] (current)
mihai_florin.neacsu [ANEX]
@@ Line 60: / Line 60: @@
 In order to send commands to ESP32 board using an Android application, we first need to setup the ESP32 as a web server and connect it to the WiFi. After the connection was resolved, in order to know where to send the request, the IP of the ESP32 will be retrieved using the ''WiFi.localIP()'' method. Because the scope of this project is to turn the relay ON/OFF (activating/deactivating the lock) the web server will respond at 2 HTTP_GET requests: http://192.168.0.128/relay/on and http://192.168.0.128/relay/off
-<code>
+    void setup() {
-#include "WiFi.h"
+        pinMode(RELAY_PIN, OUTPUT);
-#include "ESPAsyncWebServer.h"
+        digitalWrite(RELAY_PIN, HIGH);
+        Serial.begin(115200);
+        WiFi.begin(ssid, password);
+        while (WiFi.status() != WL_CONNECTED) {
+            delay(1000);
+            Serial.println("Connecting to WiFi..");
+        }
+        Serial.println(WiFi.localIP());
+        server.on("/relay/off", HTTP_GET   , [](AsyncWebServerRequest *request){
+            request->send(200, "text/plain", "ok");
+            digitalWrite(RELAY_PIN, HIGH);
+        });
+        server.on("/relay/on", HTTP_GET, [](AsyncWebServerRequest *request){
+             request->send(200, "text/plain","ok");
+             digitalWrite(RELAY_PIN, LOW);
+        });
+        server.begin();
+    }
-const char* ssid = "SSID";
+===== Android =====
-const char* password =  "PASS";
-AsyncWebServer server(80);
+{{:iothings:proiecte:2021:smartlock_android_app.png?200 |}}
-int RELAY_PIN = 27;
-void setup() {
- pinMode(RELAY_PIN, OUTPUT);
- digitalWrite(RELAY_PIN, HIGH);
- Serial.begin(115200);
- WiFi.begin(ssid, password);
- while (WiFi.status() != WL_CONNECTED) {
-   delay(1000);
-   Serial.println("Connecting to WiFi..");
- }
- Serial.println(WiFi.localIP());
- server.on("/relay/off", HTTP_GET   , [](AsyncWebServerRequest *request){
-   request->send(200, "text/plain", "ok");
-   digitalWrite(RELAY_PIN, HIGH);
- });
-  server.on("/relay/on", HTTP_GET, [](AsyncWebServerRequest *request){
-   request->send(200, "text/plain","ok");
-   digitalWrite(RELAY_PIN, LOW);
- });
- server.begin();
-}
-void loop(){}</code>
-===== Android =====
+The android application must be able to open/close the electromagnetic lock using two buttons. As a security measure, the lock can be opened only by the owner's face.
+The application is implementing a face recognition module based on a deep network. It is using a Convolutional Neural Network (CNN) that is already trained. The similarities between the owner's face and the user that is trying to open the lock are chosen by calculating the distances between the face characteristics extracted by the CNN between the two faces.
+The distance chosen is the Euclidean distance. Formula for Euclidean distance is as follows: {{:iothings:proiecte:2021:smartlock_euclidean_distance.png?100|}} , where **p**, **q** = two points in Euclidean n-space, **pi**, **qi** = Euclidean vectors, starting from the origin of the space (initial point) and **n** = n-space.
+The application has 3 buttons:
+  * One button for setting up the face of the 'Owner'
+  * One button for opening the lock by the 'User' (comparing the face of the user with the face of the owner)
+  * One button for closing the lock
 ==== Sending HTTP requests ====
+In order to send HTTP requests, the android application must have access to internet:
+    <uses-permission android:name="android.permission.INTERNET" />
+    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
+The requests will be made using //Volley// library.
+The application is using Volley in order to create a //RequestQueue// to which it will pass a //Request// objects. The //RequestQueue// is responsible for managing the threads when doing network operations (like reading from or writing to the cache) and and for parsing the responses. The //Requests// is parsing the raw response that will be dispatched to the main thread for delivery by Volley. [2]
+    RequestQueue queue;
+    String url = "http://192.168.0.128";
+    [...] // Other Code
+    private void sendHttpRequest(String auxUrl) { //auxUrl = url + "relay/on" or "relay/off"
+        // Request a string response from the provided URL.
+        StringRequest stringRequest = new StringRequest(Request.Method.GET, auxUrl,
+                new Response.Listener<String>() {
+                    @Override
+                    public void onResponse(String response) {
+                        // Display the first 500 characters of the response string.
+                        Toast.makeText(getApplicationContext(), "Success", Toast.LENGTH_SHORT).show();
+                    }
+                }, new Response.ErrorListener() {
+            @Override
+            public void onErrorResponse(VolleyError error) {
+                Log.e("ERROR", error.getMessage());
+                Toast.makeText(getApplicationContext(), "ERROR while sending the request", Toast.LENGTH_SHORT).show();
+            }
+        });
+        // Add the request to the RequestQueue.
+        queue.add(stringRequest);
+    }
 ==== Accessing Camera ====
+In order to use the camera, to take a picture of the //user// (in this case, the person that is trying to open the lock), the application must have the feature active:
+    <uses-feature android:name="android.hardware.camera" android:required="true" />
-Connecting the Android application to ESP32.
+Opening the camera, and taking a picture will be done by an //Intent// object as follows:
-Sending HTTP requests
+    Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
+    startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);
+==== Multitask Cascaded Convolutional Network ====
+One of the issues while writing the application is the face detection, because in a normal image, you will not have only the face present, you will also have the background, clothes etc.. The face detection issue was solved using a Multitask Cascaded Convolutional Network (MTCNN) – it is implemented in Java programming language using TensorFlow API. The result of using the MTCNN is a Bitmap that represent the face found in the given image.
+    private Bitmap cropFace(Bitmap bitmap){
+        Bitmap croppedBitmap = null;
+        try {
+            Vector<Box> boxes = mtcnn.detectFaces(bitmap, 10);
+            int x = boxes.get(0).left();
+            int y = boxes.get(0).top();
+            int width = boxes.get(0).width();
+            int height = boxes.get(0).height();
+            if (y + height >= bitmap.getHeight())
+                height -= (y + height) - (bitmap.getHeight() - 1);
+            if (x + width >= bitmap.getWidth())
+                width -= (x + width) - (bitmap.getWidth() - 1);
+            croppedBitmap = Bitmap.createBitmap(bitmap, x, y, width, height);
+        }catch (Exception e){
+            e.printStackTrace();
+        }
+        return croppedBitmap;
+    }
+==== FaceNet ====
+The face recognition model used is FaceNet, this model is running using TensorFlow lite API. The model was pretrained on the VGGFace2 training datasheet and is able to touch 100% accuracy on YALE, JAFFE, AT&T datasheets [3]. The result of running a face (obtained using the MTCNN) thru the FaceNet model is an ‘embedding’ (a low-dimensional space into which we can translate high-dimensional vectors [4]) that resembles the face characteristics.
+    facenet.run(FaceBitmap)
+==== Opening/Closing the lock ====
+The application has 1 button that is closing the lock and one button that is opening the lock.
+The lock button only sends the HTTP GET request to the ESP32 board.
+        closeLockButton.setOnClickListener(view -> {
+            Toast.makeText(getApplicationContext(), "Closing Door", Toast.LENGTH_SHORT).show();
+            String auxUrl = url + "/relay/off";
+            sendHttpRequest(auxUrl);
+        });
+The unlock button opens the camera intent, after the image is taken, the bitmap is retrieved, the face is cropped and the characteristics are extracted. At this point, the application should have both faces (owner face and user face) and compare the distance between each characteristic, if the faces are similar enough the lock is being opened.
+    openLockButton.setOnClickListener(view -> {
+        REQUEST_IMAGE_CAPTURE = 1;
+        PROCESS_OPEN_LOCK = 1;
+        dispatchTakePictureIntent();
+    });
+    [...]
+    protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {
+        super.onActivityResult(requestCode, resultCode, data);
+        if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {
+            Bundle extras = data.getExtras();
+            Log.i("USER", "Got a picture to process");
+            Bitmap imageBitmap = (Bitmap) extras.get("data");
+            facesForProcess.add(cropFace(imageBitmap));
+            REQUEST_IMAGE_CAPTURE = 0;
+            [...]
+            if (PROCESS_OPEN_LOCK == 1) {
+                float[][] faceToCompareCaracteristics = facenet.run(facesForProcess.poll());
+                faces.getFirst().faceScore = facenet.getSimilarityScore(faceToCompareCaracteristics, faces.getFirst().faceCaracteristics);
+                if (faces.getFirst().faceScore > 25)
+                    Toast.makeText(getApplicationContext(), "Your face is not recognised", Toast.LENGTH_SHORT).show();
+                else {
+                    Toast.makeText(getApplicationContext(), "Your face is recognised. Opening lock.", Toast.LENGTH_SHORT).show();
+                    String auxUrl = url + "/relay/on";
+                    Log.i("DEBUG", auxUrl);
+                    sendHttpRequest(auxUrl);
+                }
+                PROCESS_OPEN_LOCK = 0;
+            };
+        }
+    }
 ====== Conclusions ======
+In conclusion the ESP32 board it is a very good development board for IOT projects, having Bluetooth and Wi-Fi support it can be easily connected to an Webpage or an android application.
+Using a face recognition method for securing the lock raises some issues:
+  * Because the embedded device is not as powerful as a personal computer, the convolutional neural network that is going to be used must be pre-trained (otherwise the training will take a lot of time).
+  * Finding the face in the image, because even on a portrait image, there is still the background/clothes which do not give any relevant characteristic of the face.
 ====== Bibliography ======
 [1] https://esp32io.com/tutorials/esp32-relay, //ESP32 Relay//, 24.01.22
+[2] https://developer.android.com/training/volley/simple, //Send a simple request//, 24.01.22
+[3] https://ieeexplore.ieee.org/document/8985786, //Face Recognition using FaceNet//, 24.01.22
+[4] https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture, //Embeddings//, 24.01.22
+====== ANEX ======
+Code: https://gitlab.com/neacsumihaiflorin/smart-lock-using-esp32
+Demo: https://www.youtube.com/watch?v=qgijS8pG4ec