Recent changes

Introduction

The objective of this project is to create a smart lock for doors using an ESP32 module acting as a asynchronous web server, which will receive HTTP GET commands from an android application.

Hardware

Main components:

Components for NPN transistor switch:

10k resistor

2k resistor

NPN 2n2222 transistor

Circuit Diagram

The pins of a relay module can be categorized in two groups: low-voltage group and high-voltage group. The low-voltage group is used to turn ON/OFF the relay (connecting/disconnecting the high-voltage circuit) [1]

Figure 1. Schematic for the connection between ESP32 and the Relay module

In order to understand the correlation between the schematic from Figure 1 and the breadboard implementation we need to understand what each pin of the transistor means. This can be seen in Figure 2.

Figure 2. NPN Transistor 2n2222

Now that we understand the transistor's connections we can implement the schematic from Figure 1. The breadboard implementation can be seen in Figure 3.

Figure 3. Breadboard implementation

The Electromagnetic Lock and the 12V Source will be connected to the 5V Relay on the 'Normally Closed' (NC) position.

How will the hardware work

The 3.3V GPIO pin from the ESP32 board will send a signal to the NPN switch to turn it ON/OFF, this will send the 5V from Vin pin of the ESP32 board to the IN pin of the 'low-voltage group' of the 5V Relay turning it ON/OFF (activating/deactivating the electromagnetic lock).

If this project would've used a 3.3V Relay module, the NPN transistor switch would not be necessary, this is a workaround to give the Relay module the tension it needs to be turned ON/OFF.

Software

Arduino

In order to send commands to ESP32 board using an Android application, we first need to setup the ESP32 as a web server and connect it to the WiFi. After the connection was resolved, in order to know where to send the request, the IP of the ESP32 will be retrieved using the WiFi.localIP() method. Because the scope of this project is to turn the relay ON/OFF (activating/deactivating the lock) the web server will respond at 2 HTTP_GET requests: http://192.168.0.128/relay/on and http://192.168.0.128/relay/off

  void setup() {
      pinMode(RELAY_PIN, OUTPUT);
      digitalWrite(RELAY_PIN, HIGH);
      Serial.begin(115200);
      WiFi.begin(ssid, password);
      
      while (WiFi.status() != WL_CONNECTED) { 
          delay(1000);
          Serial.println("Connecting to WiFi..");
      }
      
      Serial.println(WiFi.localIP());
      server.on("/relay/off", HTTP_GET   , [](AsyncWebServerRequest *request){
          request->send(200, "text/plain", "ok");
          digitalWrite(RELAY_PIN, HIGH);
      });
      server.on("/relay/on", HTTP_GET, [](AsyncWebServerRequest *request){
           request->send(200, "text/plain","ok");
           digitalWrite(RELAY_PIN, LOW);
      });
      
      server.begin();
  }

Android

The android application must be able to open/close the electromagnetic lock using two buttons. As a security measure, the lock can be opened only by the owner's face.

The application is implementing a face recognition module based on a deep network. It is using a Convolutional Neural Network (CNN) that is already trained. The similarities between the owner's face and the user that is trying to open the lock are chosen by calculating the distances between the face characteristics extracted by the CNN between the two faces.

The distance chosen is the Euclidean distance. Formula for Euclidean distance is as follows: , where p, q = two points in Euclidean n-space, pi, qi = Euclidean vectors, starting from the origin of the space (initial point) and n = n-space.

The application has 3 buttons:

One button for setting up the face of the 'Owner'

One button for opening the lock by the 'User' (comparing the face of the user with the face of the owner)

One button for closing the lock

Sending HTTP requests

In order to send HTTP requests, the android application must have access to internet:

  <uses-permission android:name="android.permission.INTERNET" />
  <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />

The requests will be made using Volley library.

The application is using Volley in order to create a RequestQueue to which it will pass a Request objects. The RequestQueue is responsible for managing the threads when doing network operations (like reading from or writing to the cache) and and for parsing the responses. The Requests is parsing the raw response that will be dispatched to the main thread for delivery by Volley. [2]

  RequestQueue queue;
  String url = "http://192.168.0.128";
  
  [...] // Other Code
  
  private void sendHttpRequest(String auxUrl) { //auxUrl = url + "relay/on" or "relay/off"
      // Request a string response from the provided URL.
      StringRequest stringRequest = new StringRequest(Request.Method.GET, auxUrl,
              new Response.Listener<String>() {
                  @Override
                  public void onResponse(String response) {
                      // Display the first 500 characters of the response string.
                      Toast.makeText(getApplicationContext(), "Success", Toast.LENGTH_SHORT).show();
                  }
              }, new Response.ErrorListener() {
          @Override
          public void onErrorResponse(VolleyError error) {
              Log.e("ERROR", error.getMessage());
              Toast.makeText(getApplicationContext(), "ERROR while sending the request", Toast.LENGTH_SHORT).show();
          }
      });
      // Add the request to the RequestQueue.
      queue.add(stringRequest);
  }

Accessing Camera

In order to use the camera, to take a picture of the user (in this case, the person that is trying to open the lock), the application must have the feature active:

  <uses-feature android:name="android.hardware.camera" android:required="true" />

Opening the camera, and taking a picture will be done by an Intent object as follows:

  Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
  startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);

Multitask Cascaded Convolutional Network

One of the issues while writing the application is the face detection, because in a normal image, you will not have only the face present, you will also have the background, clothes etc.. The face detection issue was solved using a Multitask Cascaded Convolutional Network (MTCNN) – it is implemented in Java programming language using TensorFlow API. The result of using the MTCNN is a Bitmap that represent the face found in the given image.

  private Bitmap cropFace(Bitmap bitmap){
      Bitmap croppedBitmap = null;
      try {
          Vector<Box> boxes = mtcnn.detectFaces(bitmap, 10);
          int x = boxes.get(0).left();
          int y = boxes.get(0).top();
          int width = boxes.get(0).width();
          int height = boxes.get(0).height();
          
          if (y + height >= bitmap.getHeight())
              height -= (y + height) - (bitmap.getHeight() - 1);
          if (x + width >= bitmap.getWidth())
              width -= (x + width) - (bitmap.getWidth() - 1);
          
          croppedBitmap = Bitmap.createBitmap(bitmap, x, y, width, height);
      }catch (Exception e){
          e.printStackTrace();
      }
      return croppedBitmap;
  }

FaceNet

The face recognition model used is FaceNet, this model is running using TensorFlow lite API. The model was pretrained on the VGGFace2 training datasheet and is able to touch 100% accuracy on YALE, JAFFE, AT&T datasheets [3]. The result of running a face (obtained using the MTCNN) thru the FaceNet model is an ‘embedding’ (a low-dimensional space into which we can translate high-dimensional vectors [4]) that resembles the face characteristics.

  facenet.run(FaceBitmap)

Opening/Closing the lock

The application has 1 button that is closing the lock and one button that is opening the lock.

The lock button only sends the HTTP GET request to the ESP32 board.

      closeLockButton.setOnClickListener(view -> {
          Toast.makeText(getApplicationContext(), "Closing Door", Toast.LENGTH_SHORT).show();
          String auxUrl = url + "/relay/off"; 
          sendHttpRequest(auxUrl);
      });

The unlock button opens the camera intent, after the image is taken, the bitmap is retrieved, the face is cropped and the characteristics are extracted. At this point, the application should have both faces (owner face and user face) and compare the distance between each characteristic, if the faces are similar enough the lock is being opened.

  openLockButton.setOnClickListener(view -> {
      REQUEST_IMAGE_CAPTURE = 1;
      PROCESS_OPEN_LOCK = 1;
      dispatchTakePictureIntent();
  });
  
  [...]
  
  protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {
      super.onActivityResult(requestCode, resultCode, data);
      
      if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {
          Bundle extras = data.getExtras();
          Log.i("USER", "Got a picture to process");
          Bitmap imageBitmap = (Bitmap) extras.get("data");
          facesForProcess.add(cropFace(imageBitmap));
          REQUEST_IMAGE_CAPTURE = 0;
          
          [...]
          
          if (PROCESS_OPEN_LOCK == 1) {
              float[][] faceToCompareCaracteristics = facenet.run(facesForProcess.poll());
              faces.getFirst().faceScore = facenet.getSimilarityScore(faceToCompareCaracteristics, faces.getFirst().faceCaracteristics);
              if (faces.getFirst().faceScore > 25)
                  Toast.makeText(getApplicationContext(), "Your face is not recognised", Toast.LENGTH_SHORT).show();
              else {
                  Toast.makeText(getApplicationContext(), "Your face is recognised. Opening lock.", Toast.LENGTH_SHORT).show();
                  String auxUrl = url + "/relay/on";
                  Log.i("DEBUG", auxUrl);
                  sendHttpRequest(auxUrl);
              }
              PROCESS_OPEN_LOCK = 0;
          };
      }
  }

Conclusions

In conclusion the ESP32 board it is a very good development board for IOT projects, having Bluetooth and Wi-Fi support it can be easily connected to an Webpage or an android application.

Using a face recognition method for securing the lock raises some issues:

Because the embedded device is not as powerful as a personal computer, the convolutional neural network that is going to be used must be pre-trained (otherwise the training will take a lot of time).

Finding the face in the image, because even on a portrait image, there is still the background/clothes which do not give any relevant characteristic of the face.