Rotaru Razvan-Andrei AAC2

By incorporating three distinct forms of authentication - PIN pad entry, NFC technology, and facial recognition - Smart Lock is a system that is both secure and user-friendly, with the purpose of offering an access barrier, in ordure to secure a house. The PIN pad serves as a knowledge-based authentication, requiring users to enter a personal code known only to them. NFC technology serves as a possession-based authentication, utilizing an object in the user's possession, such as a smartphone or card, to grant access. Lastly, facial recognition serves as a biological-based authentication, utilizing the unique characteristics of an individual's face to confirm their identity. Additionally, an Android application is used in order to manager the system, which provides operation like seeing the user list, adding users or deleting them. Together, these three forms of authentication and the mobile application provide a robust layer of security, ensuring that only authorized individuals are granted access to the building or area in question.

The following sections will provide the details of implementation for the hardware and software architectures and will show a demonstration of the implemented system.

This section provides a detailed analysis of the hardware components that make up the project, including an NFC reader, a pinpad, a buzzer, an OLED display, and a camera. The NFC reader allows for secure communication with NFC-enabled devices, the pinpad enables users to enter their personal identification number, the buzzer provides audible feedback, the OLED display provides real-time visual information and the camera serves as a security feature. All of these components are controlled by a microcontroller in oder to implement the smart lock.

ESP32-DevKitC V4 is a small-sized ESP32-based development board produced by Espressif. Most of the I/O pins are broken out to the pin headers on both sides for easy interfacing. Developers can either connect peripherals with jumper wires or mount ESP32-DevKitC V4 on a breadboard.

The MFRC522 is a highly integrated reader/writer IC for contactless communication at 13.56 MHz. The MFRC522 breakout board is a compact and easy-to-use board that allows for quick and simple integration of the MFRC522 IC into a wide range of applications. It includes all the necessary components to get started with using the MFRC522, such as a built-in antenna, and a variety of input/output pins for connecting to other devices. It communicates with the microcontroller via SPI in order to read or write the NFC card.

In order to receive the pin input from the user a 4×3 membrane pinpad was used. It uses a matrix circuit in order to detect button presses, and interfaces with the microcontroller via 7 simple GPIO pins. When a button is pressed, it connects the row and column of the button, which can be read by the microcontroller. The microcontroller then scans through each row and column to detect which button is pressed by sending a signal to each row and reading the corresponding columns.

To display the status of the application, a small 0.96 inch OLED Display was used. The display is controlled by the microcontroller using the I2C protocol and it's powered by 3.3V. It uses the SSD1306 drive and has a resolution of 128 x 64 pixels.

In order to achieve face recognition capabilities, an additional microcontroller was needed, alongside a camera module. The Raspberry Pi 4B is a small, low-cost computer that can be used for a variety of projects, including robotics and home automation. The Raspberry Pi camera module is a small camera that can be connected to the Raspberry Pi, allowing it to capture still images and video. The camera module connects to the Raspberry Pi via a ribbon cable that is inserted into the dedicated camera port on the Raspberry Pi board and uses the CSI (Camera Serial Interface) protocol. To inter-connect the main controller (ESP32) with the Raspberry, an UART interface was established between the two.

For a better user experience, acoustic signal are played using a piezo buzzer in different moments of the program's operation, like key presses, success and fail.

The software was divided into three components: the main controller, the vision control module, and the accompanying mobile app. These components work together to enable the features described in the first section, such as adding and removing users, as well as user recognition. This section will dive deeper into each component individually, providing a detailed description of its function, code implementation, and flow diagrams to help visualize its workings.

The ESP32 serves as the backbone of the project, acting as the main controller. Its Bluetooth capabilities, in conjunction with the attached sensors, display, keypad, and NFC sensors, allow for the management of users in the internal flash memory, the establishment of a connection with the accompanying mobile application, the communication with the vision control module, the reading of input from the keypad and NFC sensors, and the processing of commands received from the mobile application.

At its' core, the software running on the ESP32 represents a state machine, that follows either the registration or user recognition routine. A better visualisation of the states and branching conditions is seen below:

typedef enum _program_state
{
    kStateReadPinpad,
    kStateReadNFC,
    kStateReadFace,
    kStateReadSuccess,
    kStateReadFail,

    kStateRegisterNFC,
    kStateRegisterFace,
    kStateRegisterSuccess,
    kStateRegisterFail,

    kStateNone
} program_state_t;

The diagram illustrates the flow of operation for both registration and recognition processes:

• In the registration process, the user inputs their name and password, taps their NFC card on the reader, and registers their face. This information is then saved in the database.
• In the recognition process, the user inputs their password, taps their NFC card on the reader, and positions their face for recognition. If the input data matches the information stored in the database, the user is granted entry. Otherwise, entry is denied.

It is important to note that the order of operations is fixed, starting with the password, then the NFC card, and finally the face. Additionally, the registration process can only be initiated through the mobile application.

A “user” structure was implemented to track and save the progress of the user throughout the operation process. This structure is defined and instantiated globally, allowing it to be shared among all files. As the user progresses through each step of the state machine, the corresponding field of the structure is populated. Upon completion of the operation, the information is either saved in flash memory (in the case of registration) or compared with saved user information (in the case of recognition). Once the state resets to the pinpad reading, the user structure is cleared.

typedef struct User
{
    char name[USER_NAME_SIZE + 1] = "";
    char password[USER_PASSWORD_SIZE + 1] = "";
    char nfc_id[USER_NFC_ID_SIZE + 1] = "";
} User;
extern User user;

The development of the software was structured in a modular fashion, with the implementation split across multiple files. To accomplish this, the Arduino framework, along with the FreeRTOS real-time operating system, was utilized. The project was developed in Visual Studio Code, utilizing the PlatformIO extension to streamline the development process and provide additional functionality.

The Bluetooth Low Energy capabilies of the ESP32 chip are used in order to create a server to which the phone application can connect to. It acts as a peripheral and advertises itself by creating a service with two characteristics, for sending and receiving. A service is a collection of data and associated behaviors that encapsulate a certain functionality. A characteristic is a basic data element within a service. The Service and the two characteristics have unique UUIDs in order for the peripheral device (the phone) to use them based on the desired operation (write or read). When two BLE devices connect, they can discover each other's services and characteristics. Once they find the service and characteristic they are interested in, they can read or write the data in the characteristic. This is how data is exchanged over BLE.

#define SERVICE_UUID           "6E400001-B5A3-F393-E0A9-E50E24DCCA9E"
#define CHARACTERISTIC_UUID_RX "6E400002-B5A3-F393-E0A9-E50E24DCCA9E"
#define CHARACTERISTIC_UUID_TX "6E400003-B5A3-F393-E0A9-E50E24DCCA9E"

#define BLE_NAME               "ESP32CONTROL"
#define BLE_MTU_SIZE           512

Above can be seen the UUIDs, the name of the BLE server as it appears when scanning and the MTU size, which defines the size of a BLE packet.

#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

void BleServerInit()
{
    // Init the server with the name and set the MTU
    BLEDevice::init(BLE_NAME);
    BLEDevice::setMTU(BLE_MTU_SIZE);

    // Create the service with callbacks for connection and disconnection
    pServer = BLEDevice::createServer();
    pServer->setCallbacks(new BLEServCallbacks());

    pService = pServer->createService(SERVICE_UUID);

    // Create the transmitting charateristic, used to write data to the phone
    pTxCharacteristic = pService->createCharacteristic(
        CHARACTERISTIC_UUID_TX,
        BLECharacteristic::PROPERTY_NOTIFY);
    pTxCharacteristic->addDescriptor(new BLE2902());
   
    // Create the receiving charateristic, used to receive data from the phone
    BLECharacteristic *pRxCharacteristic = pService->createCharacteristic(
        CHARACTERISTIC_UUID_RX,
        BLECharacteristic::PROPERTY_WRITE);
    pRxCharacteristic->setCallbacks(new BLECallbacks());

    // start the advertising
    pService->start();

    pServer->getAdvertising()->addServiceUUID(pService->getUUID());
    pServer->getAdvertising()->start();
}

The above code showcases the initialisation of the BLE server and the attached characteristics having their unique UUIDs and what properties are enabled on them. The pRxCharacteristic has a callback attached, which contains the onWrite function, where the actual receiving of the message is done in code, as it will be highlighted below. The pTxCharacteristic has a BLE2902 descriptor attached, this is because it needs a CCC(Client Characteristic Configuration) descriptor to allow the connected device to enable or disable notifications for the characteristic. In our case, the phone, as it will be described in its' specific chapter, enables the notifications on this characteristic by writing the value “1” to the descriptor in order to receive messages from it.

void BleServerSend(uint8_t *data, size_t size) {
    if (deviceConnected) {   
        // Modifies the value of the characteristic
        pTxCharacteristic->setValue(data, size);
        // Notifies the peers that the characteristic changed
        pTxCharacteristic->notify();
        delay(10);
    }
}

The app_ble.c file also exposes the BleServerSend function to the rest of the program, which notifies the TX characteristic by setting its' value to the parameter.

void onWrite(BLECharacteristic *pCharacteristic) {
    std::string rxValue = pCharacteristic->getValue();
    if (rxValue.length() > 0) {
        processMessage((char *)rxValue.c_str());
    }
}

void processMessage(char *msg) {
    if (strstr(msg, "AT+ADDUSER=")) {
        char line[40] = "";
        sscanf(msg, "AT+ADDUSER=%s\r\n", line);
        
        // Saves the received name and password into the global user
        strcpy(user.name, strtok(line, ","));
        strcpy(user.password, strtok(NULL, ","));
        
        // change the program state Register NFC
        program_state = kStateRegisterNFC;
    } else if (strstr(msg, "AT+DELETEUSER=")) {
        char name[40] = "";
        sscanf(msg, "AT+DELETEUSER=%s\r\n", name);

        // Deletes the user from the database and from the vision module
        UserDatabaseDeleteUser(name);
        VisionSend(msg);
    } else if (strstr(msg, "AT+GETUSERS=")) {
        UserDatabaseGetUsers();
    } else if (strstr(msg, "AT+RESETDB=")) {
        UserDatabaseReset();
    }
}

Finally, received messages are parsed and depending on the command, other functions are called. Right now, The supported command are:

• “AT+ADDUSER=name,password” begins the registration progress
• “AT+DELETEUSER=name” deletes the user identified by the name from the database and from the Vision module
• “AT+GETUSERS=” returns the list of users
• “AT+RESETDB=” erases the user database

The OLED display is controlled by the SSD1306 driver and is attached to the I2C interface of the ESP32. The working principle is simple. When a change of state is detected, the display task switched the text shown on it, depending on the status of the program, as can be seen below. The role if the display is to guide and inform the about the state of the program and what he must do in order to advance.

#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
...
Adafruit_SSD1306 oledDisplay(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, -1);
...
void DisplayInit() {
    oledDisplay.begin(SSD1306_SWITCHCAPVCC, DIPLAY_I2C_ADDRESS);
    ...
    xTaskCreate(vDisplayTask, vDisplayTaskName, vDisplayTaskStackSize, NULL, vDisplayTaskPriority, NULL);
}
...
static void vDisplayTask(void *pvParameters) {
    program_state_t prev_program_state = kStateNone;

    for (;;) {
        if (program_state != prev_program_state) {
            prev_program_state = program_state;
            switch (program_state) {
            case kStateReadPinpad:
                writeToDisplay((char *)"(Reading) Input the pin on the keypad");
                break;
            case kStateReadNFC:
                writeToDisplay((char *)"(Reading) Tap the card on the NFC reader");
                break;
            ...
        }
        delay(50);
    }
}

The NFC functionality is enabled by the mfrc522 board connected to the SPI. The attached task only works when the program state concers the reader, namely the NFC Reading and NFC Registering states, as below:

#include <MFRC522.h>
...
MFRC522 mfrc522(SS_PIN, RST_PIN);
...
void NFCInit() {
    SPI.begin(SCK_PIN, MISO_PIN, MOSI_PIN, SS_PIN);
    mfrc522.PCD_Init();
    ...
    xTaskCreate(vNfcTask, vNfcTaskName, vNfcTaskStackSize, NULL, vNfcTaskPriority, NULL);
}

static void vNfcTask(void *pvParameters) {
    for (;;) {
        if (program_state == kStateReadNFC || program_state == kStateRegisterNFC) {
            if (mfrc522.PICC_IsNewCardPresent()) {
                if (mfrc522.PICC_ReadCardSerial()) {
                    // converts the card's ID into string
                    char nfc_id[22] = "";
                    byte_array_to_string(mfrc522.uid.uidByte, mfrc522.uid.size, nfc_id);
 
                    // Saves the NFC id into the global user
                    strcpy(user.nfc_id, nfc_id);
                    
                    if (program_state == kStateRegisterNFC) {
                        // Send the registration command to the vision module
                        char msg[50] = "";
                        snprintf(msg, 50, "AT+REGISTER=%s\n", user.name);
                        VisionSend((const char *)msg);

                        // Change status, play a tone on the buzzer
                        program_state = kStateRegisterFace;
                    } else if (program_state == kStateReadNFC) {
                        // Send the recognition command to the vision module
                        VisionSend((const char *)"AT+READUSER=\n");
                        
                        // Change status, play a tone on the buzzer
                        program_state = kStateReadFace;
                    }
                    ...

The code snippet above demonstrates how the NFC reader is activated and used during the reading or registering process. When the status changes to reading or registering, the reader begins to pick up the first card it comes into contact with. The NFC ID is then saved into the user variable. The code also sends a command to the Vision module, depending on the program's current state. The actions taken by the Vision module in response to these commands will be further explained in the relevant chapter.

The Pinpad module describes the keypad object as a 4×3 matrix, allowing for flexibility in defining the characters for each key press. The module checks the state of the program and when it involves the keypad, it reads the key presses and builds the password character by character until the desired length is reached. The password is then saved into the global user variable and the program state is advanced to proceed with the user recognition operation.

#include <Keypad.h>
...
static char keys[ROW_NUM][COLUMN_NUM] = {
    {'1', '2', '3'},
    {'4', '5', '6'},
    {'7', '8', '9'},
    {'*', '0', '#'}};

static byte rowPins[ROW_NUM] = {R1_PIN, R2_PIN, R3_PIN, R4_PIN};
static byte colPins[COLUMN_NUM] = {C1_PIN, C2_PIN, C3_PIN};

static Keypad keypad = Keypad(makeKeymap(keys), rowPins, colPins, ROW_NUM, COLUMN_NUM);
...
void PinpadInit() {
    xTaskCreate(vPinpadTask, vPinpadTaskName, vPinpadTaskStackSize, NULL, vPinpadTaskPriority, NULL);
}

static void vPinpadTask(void *pvParameters) {
    int temp_key_sz = 0;
    char temp_key[PIN_LENGTH + 1] = "";

    for (;;) {
        if (program_state == kStateReadPinpad) {
            char key = keypad.getKey();
            if (key) {
                // Save a key
                temp_key[temp_key_sz++] = key;
                play_tone(kToneKeyPress);
                if (temp_key_sz >= PIN_LENGTH) {
                    // After 6 presses, save the password in the global user
                    strcpy(user.password, temp_key);

                    // Change state and play tone
                    program_state = kStateReadNFC;
                    play_tone(kToneSuccess);
                ...

Users are saved inside the SPI flash of the ESP32 in a simple txt file with the format name,password,nfc_id per line. The name acts as the unique identifier of the user, thus, as long as the name differs, the same nfc card and password can be used by different people. The app_user_manager.cpp file exposes different functions.

• UserDatabaseInit() is used to initialise the flash and to create the database in case it doesn't exists
• UserDatabaseReset() is used to delete the database of users
• UserDatabaseSaveUser(char *user_name, char *user_password, char *user_nfc_id) is called after the registration process finishes with success. It works by appending to the file the characteristics of the global user in a line of text. It also checks if a user with the same name exists, then sends a BLE message to the phone to inform with the result of the operation
• UserDatabaseGetUsers() send to the phone app a BLE message with the list of users in the form of “name1,name2,etc.”
• UserDatabaseMatchUser(char *user_name, char *user_password, char *user_nfc_id) returns true or false if it finds an entry in the database which contains all the parameters of the user. This is used to determine the result of user recognition.
• UserDatabaseDeleteUser(char *user_name) finds the line with the given username and deletes it. This is done by creating a new file, copying all lines that don't contain the name, skipping the one that contains and then replacing the old file with the new file. The limitation is caused by the file system which can't delete lines in a file, but only to replace bytes.

#include "FS.h"
#include "SPIFFS.h"
...
void UserDatabaseInit() {
    if (!SPIFFS.begin(FORMAT_SPIFFS_IF_FAILED)) {
        LOG_MSG("SPIFFS Mount Failed");
        for(;;);
    }

    if (!SPIFFS.exists(USER_DB_FILE_PATH))
        UserDatabaseReset();
}
void UserDatabaseReset();
void UserDatabaseSaveUser(char *user_name, char *user_password, char *user_nfc_id);
void UserDatabaseGetUsers();
bool UserDatabaseMatchUser(char *user_name, char *user_password, char *user_nfc_id);
void UserDatabaseDeleteUser(char *user_name);

The app_vision.cpp file begins with the initialisation of a serial UART with the Vision module. It then creates a task that is used to check for receiving messages. The received messages are then processed this way:

• “AT+REGISTER=OK” means that the face of the user was succesfully registered in the system, which then prompts the program to save the user in the database and to update the phone application with the result
• “AT_DELETEUSER=OK” means that the deletion of the user was completed with success
• “AT+READUSER=name” is returned from the Vision module when it succesfully detects a face. The name parameter is either the name of the user, which is the unique identifier, or ”-1” if the face detected by the module is not in the database.

The source implements the VisionSend(message) function which is used by the other files as seen in previous explanations to send the following commands to the Vision module:

• “AT+REGISTER=name” begins the face registration process
• “AT+DELETEUSER=name” deletes the specified user from the Vision database
• “AT+READUSER=” begins the face recognition process

#include "HardwareSerial.h"
...
HardwareSerial SerialPort(1);
...
void VisionInit() {
    SerialPort.begin(115200, SERIAL_8N1, VISION_UART_RX_PIN, VISION_UART_TX_PIN);
    xTaskCreate(vVisionTask, vVisionTaskName, vVisionTaskStackSize, NULL, vVisionTaskPriority, NULL);
}

static void vVisionTask(void *pvParameters) {
    for (;;) {
        if (SerialPort.available()) {
            String msg = SerialPort.readString();
            processMessage(msg.c_str());
        }
        delay(10);
    }
}

static void processMessage(const char *msg) {
    if (strstr(msg, "AT+REGISTER=")) {
        ...
        if (register_result > 0) {
            UserDatabaseSaveUser(user.name, user.password, user.nfc_id);
            BleServerSend((uint8_t *)"AT+ADDUSER=OK\r\n", strlen("AT+ADDUSER=OK\r\n"));
            program_state = kStateRegisterSuccess;
            play_tone(kToneSuccess);
        }
        
        reset_temp_user();
        program_state = kStateReadPinpad;
    } else if (strstr(msg, "AT+DELETEUSER=")) {
        ...
        if (delete_result > 0) {
            BleServerSend((uint8_t *)"AT+DELETEUSER=OK\r\n", strlen("AT+DELETEUSER=OK\r\n"));
        }
    } else if (strstr(msg, "AT+READUSER=")) {
        ...
        if (result) {
            program_state = kStateReadSuccess;
            play_tone(kToneSuccess);
        } else {
            program_state = kStateReadFail;
            play_tone(kToneFail);
        }
        ...
    }
}

void VisionSend(const char *msg) {
    SerialPort.print(msg);
}

Finally, using all the modules described above, a bigger picture of what is going on in the ESP32 can be seen below:

The Vision module is responsible for registering and detecting the user's face. This is accomplished by integrating the OpenCV engine with the Raspberry PI hardware and camera. The module comprises of two components, the SerialManager and the FaceManager. In this section, we will provide an overview of these components and describe the process of adding or detecting a face.

The programming language chosen for this project was Python, as it is well-supported on the Linux operating system of the Raspberry PI. OpenCV, the chosen engine for image processing, is easy to use and has an intuitive API. Additionally, there is a specialized library available for controlling the camera, making it an ideal choice for the project.

class SerialManager():
    def __init__(self):
        # Open UART0
        self.connection = serial.Serial("/dev/serial0", baudrate=115200, timeout=1)
        self.serial_thread = threading.Thread(target=self.read_from_serial)
        self.serial_thread.start()
    def read_from_serial(self):
        ...
        data = self.connection.readline().strip().decode('utf-8')
        ...
        if "AT+REGISTER=" in data:
            name = data[data.find("=")+1:]
            face_manager.register_face(name)
        elif "AT+READUSER=" in data:
            face_manager.recognise_user()
        elif "AT+DELETEUSER=" in data:
            name = data[data.find("=") + 1:]
            face_manager.delete_face(name)
        ...

The SerialManager is responsible for facilitating communication between the ESP32 and the Raspberry PI. This is achieved by opening a serial connection on the UART0 of the Raspberry PI, using the same baud rate as the one used on the ESP32. A separate thread is also started to continuously check for incoming messages. The supported commands are:

• “AT+REGISTER=name” proceeds to register a face with the given name as ID
• “AT+READUSER=” which the recognition process
• “AT+DELETEUSER=name” deletes the face of the user from the face database

All the above described operations are done with the help of the face manager.

class FaceManager():
    def __init__(self):
        self.camera = init_camera()
        self.faceCascade = cv2.CascadeClassifier("haarcascade_frontalface_alt2.xml")
        self.recognizer = cv2.face.LBPHFaceRecognizer_create()

The initialization of the FaceManager begins with configuring the camera hardware, which includes setting the resolution, frame rate, and rotation of the captured images. Then, it loads a classifier that is used to detect faces within the frames captured by the camera. Finally, a recognizer is loaded, which is used to train the face recognition model and make predictions on the faces in an image.

def register_face(self, user_name):
	dirName = "./dataset/" + user_name
	os.makedirs(dirName)
        ...
	for frame in self.camera.capture_continuous(self.rawCapture, format="bgr", use_video_port=True):
		if count > 100:
			break
                ...
		gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
		faces = self.faceCascade.detectMultiScale(gray, scaleFactor=1.05, minNeighbors=5, minSize=(200, 200))
                ...
		for (x, y, w, h) in faces:
			roiGray = gray[y:y + h, x:x + w]
			fileName = dirName + "/" + user_name + str(count) + ".jpg"
			cv2.imwrite(fileName, roiGray)
			count+=1
                ...
	self.train_model()
	serial_manager.write_to_serial("AT+REGISTER=1\r\n".encode("utf-8"))

The face registration function starts by creating a directory with the user's name. It then captures frames from the camera and attempts to detect faces in the images using the classifier. For each face detected, a file is created within the directory and the cropped face is converted to grayscale. Once 100 pictures have been saved with the user's face, the model is trained and a response is sent back to the ESP32 indicating that the registration process is complete.

def train_model(self):
    ...
    for root, dirs, files in os.walk(imageDir):
        for file in files:
            if file.endswith("png") or file.endswith("jpg"):
                label = os.path.basename(root)
                ...
                faces = self.faceCascade.detectMultiScale(imageArray, scaleFactor=1.05, minNeighbors=3, minSize=(30, 30))
                ...
                for (x, y, w, h) in faces:
                    roi = imageArray[y:y + h, x:x + w]
                    xTrain.append(roi)
                    yLabels.append(id_)
    ...
    self.recognizer.train(xTrain, np.array(yLabels))
    self.recognizer.save("trainer.yml")

The train_model function is used to train a model that can recognize different users. It iterates over all the directories where users have their face images saved, and creates a model that is able to detect the faces of all the registered users. The name of the user is used as a label for recognition.

def recognise_user(self):
    ...
    for frame in self.camera.capture_continuous(self.rawCapture, format="bgr", use_video_port=True):
        faces = self.faceCascade.detectMultiScale(gray, scaleFactor=1.05, minNeighbors=5, minSize=(200, 200))
        ...
        for (x, y, w, h) in faces:
            roiGray = gray[y:y + h, x:x + w]
            ...
            id_, conf = self.recognizer.predict(roiGray)
            if conf <= 80:
                detections[name] += 1
            ...    
            if detections[name] > 10:
                if name == "unknown":
                    serial_manager.write_to_serial(("AT+READUSER=" + "-1" + "\r\n").encode("utf-8"))
                else:
                    serial_manager.write_to_serial(("AT+READUSER=" + name + "\r\n").encode("utf-8"))
            ... 

The recognition process begins by capturing frames from the camera and extracting the face from them. The recognizer then attempts to predict the user by comparing the extracted face with the trained model. If the recognizer is able to predict the user with a sufficient level of confidence, the user's name is saved and the number of successful detections is incremented. If the recognizer is unable to make a prediction with confidence, the user is labeled as “unknown”. Once a user or “unknown” is detected 10 times, a response is sent back to the ESP32, either ”-1” for an unknown user or the user's name in case of a successful recognition.

The flow of operations inside the Vision module as described in the code snippets above can be highligthed in the following diagram:

The mobile application provides a user-friendly interface that allows the user to interact with the system. It offers several functionalities such as displaying the list of currently registered users, deleting a user, and registering a new user. The mobile application serves as a convenient way for the user to manage the system's user database.

• The app launches and scans for nearby Bluetooth devices, and establishes a connection with the ESP32 by matching its advertised name.
• Once connected, it discovers the services and characteristics provided by the ESP32.
• The app then searches for the UUIDs described in the Bluetooth section, and enables the notify property on the descriptor of the TX characteristic of the ESP32 to enable receiving messages.
• After enabling notifications, the bluetooth also requests a bigger MTU in order to support bigger messages.
• On the main screen, it attempts to acquire the user list from the ESP32 and displays it in a list format for the user to view.
• The user can select a user from the list and delete them through the additional menu that appears.
• The app also has a feature for adding new users, which prompts the user for their name and password when clicked.
• Upon clicking OK, the registration process starts and a loading bar appears on the screen until it is completed.
• The user is then prompted to follow the instructions displayed on the physical board to complete the registration.

ScanSettings scanSettings = new ScanSettings.Builder()
        .setScanMode(ScanSettings.SCAN_MODE_LOW_POWER)
        .setCallbackType(ScanSettings.CALLBACK_TYPE_FIRST_MATCH)
        .setMatchMode(ScanSettings.MATCH_MODE_AGGRESSIVE)
        .setNumOfMatches(ScanSettings.MATCH_NUM_ONE_ADVERTISEMENT)
        .setReportDelay(0L)
        .build();
scanner.startScan(filters, scanSettings, scanCallback);
...
public void onScanResult(int callbackType, ScanResult result) {
    BluetoothDevice device = result.getDevice();
    mBluetoothGatt = device.connectGatt(mContext, false, BLEGattCallback, TRANSPORT_LE);
}
... 
if (gattCharacteristic.getUuid().equals(CHARACTERISTIC_UUID_TX)) {
    mBluetoothGatt.setCharacteristicNotification(gattCharacteristic, true);
    BluetoothGattDescriptor descriptor = gattCharacteristic.getDescriptor(DESCRIPTOR_UUID);
    descriptor.setValue(BluetoothGattDescriptor.ENABLE_NOTIFICATION_VALUE);
    mBluetoothGatt.writeDescriptor(descriptor);
    gatt.requestMtu(512);
}      
...
public void onCharacteristicChanged(BluetoothGatt gatt, BluetoothGattCharacteristic characteristic) {
    /* Message received here */
    MainActivity.getInstance().processBLEMessage(new String(characteristic.getValue(), StandardCharsets.UTF_8));
}
...
public void sendMessage(String message) {
    BluetoothGattCharacteristic writeChar = mBluetoothGatt.getService(SERVICE_UUID).getCharacteristic(CHARACTERISTIC_UUID_RX);
    writeChar.setValue(message.getBytes(StandardCharsets.UTF_8));
    mBluetoothGatt.writeCharacteristic(writeChar);
}

The flow of operation can be seen in the following diagram:

This section will showcase a video demonstration of the project. The operations of user registration, recognition and deletion will be presented from phone's and board assembly perspectives.

Demo video on drive

In conclusion, Smart Lock is a highly secure and user-friendly system that utilizes a combination of three distinct forms of authentication - PIN pad entry, NFC technology, and facial recognition - to ensure that only authorized individuals are granted access to a building or area. The system also includes an Android application, which allows users to manage the system and perform functions such as adding or deleting users and viewing the user list. Overall, Smart Lock's innovative use of multiple forms of authentication and its accompanying mobile application provide a robust layer of security, making it a reliable and effective solution for securing a home or other building.

Challenges and future improvements:

• UART between ESP32 and Raspberry PI is slow at times.
• Tried to add a touch display over the Raspberry PI in order to playback the camera feed, but failed due to a driver error. Couldn't replace hardware in time.
• The possibily of starting the recognition process from the phone by clicking on a user and putting the pin, then following with NFC and face on device.
• Integrate Matter/Google Home into the system.
• Improve face recognition with Infrared and 3D camera to avoid spoofing.

• ESP32 Devkit
• MFRC522
• Membrane keypad
• OLED Display
• Raspberry PI 4B
• Raspberry PI Camera

• FreeRTOS
• ESP32 BLE Server
• OLED Display
• MFRC522
• Keypad
• ESP32 Flash
• UART Serial
• RaspberryPi Face Recognition
• Android BLE

Project github repo Smart Lock presentation