Show page

Differences

This shows you the differences between two versions of the page.

--- iothings:proiecte:2025sric:digitalcamera [2025/05/25 21:54]
adrian.vladescu
+++ iothings:proiecte:2025sric:digitalcamera [2025/05/28 21:35] (current)
adrian.vladescu
@@ Line 1: / Line 1: @@
-====== ESP32 DIGITAL CAMERA ======
+====== ESP32 Digital Camera ======
   * Student: Vladescu Adrian-Marian
@@ Line 56: / Line 56: @@
 Two esp32 boards are integrated in this project: one for capturing and sending image frames, and another for receiving and rendering live streams. A flask server serves as a central location for receiving, processing, storing, and serving frames.  Each ESP32 device carries out distinct real-time duties in the system's distributed Internet of Things architecture, while the central Flask server controls user access and state.
 The ESP32-CAM Firmware
 Initializing the OV2640 camera sensor, establishing a Wi-Fi connection, sending JPG frames to the server on a regular basis, and pooling the server for flash control state are its primary duties.
@@ Line 82: / Line 83: @@
 The GPIO4 pin controls the onboard flash LED, toggled based on the server state.
+The ESP32-Display Firmware
+Enabling snapshot capturing and gallery browsing, rendering frames in real-time on a 320x240 TFT screen, showing and managing filter settings via touch, and connecting to the Flask /video_feed endpoint to decode MJPEG are among the primary duties.
+.	JPEG Streaming and Decoding
+The Flask server uses a MJPEG-style endpoint (/video_feed) that sends JPEG frames continuously.  The JPEG byte stream is retrieved by the ESP32 display board by establishing a constant TCP connection in the manner described below:
+<code>
+streamClient.connect(flask_ip, 5000);
+streamClient.println("GET /video_feed HTTP/1.1");
+</code>
+As data arrives, the board detects JPEG frame boundaries using standard JPEG markers:
+  * 0xFFD8 → Start of Image (SOI)
+  * 0xFFD9 → End of Image (EOI)
+Once a full frame is received, it is passed to the TJpg_Decoder library:
+<code>
+TJpgDec.drawJpg(0, 0, jpgBuf, pos);
+</code>
+Using a callback, the TJpgDec.drawJpg() function renders the JPEG data to the screen block by block after decoding it straight from memory.  This prevents full image buffering, lowers RAM use, and allows for fluid, real-time display for devices with limited memory, such as the ESP32.  For direct TFT compatibility, color conversion to RGB565 is managed internally.
+.	Touch UI and Button Mapping
+Four primary buttons are mapped on the touchscreen interface: one to access the filter menu, which opens an overlay selection; another to capture the frame; a gallery button, which loads stored photographs kept on the flask server; and a button to turn the LED flash on or off.
+Touch input is handled using the XPT2046_Touchscreen library. Raw analog readings are remapped to screen coordinates:
+<code>
+int x = map(p.y, 200, 3800, 0, SCREEN_W);
+int y = map(p.x, 240, 3700, 0, SCREEN_H);
+</code>
+The screen is divided into interaction zones based on button boundaries:
+^ No ^ Area      ^ Action       ^
+| 1 | y > 200    | Bottom UI buttons (4 zones)    |
+| 2 | x < 80 && Y < 80    | Filter menu (4 filters)    |
+| 3 | galleryMode && x < 60 or x > 180    | Prev/Next Buttons     |
+{{ :iothings:proiecte:2025sric:img_2426.jpeg?200 |}}
+.	Real Time Filter Processing
+Before rendering, frames can be processed in software to apply effects, implemented by iterating over each pixel's color components:
+  * Grayscale gray = (r + g + b) / 3
+  * Sepia: Applies weighted RGB transformation for warm tone
+  * Invert r = 255 -r
+<code>
+if (menuVisible && x < 80 && y < 80) {
+    if (y < 20) currentEffect = "None";
+    else if (y < 40) currentEffect = "Grayscale";
+    else if (y < 60) currentEffect = "Sepia";
+    else if (y < 80) currentEffect = "Invert";
+    menuVisible = false;
+    needsUIRedraw = true;
+    return;
+  }
+</code>
+This transformation is embedded within the jpegDrawCallback():
+<code>
+bitmap[i] = ((r & 0xF8) << 8) | ((g & 0xFC) << 3) | (b >> 3);
+</code>
+.	Snapshot and Gallery Handling
+<code>
+void captureSnapshot() {
+  HTTPClient http;
+  http.begin("http://" + String(flask_ip) + ":5000/save_snapshot");
+  http.POST("");
+  http.end();
+}
+void fetchGalleryList() {
+  HTTPClient http;
+  http.begin("http://" + String(flask_ip) + ":5000/gallery_list");
+  int res = http.GET();
+  if (res == 200) {
+    String payload = http.getString();
+    DynamicJsonDocument doc(2048);
+    deserializeJson(doc, payload);
+    gallery_count = 0;
+    for (JsonVariant v : doc.as<JsonArray>()) {
+      gallery_list[gallery_count++] = v.as<String>();
+      if (gallery_count >= 50) break;
+    }
+  }
+  http.end();
+}
+void showGalleryImage() {
+  if (gallery_count == 0 || !jpgBuf) return;
+  String filename = gallery_list[current_gallery_index];
+  String url = "http://" + String(flask_ip) + ":5000/snapshots/" + filename;
+  HTTPClient http;
+  http.begin(url);
+  int httpCode = http.GET();
+  if (httpCode == 200) {
+    WiFiClient* stream = http.getStreamPtr();
+    int pos = 0;
+    while (stream->connected() && stream->available() && pos < 60 * 1024) {
+      int c = stream->read();
+      if (c < 0) break;
+      jpgBuf[pos++] = (uint8_t)c;
+    }
+    if (pos > 1000) {
+      TJpgDec.drawJpg(0, 0, jpgBuf, pos);
+      if (needsUIRedraw) drawUI();
+    }
+  }
+  http.end();
+}
+</code>
+Triggers the Flask server to save the current last_frame as a .jpg file, and the gallery images are fetched by file naming using /snapshots/name and rendered one by one, the user can access all of the photos by tapping the edge of the screen to change the current photo displayed.
+==== Final Product ====
+After successfully uploading the ESP32 firmware and setting up the Python server, the final project should appear as follows:
+ESP32 Display:
+{{ :iothings:proiecte:2025sric:img_2494.jpeg?400 |}}
+Web Server
+Home Page:
+{{ :iothings:proiecte:2025sric:livecamera.png?400 |}}
+Gallery:
+{{ :iothings:proiecte:2025sric:gallery.png?400 |}}
+==== Issues and Solutions ====
+Numerous difficulties were faced throughout the project's development, notable among them being those pertaining to memory management, touchscreen input accuracy, real-time streaming performance, and hardware-specific constraints.
+Achieving seamless MJPEG video streaming from the ESP32-CAM to the ESP32 display board was one of the initial challenges.  Image tearing and some dropped frames were noted, particularly when frame boundaries were not accurately identified.  In order to fix this, JPEG start and end markers (0xFFD8 and 0xFFD9) were carefully parsed, and any incomplete frames were discarded.  Reliability was further enhanced by raising the JPEG buffer to 60 KB.
+Real-time JPEG image decoding and rendering while maintaining touchscreen UI responsiveness was another formidable obstacle.  Because of its effectiveness in decoding JPEGs straight to the TFT display with minimal memory overhead, the TJpg_Decoder library was chosen, enabling real-time rendering at 10–30 frames per second, depending on network speed.
+Using the XPT2046 controller to integrate the touchscreen also needed additional work.  A second SPIClass had to be manually constructed and the coordinates had to be adjusted to match the screen's landscape orientation because the touch controller was interfaced via the HSPI bus.  In order to prevent unintentional tapping, extra attention was made to debounce inputs and segregate UI areas from video rendering.
+The ESP32-2432S028R "cheap yellow display" board's limited documentation and patchy support constituted a significant development obstacle. Important details such as touchscreen wiring, lighting control, and SPI pin mappings were not easily accessible in official sources. Reverse engineering, trial-and-error testing, and modification from community forum examples were used to achieve a large portion of the integration.
+Timeouts or incomplete answers were occasionally the outcome of HTTP communication with the Flask backend. This was fixed by decoupling lengthy operations (such as uploading a snapshot) from the user interface loop and encapsulating all HTTP requests with error checking and retry logic.
+Despite these issues, all major system components now function as intended. The lessons learned highlight the importance of resource management, firmware modularity, and planning for hardware inconsistencies during embedded system development.
+==== Conclusions ====
+This project uses two ESP32 boards—the ESP32-CAM for video capture and the ESP32-2432S028R for display—to demonstrate a real-time wireless image streaming system.  MJPEG streaming, realtime picture filtering, snapshots, and a touch-controlled gallery are all made possible by a specially designed Flask server.  Despite the hardware limitations of the ESP32, we were able to create an interactive, lightweight GUI by integrating Wi-Fi communication with libraries such as TJpg_Decoder, TFT_eSPI, and XPT2046_Touchscreen.  The project demonstrates how responsive IoT camera interfaces may be made with inexpensive microcontrollers and open-source technologies, potentially leading to applications for handheld cameras, monitoring, and surveillance.
+==== References ====
+[1] [[https://randomnerdtutorials.com/esp32-cam-ai-thinker-pinout/|ESP32-CAM AI Thinker datasheet and pinout]] \\
+[2] [[https://www.espressif.com/sites/default/files/documentation/esp32-wroom-32_datasheet_en.pdf|ESP32-WROOM-32 datasheet]] \\
+[3] [[https://cdn-shop.adafruit.com/product-files/4313/ST7789V.pdf|ST7789 TFT Controller Datasheet]] \\
+[4] [[https://github.com/Bodmer/TJpg_Decoder|TJpg_Decoder GitHub Repository]] \\
+[5] [[https://github.com/Bodmer/TFT_eSPI|TFT_eSPI Library Documentation]] \\
+[6] [[https://github.com/PaulStoffregen/XPT2046_Touchscreen|XPT2046_Touchscreen Library]] \\
+[7] [[https://arduinojson.org/|ArduinoJson Library Documentation]] \\
+[8] [[https://www.aliexpress.com/item/1005006861659956.html?spm=a2g0o.productlist.main.3.41c0f14e7uMYWE&algo_pvid=7d1eca47-6ef4-48c2-b752-471d25d72f64&algo_exp_id=7d1eca47-6ef4-48c2-b752-471d25d72f64-2&pdp_ext_f=%7B%22order%22%3A%224412%22%2C%22eval%22%3A%221%22%2C%22orig_sl_item_id%22%3A%221005006861659956%22%2C%22orig_item_id%22%3A%221005006160645147%22%7D&pdp_npi=4%40dis%21RON%21134.75%2164.68%21%21%21212.11%21101.81%21%402103956a17481862378891445e6b7d%2112000038545687790%21sea%21RO%213021543614%21X&curPageLogUid=AAs0NUd5VU7V&utparam-url=scene%3Asearch%7Cquery_from%3A#nav-specification/|ESP32 Display Board]]\\
+[9] [[https://www.aliexpress.com/item/1005008044751423.html?spm=a2g0o.productlist.main.1.5f9358f6QraLUr&algo_pvid=032894a5-d8b6-4490-831b-546059400468&algo_exp_id=032894a5-d8b6-4490-831b-546059400468-0&pdp_ext_f=%7B%22order%22%3A%22264%22%2C%22eval%22%3A%221%22%7D&pdp_npi=4%40dis%21RON%218.13%218.13%21%21%2112.80%2112.80%21%402103868817481861203547566e6e7b%2112000043405502723%21sea%21RO%213021543614%21X&curPageLogUid=R8aoEdwprt9Y&utparam-url=scene%3Asearch%7Cquery_from%3A#nav-specification|ESP32-CAM Module]]