This project consists of building a robotic hand with 5 fingers, each actuated by a servo motor, placed over 5 piano keys (DO, RE, MI, FA, SOL). The system can detect musical notes from different sources and reproduce them physically on the keys.

• What it does: The hand listens to or reads music and plays it back on an electronic keyboard by pressing the keys with servo-driven fingers. It supports three operating modes: playing a hardcoded melody from memory, listening to a short melody through a microphone and reproducing it, or reading a .wav file from an SD card and playing the detected notes.
• Purpose: The goal is to explore the integration of audio signal processing (FFT-based note detection) with real-time motor control on an embedded platform, while also covering a wide range of communication protocols studied in the PM course.
• Starting idea: The idea came from wanting to build something that bridges digital signal processing and physical actuation — a robot that can actually “hear” music and play it back.
• Why it is useful: It demonstrates a practical application of FFT on a microcontroller, combining audio analysis with mechanical output.

System Block Diagram: The system is centered around an ESP32 DevKit C microcontroller that manages three main tasks: user interaction (buttons + LCD display), audio input/analysis (microphone ADC or SD card SPI), and mechanical output (5 servo motors via PWM).

The user selects between the three operating modes using two physical buttons, and the current mode and status are shown on an LCD 1602 display. Depending on the selected mode:

1. Mode 1 — Hardcoded Melody: The firmware contains a pre-stored sequence of notes with timing information. The ESP32 iterates through the array and drives the corresponding servo for each note.
2. Mode 2 — Microphone Input: The MAX4466 electret microphone module captures ambient audio. The analog signal is sampled via the ESP32's internal ADC, then processed through a 2048-point FFT pipeline to detect the dominant frequency. The detected frequency is matched to the closest musical note using a logarithmic (semitone-based) chromatic scale comparison. A histogram of all detected notes is built automatically, and the top 5 most frequent notes are mapped to the 5 fingers.
3. Mode 3 — WAV File from SD Card: A .wav file (8-bit mono PCM) stored on a Micro SD card is read over SPI. The same FFT pipeline from Mode 2 is applied to the audio data to extract notes, which are then played on the servos.

Bill of Materials:

Export to PDF