In our first year of uni, we built a real-time music note reader using an FPGA and a camera. It can read custom sheet music and play it in real-time. The pitch is also adjustable using a blue marker placed anywhere in front of the camera.
Switching out the sheet music while it was playing a piece, and hearing the notes change to the new song was magical.
** Design
The three main parts of the project were
- offline automatic music sheet conversion
- real time note reading
- real time music playing while showing the current note being played
*** Music Sheet Conversion
The custom music sheet is a grid which represents different notes that are to be played.
Figure 1: Custom music sheet as a grid.
The notes are encoded in binary in each column using 5 bits which allows for 32 different pitches. The grid supports 32 different notes that are played sequentially.
The grid is automatically converted by a program we wrote using OpenCV called NoteReader. The main method used to detect the stave and the individual notes for simple sheet music was to generate a histogram of intensities vertically and horizontally. Below is an example of this process using twinkle twinkle little star as an example.
Figure 2: Horizontal histogram.
Figure 3: Vertical histogram.
The maximum was then used to as a threshold value to determine the individual notes and where the stave lines and notes were located.
Figure 4: Notes detected and turned grey
The grid can then be generated by encoding the pitch of the note as a 5 bit binary number and drawing the grid.
One problem that was encountered during the generation was that the detection algorithm only worked on one line at a time, and threw away the rest of the sheet music. In addition to that, it was hard to test the grid generation and the note detection at the same time. We solved this by first generating a file with all the notes present, then uses that file to generate the grid. This allowed the grid generation and note detection to be completely separated and independent of each other, making them much easier to test as well.
*** Real-time Note Reading
The purpose of the custom grid is to allow the FPGA to read the notes and then play them in real time. The grid is designed so that the orientation can be picked up by the FPGA easily using two red markers in the top two corners. Now that the FPGA has the orientation and size of the grid, it can detect all the squares that define the notes. It goes through the grid columns sequentially and reads the notes as 5 bits.
*** Real-time Note Playing
Once a note is detected from the grid, the 5 bit number has to be converted to a frequency so that the note can be played. The frequencies are stored in a look-up table in the FPGA, which can be indexed using the 5 bit number.
A wave generator then outputs a square wave with the required frequency through one of the GPIO pins on the FPGA board. Using a low-pass filter and an amplifier, the square wave is transformed into a continuous sinusoidal wave. This wave is passed to a speaker so that the tone is heard.