This week was the perfect week to accomplish a project that I had at the back of my head since a long time. I wanted to develop a workflow for live image processing that allows to interact with an experiment or a physical system (here a puzzle). Virtual reality can really change the way we interact with things and how we use our intuition. As I already knew about image processing and Matlab, I started from there and learned a few new tools such as Python, OpenCV, Raspberry Pi, the pi camera and live image processing. That was a great week that will be useful for my final project.
I had this project at the back of my head for a long time, this assignment was the opportunity to try and make it. A few years ago, Axel, who is a technician in my lab, had found and made a weird puzzle: a Fibonacci Spiral Jigsaw Puzzle (see image below). This puzzle is really difficult to make. The pieces have all the same shape but have all slithly different sizes. I struggled for about half an hour to make it, placing and removing pieces until all the pieces fit ! Fun but really difficult !
I was wondering if we could get a little help from technology to help our intuition that would help us to make it. What if we could set a camera on top of the puzzle, that would measure properties of each puzzle piece and indicate in live information that would be helpful to make the puzzle.
First, I used photoshop to make the following 8bits image of the puzzle.
Then, I imported the black and white image in Matlab to analyze the properties of each puzzle piece and label them. I used 2 important functions: regionprops and bwpropfilt. regionprops allow to extract the white pieces and measure properties on each of them. I then used bwpropfilt to extract the pieces corresponding to a certain size and add a color and a number to it.
Here is what the matlab code does
Here is the final image. Dark blue are the smallest pieces. Dark red are the largest. There is a number at the center of each piece, 1 being the smallest.
After having downloaded the file from here, I cut it in plywood at the laser cutter. Prior cutting, I put some tapes to protect the wood from burned resin traces. Tape that I will remove once the puzzle is cut.
Here is a video of the puzzle being cut by the laser cutter.
Here is the assembled puzzle
So hard.
I set all the puzzle pieces in a light tent on a blue textile and I took an image using a camera.
I opened the image in imageJ and made a color threshold to extract the puzzle pieces.
From this color threshold, I've made a black and white image
Then I ran the Matlab code shown above and got this image as a result.
Then I displayed the labeled image in front of the real puzzle pieces. And I sorted the pieces by size, starting by the smallest one. This was easy.
Once sorted by size, the puzzle is fun and easy to assemble !
Done
For this part I explored the use of a camera input device. I chose a pi camera and a Raspberry Pi model 2. As a language to interface it, I chose Python along with the openCV library and the Picamera.camera module.
Here is the setup: the light tent, the puzzle pieces, the pi camera, the raspberry connected to a screen, a keyboard and a mouse.
Here I interfaced the camera and got a live feed.
Then, I wrote a code based on this tutorial XXX. I struggled a lot with the colorspace and the camera settings to have a stable a exploitable shape detection.
Here are two tutorials that really helped me to make the python code presented here below:Here is what the python code I developed does :
Here is a screen shot of the running python code on the Raspberry PI
Here is a video of the live shape detection.
For this part I used the OpenCV function findContours and manage to make a live detection. Different colors corresponds to different piece size.
I've continued the python code as follows :
Here is a screen shot of the running python code on the Raspberry PI
Here is a video showing the live recognition of piece sizes.