How to distinguish apples and pears with Raspberry Pi

Hardware

1 Laptop, 1 Raspberry PI 2, 1 Web-camera, 1 Apple, 1 Pear

Dependencies

1. Follow guide  to install - python, pip, virtual environment, OpenCV.
2. Install Jupyter notebook or Visual Studio Code for code editing.
3. Install Tensorflow , Darkflow following step-by-step guides.

Getting process started

All steps are coded in Fruit.ipynb, so you can use it to pass through all steps on your own.

 

Step 1. Collect data

Download prepared data here or follow next steps to do it by yourself:

1. Checkout Imagenet website
2. Fill the search field with ‘apple’. Press ‘Search’ button.
3. Check search result. You can find a lot of categories connected to apples. Look at definition of each category and choose one that fits best for your needs. For our task category ‘Synset:apple’ fits best, so go in.
4. Look at images to be sure that selected object is apple.
5. Loading images isn’t so easy. We need to go deeper and take a look at Imagenet API.

Step 2. Data preprocessing

• Check that data is valid
• Check annotations data format.

Pascal Voc annotation format is simply describes all information about image: size, objects and its pose, bounding boxes, names. Change next information in files:

• Add extension to filenames n07739125_12  -> n07739125_12.jpg  

• Change object names from class wnid to real names  n07739125 -> apple 

Split to train/test. Common practice is to use 80% of data on training and 20% of data on testing.

After all manipulations with data we got folder with train/test data corresponding to each class with images and annotations in it.

Step 3. Training model

YOLO is an algorithm chosen for object detection. Get more information in paper or web-site. Implementation of this architecture from scratch could be challenging, so we will use darkflow code.

1. Clone repository

git clone https://github.com/thtrieu/darkflow.git

      2. Go to project folder

cd darkflow

      3. Install darkflow

pip install .

 Now let’s create model configuration.

 

1. Edit labels.txt file. Substitute existing labels with new ones : apple, pear

 

Note: labels name in labels.txt should match with object name in annotations files, because darkflow is looking directly for specified object names.

 

      2.  Design net

Go to ../darkflow/cfg/

Make a new copy of tiny-yolo.cfg and rename it to yolo_fruits.cfg

Open yolo_fruits.cfg and make simple changes: 

line 110
[convolutional]
filters=35 #(2 + 4 + 1) * 5, 

where  2 - num_classes (pear/apple)
4 - x, y, w, h (parametrs of bounding box)
1 - confidence of bounding box
5 - num_anchors ( look at region section )

line 120
[region]
classes=2
num=5

3. Calculate anchors

line 118
[region]
anchors=7.19,7.25, 5.25,2.59, 6.06,7.34, 9.09,5.47, 9.87,4.5625

What anchors represents?

Anchors are an averaged width and height of all objects bounding boxes. It calculates with k-Means algorithm. For example, in our case k-Means algorithm found 5 closest bounding boxes groups and calculates centroids for each group. That is why we have 10 numbers (5(num_anchors) * 2(w, h)).

Original anchors are the next values: [230 232], [168  83], [194 235], [291 175], [316 146]:

 

So why anchors represented by small numbers with floating point instead of original values?

Due to YOLO algorithm we need to scale anchors with respect to the feature map in neural network architecture. For example in YOLO-tiny input image has size 416x416 and that last feature map size is 13x13. Scale ratio is 13/416. We mutiply each value of original values with scale ratio: [230, 232] * (13/416)=[7.19, 7.25]. That’s right - we got same numbers.

 

4. Download weights (here) and place it to darkflow/bin

tiny-yolo-voc.weights corresponds to tiny-yolo.cfg

yolo.weights corresponds to yolo.cfg

 

5. Follow instructions on how you can run a model

flow --model cfg/yolo-fruits.cfg --load bin/yolo-tiny.weights --train --gpu 1.0 --annotation ../train/annotations --dataset ../train/images

 --gpu 1.0 - shows how much performance of GPU is going to be used. If there is no GPU in your PC remove this flag (But be ready to wait a while when you training on CPU). If you paused or stopped training process you can easily continue with the same command but with flag:  

--load -1 - takes the latest checkpoint from ../darkflow/ckpt/

 I suggest you to look at Tensorboard while training model. Darkflow automatically saves summaries to ../darkflow/summary/ folder. Simply run:

tensorboard --logdir=../darkflow/summary --port=6001

 Go to your browser, print https://localhost:6001 and checkout loss function. Loss graph updates in real-time so you can easily check when there are any mistakes. If loss value do not become smaller then network does not learn anything.

 

Possible problems:

‘None type’ has no attribute ‘shape’

This means that there is no image according to specified filename in annotations. Check that image filenames in annotations corresponds to filenames in images folder.

OUT_OF_MEMORY (when using GPU)

When using GPU all data has to be loaded to GPU memory buffer. Sometimes there is not enough space to place all data. As possible solution you can decrease batch size (default=16) using flag:

--batch ...

Evaluate model

Simplest way to evaluate model is to use darkflow command line tool:

flow --imgdir ../test/images --model cfg/yolo-fruits.cfg --load -1 --threshold 0.1 --gpu 1.0

Notice that we set threshold value to 0.1 (default value - 0.6), because we want to see all detections from our model. But play with threshold and find best value.

Export model

Once you satisfied with the accuracy of your model you can export it and use in real application. Just type the next command:

flow --model cfg/yolo-fruits.cfg --load -1 --savepb

You can find your model yolo_fruits.pb and .meta in ../darkflow/built_graph/ folder. For detailed explanation look at section ‘Save the built graph to a protobuf file (.pb)’.

Step 4. Deploy model

Let prepare Raspberry PI.
Install OS. Check out this guide.

Note: I’ve tested model deployment on Raspbian OS.

Install Tensorflow. Go here.

Note: Look at section “Installing from Pip”  and do exactly the same steps. Install for python 3.

Install OpenCV. Go here.

Install Darkflow as in previous steps.

Note: To install Cython use:

sudo pip3 install Cython

Now you have everything ready to use and test your model. (Code)

1. Copy yolo_fruits.pb and yolo_fruits.meta to Raspberry Pi.

2. Run command in darkflow folder:

flow --pbLoad yolo_fruits.pb --metaLoad yolo_fruits.meta demo camera

 Demos:

Here is a demo how it works on Raspberry PI (slow)

Demo on Intel i5

Notes:

• Forward pass neural networks on CPU is time consuming so be ready to get from <1 FPS. I’ve tested on Raspberry PI 2 and got 0.3 FPS, but it could be slight faster on Raspberry PI 3.

• Using model on smartphone would be much faster as it have GPU. 

Additional features:

• Change color of bounding boxes:

  • Go to .meta file and find ‘colors’ field. Change colors in RGB format

• Change labels names:

  • Go to .meta file and find labels field. Change colors names

• Change threshold to identify object with more confidence.