Implementing gesture control on Raspberry Pi

Hardware

    1 Laptop, 1 Raspberry PI 2, 1 Web-camera

Dependencies

     1. Install conda (downloads)

     2. Get code from repository

git clone https://github.com/jackersson/hand_mouse_control.git
cd hand_mouse_control

     3.  Create conda environment (prepared environment in repository folder environment.yml)

conda env create -f environment.yml
source activate tutor # environment name - ‘tutor’

 Required libraries: python3, OpenCV, numpy, pyautogui

 Additional: Learn how to manage conda environments.

      4. Install Visual Studio Code for code editing.

Launch Guide

      1. Start program

python main.py -c 0
# where ‘-c 0’ defines camera index (if only one camera plugged to Raspberry PI use 0)

Note: to list connected cameras use

ls -ltrh /dev/video*

     2. Hotkeys:

•   press ‘a’ - start/stop mouse control by hand
•   press ‘w’ - draw blind zones (transparent green colored zones) where hand can’t be placed
•   press ‘h’ - show/hide green rectangles for hand extraction
•   press ‘s’ - update intensity threshold for color hand extraction
•   press ‘q’ - quit program

  After you launched application, you are going to see two windows - ‘Hand mouse control’ with live video from webcamera and ‘Colored mask’ - with extracted skin segments from camera stream.

•     Place your hand on green rectangles and press ‘s’ until image in window (‘Hand mouse control’) show you only your extracted hand. Better extraction - better quality of hand movement.
•     Press ‘h’ to hide green rectangles
•     Press ‘a’ to start mouse movements with your hand.
•     Move your left hand and watch at mouse. Cursor position should be updated when you hand moves. Remember gestures to control application.

•    Press ‘w’ to see blind zones

Here are some demos

     Simple mouse movement with gesture recognition

     Painting, Drag&Drop with gesture recognition

Conclusion     

    Hope this project will inspire you to modify this code and make more advanced system. This could be also extremely useful in Augmented Reality applications. So looking forward to your suggestions and possible collaboration.