Who made the mess?

Office kitchens are not always the nicest places. It only takes one person to leave a cup in the sink and then civilisation as we know it teeters on the edge of destruction. This solution uses computer vision to identify objects and likely culprits. It does not measure out judgement on the guilty (for now)

Documentation

This project is broken down into a pair of streams:

  1. Identifying common kitchen items we might find in the sink; and
  2. Identifying people.

These two tasks are similar, but different.

Finding a mess

Mess detection uses YOLO (you only look once) developed by Joseph Redmon and Ali Farhadi. These folks are far smarter than I will ever be and I not only acknowledge their work, but I want to thank them for making it public.

In short, we take an image every 30 seconds from a Basler Dart Camera (supplied as part of the evaluation kit), and passes this through YOLO using the standard training model. If there is a probablility greater than 80% that a domestic item (cup, plate, glass, etc) is in the field of view, then we have a "mess" event.

pypylon handles the control of the camera, then passes off a standard image to OpenCV and YOLO for processing.

Because YOLO contains dozens of objects which could find their way into the sink, it's not practical to break it down with a switch statements. In order to lighten the load on the Up Board both in performance and storage, I offloaded this task to an Amazon Lambda function, via the API Gateway.

The use of an external function also allows the camera platform to be used for general identification of other types of objects as well. No code changes would be required on the main routines.

Finding the culprit

A second process is running on the Up-Board, which uses an additional camera to look at the area surrounding the kitchen. In this case, we are using OpenCV to identify faces of people and note the time they were in the area.

When a mess-event occurs, we look up to see who may have been in the area at the time. An external script could be called to name and shame the likely offenders.

Building

You will need either the Up Embedded Vision Starter Kit, or the components which are available separately.

First get your board up and running. I used a standard Ubuntu Desktop 16.04.10 LTS, and installed manually. If you remove the image display code, the application could also be run on Ubuntu Server. Make sure that you follow Up's documentation on getting the device-specific kernel installed.

All my code is based around Python 3.6. I have not tested it against Python 2, but there should be few modifications required.

I also added the Intel OpenVINO toolkit, because OpenCV is optimised there. The installation instructions are located at https://software.intel.com/en-us/articles/OpenVINO-Install-Linux . Note that registration is required to download.

The Basler Dart camera included with the kit does not integrate as a standard V4L or Webcam device, you need two more components:

Basler Pylon Suite - the software which drives the Basler cameras

Basler PyPylon - a Python interface into the Pylon software

By the way, there is no need to build pypylon for yourself, just use the release to match your operating system.

Finally, install numpy for Python3

You will need an Amazon Web Services (AWS) account for lambda, api and dynamodb functions. All of these services qualify for the free tier if you want to experiment. You will need to add you API endpoint into messfinder.py

When all this is done, you need to aim your camera at the sink and focus it carefully. This is the only real criticism I have of the Basler Dart camera. It is not easy to mount. If I had access to a 3D printer or a CNC milling machine I am confident I could create an elegant mounting system, but time was against me. I used a length of acrylic rod and a hot glue gun. Not pretty, but it got the job done.

Before running messfinder.py, load the Basler-recommended kernel settings by executing "tools/optimal-settings.sh"

The face detection routine needs to be trained. in the directories beneath "training-data", place photos of each face to be recognised (one directory per person) and run recogniser.py. This will create a series of "dat" files which are used by searcher.py to identify the face in an image.

Training can take quite some time, and may be better run on a more powerful machine. Then the dat files can be copied onto the UP Board for use. This improves the time it takes to get ready.

FaceFinder is copied from my FaceFinder project on bitbucket (https://bitbucket.org/chrismor/facefinder) . The full version can be found there, and may be more up-to-date.

Outstanding Issues

The biggest problem is that the system is still too slow. YOLO is brilliant, but the CPU on the UP Board is working at 100% on all cores, and all available memory is used up, so perhaps the 4GB model might be a better plan for continual observation. Although it is too late for this contest entry, I have started experimenting with using a Movidius Neural Compute Stick, and the results are looking promising. I will probably create another post in the future on the results of my work in this area.

@article{yolov3, title={YOLOv3: An Incremental Improvement}, author={Redmon, Joseph and Farhadi, Ali}, journal = {arXiv}, year={2018} }

Code

GitHub Repository

https://github.com/au-chrismor/messfinder

Commits

Christopher Moran
pushed 0b6eb38a2bc157e434abf6f50119291d7ff1f451
Fixed API Gateway URL
2018-11-23 03:17:48 UTC
Christopher Moran
pushed a570ca93a1604eb575f590ffa61d67ada7af87da
Updated license
2018-11-22 13:05:38 UTC
Christopher Moran
pushed 71baa4cd4d9e875923bb47b97fb86c59bbd8ec0b
Made getObject() more robust
2018-11-22 13:03:58 UTC
Christopher Moran
pushed 59cf4b4b469dd2dfa9b479679943fee3894bb120
API Gateway integration
2018-11-22 13:00:47 UTC
Chris Moran
pushed a03a6305ef44b73091a46b942a456956fd1a9127
Create README.md
2018-11-22 12:56:25 UTC
Christopher Moran
pushed bf61b8deb6507826026c53474434c7f2a93c964e
Script to set performance parameters
2018-11-22 12:53:08 UTC
Christopher Moran
pushed 3cc228273ae73cf4366d85eee030b936777789a2
Make messfinder.py executable
2018-11-20 10:12:17 UTC
Christopher Moran
pushed 7c1527eaf9457926bd5c3ebb8802124f115d2a34
Cleaning up facefinder
2018-11-20 03:58:08 UTC
Christopher Moran
pushed d5a060b2108943779b2f13fdd6e069a222aab2cd
Removed unneeded firstdetect
2018-11-20 03:30:00 UTC
Christopher Moran
pushed 49f88021599890606aaa77ada116760f5d72d440
Adding facefinder
2018-11-20 03:25:41 UTC
Chris Moran
pushed bd50c68deef7c5b813d529bc0ccc51daa64e5daa
Create README.md
2018-11-20 03:15:24 UTC
Christopher Moran
pushed 155a19510aa33a53eead52be3d3f3aa6ed5cfe9d
fix errant README
2018-11-20 03:13:17 UTC
Christopher Moran
pushed 9aaad8105ab97c39579792487d63f65d2c469366
Adding README files
2018-11-20 03:11:52 UTC
Christopher Moran
pushed b6dedba2e4bd672cf3422e7246bacaff38863543
Cloudformation for DynamoDB Table
2018-11-20 02:59:20 UTC
Christopher Moran
pushed cbc72a5016ecc91bcbade4efbb9a911dc1ac7bac
Check-In lambda function to decipher mess
2018-11-20 02:24:12 UTC
Christopher Moran
pushed 432efcfa835434ec775d7d5f9eb3f00186ce3467
Adding API hook for recognising
2018-11-20 02:10:41 UTC
Christopher Moran
pushed ab6a2206c2cef41c2bab538a427e8a101c9a6dc6
Add Slack Hook
2018-11-20 01:40:50 UTC
Christopher Moran
pushed 789b28b347d5f1a819392da775fe72c155328c75
Restore default camera settings
2018-11-20 01:28:36 UTC
Christopher Moran
pushed 2afc0705f8b3b1143573c2288bfc8042b580e9cf
Added camera settings
2018-11-19 23:56:45 UTC
Christopher Moran
pushed ddbfb904da074f9eff2d51b536bd389604a0bc8c
Add Slack Hook reference
2018-11-19 22:30:22 UTC
Christopher Moran
pushed 782b4b2e17ff6ddf2d62a20be0653122b493968b
Repo cleanup
2018-11-19 22:24:01 UTC
Chris Moran
pushed 540b5dd7a6d5b111f2f554b40a2fa24df8f17733
Create README.md
2018-11-19 14:08:32 UTC
Christopher Moran
pushed 8f509eaee5672341e1a464960325e98f62991665
Adjusted waitKey timeout
2018-11-19 13:09:27 UTC
Christopher Moran
pushed 331a68611c612ba2c826a24e82ce38bb73dff80b
Added YOLO integration
2018-11-19 13:07:24 UTC
Christopher Moran
pushed 9a975af853303b1703f66f262ef1c90b1180f73b
Adding combined app
2018-11-19 12:52:04 UTC
Christopher Moran
pushed e1793caedf7ece65c2846ce0d3186c8adbd2db9f
Adding reference output
2018-11-19 12:08:25 UTC
Chris Moran
pushed 219dbdff55b20271e8358965142d854c68b2f4cb
Update README.MD
2018-11-19 10:42:48 UTC
Christopher Moran
pushed 983b75e16462d21d27d65c8b595f56cab99c5530
Adding First Detection Algorithm
2018-11-19 10:41:34 UTC
Christopher Moran
pushed c3e3412bcdb4f2f40b65600e52f8fb444eb35c0e
Explain that base training files are missing
2018-11-19 05:05:13 UTC
Christopher Moran
pushed d22ad6ad237f03cef877c5ab5adda55b235a3c9b
Initial Check-In
2018-11-19 05:02:14 UTC
Files
Title Description Format
Detection Screen during development Dirty sink for illustration purposes only :-) jpg
All the YOLO models and configuration zip
Bill of materials
Name Article number Link Quantity Unit Price
UP Embedded Vision Started Kit RE-UP-PACK-VISION-002 up-shop.org 1 349.00 $
Total 349.00 $
Tasks
Task Owner Creation Date
Improve the performance
au_chrismor
2018-11-19
Info

Project State

Public Project

Licences

Software Licence: BSD-3-Clause
Hardware Licence: Project has no hardware

Project Tags

Admins

au_chrismor

Members

Does this project pique your interest?

Login or register to join or follow this project.

Comments
Back to top

Your comments, please!

Want to comment this ... Show more