ANPR/ALPR on the UP Embedded Vision Starter kit

I've already developed a reliable algorithm to read UK number plates, it's coded in python3 and uses openCV2 to do the heavy lifting. I have a working system based on an UPSquared board linked to an IP camera, but can it be made to work on an UP with a Basler usb camera?

As a hardware engineer and part-time python programmer I was not prepared to start coding in C or C++, so for the starter kit to be of any use I needed to get pypylon working. Pypylon is Basler's python wrapper for their pylon C++ camera API. To avoid a steep C++ learning curve and have any chance of success I had to get python3, openCV and pypylon working on the UP board:

From zero to Pypylon...

Download the ubilinux iso image from the up-community web site

Note ubuntu was not available at time of writing

Download rufus image-writer software

Use rufus to wite the iso image to a usb stick in UEFI / DD mode

Boot the up-board with the usb stick

Allow the os installation to proceed

Up board powers-down after installation

Once powered-down, remove the usb stick & re-start with the little white button

Note default pwd is ubilinux, change as required

$sudo apt-get update
$sudo apt-get upgrade

Reboot as required

Install Basler's pylon s/w direct from their web site:

Note goo.gl url in UP's paper documentation is broken

Download the tar ball and un-pack it

  tar -xzf pylon-5.1.0.12682-linux-x86_64.tar.gz
  cd pylon-5.1.0.12682-x86_64
 Follow the instructions in the README & INSTALL files
  sudo tar -C /opt -xzf pylonSDK-5.1.0.12682-linux-x86_64.tar.gz
  ./setup-usb.sh

Follow the instructions in the README & INSTALL files

I needed a reboot at this stage

Now plug in the camera

Run Basler's PylonViewerApp (find it in /opt/pylon5/bin)

Camera should appear under the USB devices list

Double click to connect to it

Grab a frame to test the camera

Use continuous grab mode to set-up the lens's focus

Close the application once test is complete

$sudo apt-get install python3-pip # get python3's pip package installer
$sudo pip3 install opencv-python # also installs numpy

At github.com/basler/pypylon...

From the releases page download pypylon wheel for py3.5 on x86-64 linux

In my case filename was pypylon-1.3.1-cp35-cp35m-linux_x86_64.whl

Also, from samples download or copy opencv.py

$sudo pip3 install pypylon-1.3.1-cp35-cp35m-linux_x86_64.whl
$sudo apt-get install idle3
$idle3  # test the pypylon install

file->open opencv.py

F5 to run

The camera's live image appears on-screen, success!

Testing ANPR

OK, perfect, I have an openCV python3 development platform with a live image source from the Basler dart usb camera. Now to test the UKRegOCR number-plate identification & ocr code - this was developed earlier for an UPSquared board, but how well will it work on the lower-performance UP board?

Starting with the opencv.py code from github I added some lines to make it easier to work on as I like to edit on my Windows PC. Next I downloaded the UKRegOCR package from: http://www.marvellconsultants.co.uk/ANPR/UKRegOCR.zip and unpacked the files to the same directory. I simply added an import for the UKRegOCR module to opencv.py and a call to the lookForPlate() function on each frame grabbed from the camera. This function uses a well-trained Haar Cascade classifier to spot and locate number-plate-like objects in an image. Doing this slowed the frame-rate down to about 1 per second - much slower than on an UPSquared. Calling the ReadPlate() function to do the actual OCR process (on just the portion of the frame that has a plate in it) as expected slowed things down further to around 5-6 seconds per frame, about three times slower than an UPSquared but still faster than a RPi3. Oh well, that will maybe limit the range of applications that the UP platform could service but there's always the UPSquared for those jobs - the UP's advantage is its low cost.

See BaslCam1.py in Code section

After checking processor usage during the operation of the above code it became clear that the OCR function was only making use of one processor core. Maybe if I run the OCR code in separate threads then perhaps I can carry on grabbing frames at a steady rate while OCRing selected ones in the background. Ideally you'd read two or three images of the same plate as it passes to obtain a higher confidence in the read. Without threading, multiple 6 second reads would risk missing some vehicles during the time spent reading others.

Sure enough running 3 concurrent OCR threads brings processor usage up to 80-100%, and allows the grab and plate-classifier loop to cycle at <2 seconds per frame. Unfortunately the threaded code suffers from very occasional untraceable non-python 'std::out_of_range' RTEs. Reducing the OCR threads to a max of two seems to make this go away and results in an average frame rate of 40/min with a plate-read rate of 14/min. So usable in applications encountering up to about 5vehicles/min. This approach could make some applications feasible on the UP platform, and be of use for widening the range of applications viable on UPSquared. Running the same script on an UPSquared Pentium N4200 board the average frame rate was 36/min but with a plate-read rate of 29/min.

See BaslCam_threaded.py in Code section

Back on the original UP board, I wasted much time trying to nail down the RTE without success. If anyone can help I'd be very grateful.

AI

So next I wanted to try a Movidius AI compute-stick with the UP board to see how well Intel's secutity barrier camera demo works on UK sample images. However this requires changing the os from UbiLinux to Ubuntu 16.04: Follow the steps at: https://wiki.up-community.org/Ubuntu I removed a load of bloat (games/libre-office/etc) after the first ubuntu boot. Allow a good couple of hours and note the Intel graphics driver install has been obsoleted. Now install pip3, pylon, open-cv, pypylon, idle, etc as above. Once that's all working again go to developer.movidius.com/start and download openVINO via the link - I selected the latest 2018.4.420 full-package option then followed these [laborious] instructions:

software.intel.com/en-us/articles/OpenVINO-install-Linux

BUT added -d MYRIAD to the ./demo-squeezenet-download-convert-run.sh command line. I got 'C++ internal compiler error's at the above Image Classification demo step. It seems C++ needs at least 6GB of ram storage (so great error message guys!), I set up a 6G swapfile and re-started the demo script, which took hours to complete, but ran without error.

Now to try the security-barrier-camera-demo - this sample code uses three neural nets: a vehicle recogniser, a vehicle attributes recogniser (for identifying where the number-plate is) and finally a number-plate reader. It works well on Intel's one sample image (car_1.bmp) but failed, to varying degrees, on every single one of my 18 sample images! See SAMPLES.zip in FILES. The MCL reader accurately reads 16 of those 18. So much for AI!

So, in summary, the UP (or UPSquared) is a great little platform for ANPR and by all means check out AI techniques (if you've the spare time) but if you want to reliably read number plates in the real world, for now, do it the old-fashioned way.

Further work

For optimal results in day and night it's necessary to operate in infra red, with an illuminator. To save power in the illuminator it should be tied to the exposure time of the camera which should itself employ a global-shutter type sensor. I've now started work on a Raspberry pi IR camera as proof of concept and initial results are promising, despite the sub-optimal rolling shutter. Next step will be to investigate a more specialised IR-optimised Basler camera with global shutter.