Real-time Image Processing on NVIDIA GPU with Basler pylon and Fastvideo

This project provides a prototype that allows for real-time image processing on Nvidia GPUs. The prototype is based on the Basler pylon Camera Software Suite and Fastvideo SDK and enables testing of sophisticated image processing algorithms with Basler USB3 Vision cameras on Windows 7/8.1/10 64bit.

Documentation

Challenges:

Basler’s machine vision cameras use an internal FPGA for certain image processing tasks, e.g., debayering, automatic exposure time, automatic gain, automatic white balance, gamma, noise reduction, sharpness enhancement, vignetting correction, etc. This means that real-time image processing takes place directly on the camera without causing any delays or additional CPU load on the host computer. However, there are a number of applications with requirements exceeding the feature capabilities offered by Basler’s machine vision cameras. For example, customers may need additional features not included in the respective camera firmware, e.g., rotation, flat-field correction, etc., or extended ranges of existing camera features, e.g., more binning factors. Another aspect is that using the camera’s FPGA to perform certain image processing algorithms in some cases may adversely affect camera performance or frame rate. For example, performing debayering on the camera’s FPGA may require more bandwidth to transfer the image data than the given interface supports, e.g., Gigabit Ethernet (115 MByte/s) or USB 3.0 (380 MByte/s). This may lead to reduction in the camera’s frame rate. To overcome the challenges detailed above, performing real-time image processing on a GPU is worth considering as an alternative approach.

What Does the Prototype Support?

The prototype created in this project supports and is based on the following:

  • Basler USB3 Vision cameras

  • Operating systems: Windows 7, 8.1, 10, 64-bit

  • NVIDIA GPUs: 6xx series and above (computing capability ≥ 3.0). The prototype has been tested with:

    • NVIDIA GeForce GT 1030, 2 GB
    • NVIDIA GeForce GTX 1050, 4 GB
  • CUDA Toolkit 10.2

  • Basler pylon Camera Software Suite 5.2 for Windows 64-bit:

    • Copy-deployed version of the pylon Viewer is included in the prototype.
    • pylon programming samples are included in the prototype.
    • For USB3 Vision drivers, a regular pylon 5.2 version has to be installed.
  • Fastvideo SDK for Windows 64-bit with watermark (included within the prototype).

Which Image Processing Algorithms Does the Prototype Offer?

  • Unpacking of raw image data, e.g., Mono12p or Bayer12p packed data
  • Dynamic defect pixel and hot-spot correction
  • Flat-field correction
  • Fixed-pattern noise subtraction
  • Binning: 2x2, 3x3, 4x4 (monochrome only; color and higher binning factors possible on request)
  • 1D LUT (lookup table)
  • White balance correction (color only)
  • Debayering: 5x5, 11x11, 23x23 (color only)
  • Base color correction (color only)
  • Denoising
  • Rotation: 90°, 180°, 270°, flip-flop (additional angles possible on request)
  • Cropping
  • Resizing, e.g., downscaling (upscaling possible on request).
  • Sharpness
  • Gamma correction
  • JPEG image encoding and saving
  • MJPEG video encoding and saving

Which Additional Features Are Possible on Request?

  • GigE Vision camera support
  • CXP camera support
  • Linux x86_64 and Linux ARM aarch64 support
  • J2K lossy and lossless encoding for 8–16 bit
  • H.264/H.265 encoding
  • Low-latency streaming
  • Interoperability with FFmpeg for output encoding and streaming
  • Live video output via OpenGL
  • Decompression for Basler ace 2 cameras
  • Additional rotation angles
  • Support of DCP profiles (Raw to RGB)
  • 3D LUT support (color grading)
  • HDR support
  • Perspective transforms (parallel edge distortion)
  • Undistortion via LCP profile (removal of barrel or pincushion effects)
  • Custom FFmpeg filters and codecs on GPU
  • Others

Performance & Test Results

The following table shows test results that clearly demonstrate that using GPU processing doesn’t reduce camera performance or frame rate. In the case of performing debayering on the GPU, the camera’s frame rate can even be increased.

enter image description here

Performance Issues

The following aspects typically affect the upload and download times possible with the GPU:

  • The PCIe generation itself
  • Number of available/active lanes
  • Other devices using the PCIe bus

Therefore, it’s advisable to test different scenarios in order to determine the PCIe performance.

Next Steps

Have you tested and enjoyed the image processing algorithms implemented in the prototype?

Would you like to use them within your own application without watermark?

Are you looking for even more functionality above what the current prototype offers?

Please refer to the official Fastvideo solution on GitHub (https://github.com/fastvideo/gpu-camera-sample), which provides other/additional functionality. Alternatively, simply get in touch with Fastvideo LLC (https://www.fastcompression.com/) directly.

Code

GitHub Repository

https://github.com/fastvideo/gpu-camera-sample

Commits

Fyodor
pushed c16ed86612d8864cc68c0f8e643e374c12f9a54a
Update README.md
2021-07-07 09:12:58 UTC
AlexVRom
pushed f516015cc85bd44013cc2e651aead2e4bd5aa7c9
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2021-07-05 05:39:14 UTC
AlexVRom
pushed 3790be0ad500aa7057c89e2651b356b91af0c1ef
Added some error handling
2021-07-05 05:32:46 UTC
AlexVRom
pushed 10345aa7262ea3541872011987576db4c85d7554
Fixed GenICam libs for Windows
2021-06-28 14:52:38 UTC
AlexVRom
pushed 94f01ff07860e921c68a44952a6d6353168da1ca
Fixed ffmpeg libs for Windows
2021-06-28 12:39:40 UTC
AlexVRom
pushed b844aad836fc71bab6c9ca31c50f6a35f71e107a
Fixed common_unix.pri
2021-06-28 11:42:45 UTC
AlexVRom
pushed 9d6c67b463f4637259018995adcbd2e0beaa151c
Fixed common.pri
2021-06-28 11:41:46 UTC
AlexVRom
pushed 862a7d7520ff7dadc4441822dccee9a3a5372538
Added GPUCameraSample.sh
2021-05-23 20:05:15 UTC
Fyodor
pushed 509f0b2ba458abc13396e597567df16533fbbe2d
Update README.md
2021-05-23 20:01:27 UTC
AlexVRom
pushed 04632d6dda97433041d3ba5316beb8fbb8343088
Merge pull request #8 from kolabit/master
2021-05-23 19:57:53 UTC
kolabit
pushed 7ed849b754df511dfa8592f88a8ebecd5e54f9c3
Camera Statistics panel added. Crash in GeniCamCamera::setParameter fixed. Minor improvements in CameraSetupWidget
2021-05-21 20:19:01 UTC
kolabit
pushed 81cae3fe06231d2b93c95359002e2662c6bb5267
Imperx cameras support via SDK on Linux done
2021-05-20 19:10:39 UTC
kolabit
pushed 54da3a311a2f35e48f65186409dd3c125fa8b4fa
Imperx cameras supported in Linux via GenICam
2021-05-20 18:54:40 UTC
AlexVRom
pushed e2279c31d47d7055685c25960c4f9e166421cb01
Update README.md
2021-05-19 07:13:56 UTC
Fyodor
pushed 127a10d08f017ddb113d38f7647ec1a73ad83805
Update README.md
2021-05-19 06:49:04 UTC
AlexVRom
pushed 12f9433b952f20117f569944f2c109bffe1c1fc0
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2021-05-19 06:15:10 UTC
AlexVRom
pushed 0185d67a0d28a4eaea442e74d65dc48f91bfa247
Fixed GenApi version back
2021-05-19 06:14:16 UTC
Fyodor
pushed 054696ae4277964480e13e86cc23a27a63ab23bf
Update README.md
2021-05-18 13:26:22 UTC
Fyodor
pushed a485b3d74a3fe65e8a91a52a81f1aea098d8a39f
Update README.md
2021-05-18 13:22:53 UTC
Fyodor
pushed 518443217eec294ee9fa89cda8d93b16453e226a
Update README.md
2021-05-18 13:22:14 UTC
AlexVRom
pushed e9f105209e51e6de8a100ea70356220889cc60ea
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2021-05-18 12:41:27 UTC
AlexVRom
pushed 5ca7d8026c10b0235d682a63483e66298d646d8b
Fixed Linux build issues
2021-05-18 12:40:17 UTC
Fyodor
pushed 87838b0cdad9c9b6f0c045daf549602b492a1381
Update README.md
2021-05-15 19:35:24 UTC
AlexVRom
pushed 45c3f7ad97d5d2c19375c0273c08037fd1b58897
Merge pull request #7 from kolabit/master
2021-05-15 18:57:28 UTC
Fyodor
pushed 249abf4e7b979192ffc0538ead451cff9cdbebe5
Update README.md
2021-05-15 16:15:38 UTC
AlexVRom
pushed 9ac2e251a77b1895ffd0c23d6388c7107e517891
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2021-05-12 12:22:09 UTC
AlexVRom
pushed ae220a62142b4783ca552aa70566f54976d7b062
Fixed CameraSetupWidget signals / slots
2021-05-12 12:19:36 UTC
Nikolay Bitkin
pushed 9db754d4f0e29df0a1e1a60de0123c7b9525b455
Video acquisition from Imperx cameras via Imperx SDK implemented
2021-05-11 20:09:07 UTC
Fyodor
pushed bcc2d766634575fe12dbb788a349a8cc37b91390
Update README.md
2021-05-08 06:49:10 UTC
Fyodor
pushed e1c7e084242899789fc2f37f0f4c3bc0f5163fd0
Update README.md
2021-05-08 06:45:06 UTC
Files
Title Description Format
pylon4NvidiaGPU_03032021_wm This is the pylon4NvidiaGPU prototype itself. 7z
README The README explains how to set up and use the prototype. txt
Info

Project State

Public Project

Licences

Software Licence: BSD-3-Clause
Hardware Licence: Project has no hardware

Project Tags

Admins

mbinev

Does this project pique your interest?

Login or register to join or follow this project.

Comments
Back to top

Your comments, please!

Want to comment this ... Show more