Real-time Image Processing on NVIDIA GPU with Basler pylon and Fastvideo

This project provides a prototype that allows for real-time image processing on Nvidia GPUs. The prototype is based on the Basler pylon Camera Software Suite and Fastvideo SDK and enables testing of sophisticated image processing algorithms with Basler USB3 Vision cameras on Windows 7/8.1/10 64bit.

Documentation

Challenges:

Basler’s machine vision cameras use an internal FPGA for certain image processing tasks, e.g., debayering, automatic exposure time, automatic gain, automatic white balance, gamma, noise reduction, sharpness enhancement, vignetting correction, etc. This means that real-time image processing takes place directly on the camera without causing any delays or additional CPU load on the host computer. However, there are a number of applications with requirements exceeding the feature capabilities offered by Basler’s machine vision cameras. For example, customers may need additional features not included in the respective camera firmware, e.g., rotation, flat-field correction, etc., or extended ranges of existing camera features, e.g., more binning factors. Another aspect is that using the camera’s FPGA to perform certain image processing algorithms in some cases may adversely affect camera performance or frame rate. For example, performing debayering on the camera’s FPGA may require more bandwidth to transfer the image data than the given interface supports, e.g., Gigabit Ethernet (115 MByte/s) or USB 3.0 (380 MByte/s). This may lead to reduction in the camera’s frame rate. To overcome the challenges detailed above, performing real-time image processing on a GPU is worth considering as an alternative approach.

What Does the Prototype Support?

The prototype created in this project supports and is based on the following:

  • Basler USB3 Vision cameras

  • Operating systems: Windows 7, 8.1, 10, 64-bit

  • NVIDIA GPUs: 6xx series and above (computing capability ≥ 3.0). The prototype has been tested with:

    • NVIDIA GeForce GT 1030, 2 GB
    • NVIDIA GeForce GTX 1050, 4 GB
  • CUDA Toolkit 10.2

  • Basler pylon Camera Software Suite 5.2 for Windows 64-bit:

    • Copy-deployed version of the pylon Viewer is included in the prototype.
    • pylon programming samples are included in the prototype.
    • For USB3 Vision drivers, a regular pylon 5.2 version has to be installed.
  • Fastvideo SDK for Windows 64-bit with watermark (included within the prototype).

Which Image Processing Algorithms Does the Prototype Offer?

  • Unpacking of raw image data, e.g., Mono12p or Bayer12p packed data
  • Dynamic defect pixel and hot-spot correction
  • Flat-field correction
  • Fixed-pattern noise subtraction
  • Binning: 2x2, 3x3, 4x4 (monochrome only; color and higher binning factors possible on request)
  • 1D LUT (lookup table)
  • White balance correction (color only)
  • Debayering: 5x5, 11x11, 23x23 (color only)
  • Base color correction (color only)
  • Denoising
  • Rotation: 90°, 180°, 270°, flip-flop (additional angles possible on request)
  • Cropping
  • Resizing, e.g., downscaling (upscaling possible on request).
  • Sharpness
  • Gamma correction
  • JPEG image encoding and saving
  • MJPEG video encoding and saving

Which Additional Features Are Possible on Request?

  • GigE Vision camera support
  • CXP camera support
  • Linux x86_64 and Linux ARM aarch64 support
  • J2K lossy and lossless encoding for 8–16 bit
  • H.264/H.265 encoding
  • Low-latency streaming
  • Interoperability with FFmpeg for output encoding and streaming
  • Live video output via OpenGL
  • Decompression for Basler ace 2 cameras
  • Additional rotation angles
  • Support of DCP profiles (Raw to RGB)
  • 3D LUT support (color grading)
  • HDR support
  • Perspective transforms (parallel edge distortion)
  • Undistortion via LCP profile (removal of barrel or pincushion effects)
  • Custom FFmpeg filters and codecs on GPU
  • Others

Performance & Test Results

The following table shows test results that clearly demonstrate that using GPU processing doesn’t reduce camera performance or frame rate. In the case of performing debayering on the GPU, the camera’s frame rate can even be increased.

enter image description here

Performance Issues

The following aspects typically affect the upload and download times possible with the GPU:

  • The PCIe generation itself
  • Number of available/active lanes
  • Other devices using the PCIe bus

Therefore, it’s advisable to test different scenarios in order to determine the PCIe performance.

Next Steps

Have you tested and enjoyed the image processing algorithms implemented in the prototype?

Would you like to use them within your own application without watermark?

Are you looking for even more functionality above what the current prototype offers?

Please refer to the official Fastvideo solution on GitHub (https://github.com/fastvideo/gpu-camera-sample), which provides other/additional functionality. Alternatively, simply get in touch with Fastvideo LLC (https://www.fastcompression.com/) directly.

Code

GitHub Repository

https://github.com/fastvideo/gpu-camera-sample

Commits

AlexVRom
pushed 7931e0281230d62aad1c61d10c5128be07adcaa7
Updated readme
2022-05-24 08:11:12 UTC
AlexVRom
pushed e9f57b0891c60a1671748318d49788e6a3e78b96
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2022-05-24 08:06:52 UTC
AlexVRom
pushed 10a0bee9a1a387f6ce92325d0fc54617809bccf5
Updated precompiled libs
2022-05-24 08:05:55 UTC
Fyodor
pushed 7408b242a72dc0f6a5ffcd662f2c940360f92f10
Update README.md
2022-05-20 15:45:26 UTC
AlexVRom
pushed 4731513313e117664e458185461d4a55163b3b5a
Updated SDK download links
2022-05-20 12:21:48 UTC
artthirt@gmail.com
pushed ed12e9516480de405d9db93c45d8253713c803d4
MIPI: fixed for non-arm platform
2022-05-18 15:09:23 UTC
user
pushed 438d9cf98ee62ccd2922cc824745211c57f3ee8c
update ui. update work with MIPI cameras
2022-05-18 15:04:13 UTC
AlexVRom
pushed 86aefb9f4db7e1043da91d661823de98c369ad43
Fixed issues from previous merge
2022-05-18 12:18:20 UTC
AlexVRom
pushed 1f383f99013ad496fe0eb045156f5f301242b91d
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2022-05-18 10:30:40 UTC
AlexVRom
pushed 6a9522f84700fbee374180581148fae2acba535b
Merge branch 'jetson-mipi' of https://github.com/fastvideo/gpu-camera-sample.git
2022-05-18 10:24:20 UTC
artthirt@gmail.com
pushed b2490384689952d82ef5d29ca414c5af7d5ab430
added the search path for fastvideo sdk libraries for arm64
2022-05-15 08:39:49 UTC
artthirt@gmail.com
pushed 012d56f4533466bd36dc4fbc5dac60601a550dbd
update CMakeLists.txt for nvidia tegra
2022-05-15 05:20:24 UTC
user
pushed 245edf431a04de26c24d87166b4438d6a4804b5a
fix cmake project for linux
2022-05-12 12:18:20 UTC
artthirt@gmail.com
pushed 86d09c3cadb6e227aee82915ee7d6f82fae75df4
added copy dll to binary output (windows)
2022-05-12 11:29:03 UTC
artthirt@gmail.com
pushed 3977e6587856ab5ca7fd8b8b661211261d5ea754
added output directory for executable files
2022-05-12 10:30:08 UTC
user
pushed eba00b4b423c6db0bfaf310298fce0315c0cc94e
updated cmake for compile on ubuntu 18.04
2022-05-12 10:19:46 UTC
artthirt@gmail.com
pushed e94874ad30918d504db1741ed89346d83a28b3d2
added cmake project files (windows)
2022-05-12 07:54:17 UTC
user
pushed 253f7609dde20db8ee384ce1f5546913441dd1ce
updated for ubuntu 18.04 (x86)
2022-05-12 05:08:08 UTC
artthirt@gmail.com
pushed afb7528beebcf7c9da8f2de516b63fd7ad22fbef
updated for new demo lib (windows)
2022-05-11 15:26:01 UTC
Fyodor
pushed 3b3431e1d075c8f1e08ce8a10297d44090c029d1
Update README.md
2022-05-11 12:55:27 UTC
Fyodor
pushed eb929f3b8b4be115d9e3a3e594c30d59bdf2f3d8
Update README.md
2022-05-03 09:09:03 UTC
Fyodor
pushed aa2772ac01b4daf2434394e8419859a379e4fad7
Update README.md
2022-05-03 09:06:29 UTC
Fyodor
pushed 9f8e3646f217d346215103233a98deb520673e99
Update README.md
2022-05-03 09:05:37 UTC
Fyodor
pushed 4bb8ac23982cb91c7366483e143157599e1fdd43
Update README.md
2022-04-09 11:12:07 UTC
Fyodor
pushed 29c89adbc7cfae5165615248524088a800b0e1ad
Update README.md
2022-04-07 14:45:31 UTC
Fyodor
pushed f073ba2c471abeb0831876b06792f53604f8ce0f
Update README.md
2022-04-07 14:13:35 UTC
AlexVRom
pushed d71827462bed6af7d30856cfc42c1ee132288f87
Update README.md
2022-02-28 10:32:13 UTC
Fyodor
pushed 7283892b9dd0ceeca53acdf9f8270d6d75e6f8d4
Update README.md
2022-02-21 07:45:41 UTC
Fyodor
pushed 47a27027e0de9dec913b1342579c8aad0a602e44
Update README.md
2022-01-14 10:22:23 UTC
Fyodor
pushed c0efab48874c3d5f4975dd28d1e40262fa6c38bf
Update README.md
2022-01-14 10:20:26 UTC
Files
Title Description Format
pylon4NvidiaGPU_03032021_wm This is the pylon4NvidiaGPU prototype itself. 7z
README The README explains how to set up and use the prototype. txt
Info

Project State

Public Project

Licences

Software Licence: BSD-3-Clause
Hardware Licence: Project has no hardware

Project Tags

Admins

mbinev

Does this project pique your interest?

Login or register to join or follow this project.

Comments
Back to top

Your comments, please!

Want to comment this ... Show more