Real-time Image Processing on NVIDIA GPU with Basler pylon and Fastvideo

This project provides a prototype that allows for real-time image processing on Nvidia GPUs. The prototype is based on the Basler pylon Camera Software Suite and Fastvideo SDK and enables testing of sophisticated image processing algorithms with Basler USB3 Vision cameras on Windows 7/8.1/10 64bit.

Documentation

Challenges:

Basler’s machine vision cameras use an internal FPGA for certain image processing tasks, e.g., debayering, automatic exposure time, automatic gain, automatic white balance, gamma, noise reduction, sharpness enhancement, vignetting correction, etc. This means that real-time image processing takes place directly on the camera without causing any delays or additional CPU load on the host computer. However, there are a number of applications with requirements exceeding the feature capabilities offered by Basler’s machine vision cameras. For example, customers may need additional features not included in the respective camera firmware, e.g., rotation, flat-field correction, etc., or extended ranges of existing camera features, e.g., more binning factors. Another aspect is that using the camera’s FPGA to perform certain image processing algorithms in some cases may adversely affect camera performance or frame rate. For example, performing debayering on the camera’s FPGA may require more bandwidth to transfer the image data than the given interface supports, e.g., Gigabit Ethernet (115 MByte/s) or USB 3.0 (380 MByte/s). This may lead to reduction in the camera’s frame rate. To overcome the challenges detailed above, performing real-time image processing on a GPU is worth considering as an alternative approach.

What Does the Prototype Support?

The prototype created in this project supports and is based on the following:

  • Basler USB3 Vision cameras

  • Operating systems: Windows 7, 8.1, 10, 64-bit

  • NVIDIA GPUs: 6xx series and above (computing capability ≥ 3.0). The prototype has been tested with:

    • NVIDIA GeForce GT 1030, 2 GB
    • NVIDIA GeForce GTX 1050, 4 GB
  • CUDA Toolkit 10.2

  • Basler pylon Camera Software Suite 5.2 for Windows 64-bit:

    • Copy-deployed version of the pylon Viewer is included in the prototype.
    • pylon programming samples are included in the prototype.
    • For USB3 Vision drivers, a regular pylon 5.2 version has to be installed.
  • Fastvideo SDK for Windows 64-bit with watermark (included within the prototype).

Which Image Processing Algorithms Does the Prototype Offer?

  • Unpacking of raw image data, e.g., Mono12p or Bayer12p packed data
  • Dynamic defect pixel and hot-spot correction
  • Flat-field correction
  • Fixed-pattern noise subtraction
  • Binning: 2x2, 3x3, 4x4 (monochrome only; color and higher binning factors possible on request)
  • 1D LUT (lookup table)
  • White balance correction (color only)
  • Debayering: 5x5, 11x11, 23x23 (color only)
  • Base color correction (color only)
  • Denoising
  • Rotation: 90°, 180°, 270°, flip-flop (additional angles possible on request)
  • Cropping
  • Resizing, e.g., downscaling (upscaling possible on request).
  • Sharpness
  • Gamma correction
  • JPEG image encoding and saving
  • MJPEG video encoding and saving

Which Additional Features Are Possible on Request?

  • GigE Vision camera support
  • CXP camera support
  • Linux x86_64 and Linux ARM aarch64 support
  • J2K lossy and lossless encoding for 8–16 bit
  • H.264/H.265 encoding
  • Low-latency streaming
  • Interoperability with FFmpeg for output encoding and streaming
  • Live video output via OpenGL
  • Decompression for Basler ace 2 cameras
  • Additional rotation angles
  • Support of DCP profiles (Raw to RGB)
  • 3D LUT support (color grading)
  • HDR support
  • Perspective transforms (parallel edge distortion)
  • Undistortion via LCP profile (removal of barrel or pincushion effects)
  • Custom FFmpeg filters and codecs on GPU
  • Others

Performance & Test Results

The following table shows test results that clearly demonstrate that using GPU processing doesn’t reduce camera performance or frame rate. In the case of performing debayering on the GPU, the camera’s frame rate can even be increased.

enter image description here

Performance Issues

The following aspects typically affect the upload and download times possible with the GPU:

  • The PCIe generation itself
  • Number of available/active lanes
  • Other devices using the PCIe bus

Therefore, it’s advisable to test different scenarios in order to determine the PCIe performance.

Next Steps

Have you tested and enjoyed the image processing algorithms implemented in the prototype?

Would you like to use them within your own application without watermark?

Are you looking for even more functionality above what the current prototype offers?

Please refer to the official Fastvideo solution on GitHub (https://github.com/fastvideo/gpu-camera-sample), which provides other/additional functionality. Alternatively, simply get in touch with Fastvideo LLC (https://www.fastcompression.com/) directly.

Code

GitHub Repository

https://github.com/fastvideo/gpu-camera-sample

Commits

art
pushed 6c0078795295f21f938c1d7728267a3d2531f2e0
added parallel(openmp) version of unpacker
2022-09-16 07:31:50 UTC
Fyodor
pushed cc8ee837bb528744a187a3684bdd0d73c7cf83b8
Update README.md
2022-09-15 13:19:48 UTC
aaasas
pushed c04ea8e8a67932551c4c93b6e88c00504d2e6c3c
fix for MU181CR-ON
2022-09-15 13:06:05 UTC
AlexVRom
pushed 2686d91188d2ccf1491b2c7de283604329bf0c56
Fixed WB picker for gray camera, added view menu, merged 2 camera setup widgets into 1
2022-08-19 07:19:47 UTC
AlexVRom
pushed 3c171cbc0aa9526dd263d584c95fc697e56595b8
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2022-08-18 07:04:48 UTC
AlexVRom
pushed 428bd82148dcb2b2319489b38ec397877b2fcb10
Fixed gray pipeline
2022-08-18 07:03:28 UTC
Fyodor
pushed f8401416b18b88c2e67a05b724162011165df2a8
Update README.md
2022-08-17 15:50:35 UTC
Fyodor
pushed 3ebe064ddad4c45a190cf7967b2d831a6260191b
Update README.md
2022-08-17 14:59:55 UTC
user
pushed 7f570d9bf27a15cafeda8f766b10f80425dd6bd1
added the unpacked procedure for the "MU181CR-ON" camera model
2022-07-09 07:41:07 UTC
Fyodor
pushed 5bab108c6e69e92a18581d0e19472b22ac0eae46
Update README.md
2022-07-08 15:21:31 UTC
Fyodor
pushed c1a8121981b1c21e9b82744721d88f1acfeb4526
Update README.md
2022-06-04 15:20:52 UTC
AlexVRom
pushed 90f3a3b5fa6e867b04dec927c457e254bcb23883
Fixed download links
2022-05-30 08:57:05 UTC
AlexVRom
pushed e1d487fb074fb1a7fa0b57ccfccec6ebca00cbbe
Fixed Windows build issues
2022-05-27 07:13:40 UTC
AlexVRom
pushed 7931e0281230d62aad1c61d10c5128be07adcaa7
Updated readme
2022-05-24 08:11:12 UTC
AlexVRom
pushed e9f57b0891c60a1671748318d49788e6a3e78b96
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2022-05-24 08:06:52 UTC
AlexVRom
pushed 10a0bee9a1a387f6ce92325d0fc54617809bccf5
Updated precompiled libs
2022-05-24 08:05:55 UTC
Fyodor
pushed 7408b242a72dc0f6a5ffcd662f2c940360f92f10
Update README.md
2022-05-20 15:45:26 UTC
AlexVRom
pushed 4731513313e117664e458185461d4a55163b3b5a
Updated SDK download links
2022-05-20 12:21:48 UTC
artthirt@gmail.com
pushed ed12e9516480de405d9db93c45d8253713c803d4
MIPI: fixed for non-arm platform
2022-05-18 15:09:23 UTC
user
pushed 438d9cf98ee62ccd2922cc824745211c57f3ee8c
update ui. update work with MIPI cameras
2022-05-18 15:04:13 UTC
AlexVRom
pushed 86aefb9f4db7e1043da91d661823de98c369ad43
Fixed issues from previous merge
2022-05-18 12:18:20 UTC
AlexVRom
pushed 1f383f99013ad496fe0eb045156f5f301242b91d
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2022-05-18 10:30:40 UTC
AlexVRom
pushed 6a9522f84700fbee374180581148fae2acba535b
Merge branch 'jetson-mipi' of https://github.com/fastvideo/gpu-camera-sample.git
2022-05-18 10:24:20 UTC
artthirt@gmail.com
pushed b2490384689952d82ef5d29ca414c5af7d5ab430
added the search path for fastvideo sdk libraries for arm64
2022-05-15 08:39:49 UTC
artthirt@gmail.com
pushed 012d56f4533466bd36dc4fbc5dac60601a550dbd
update CMakeLists.txt for nvidia tegra
2022-05-15 05:20:24 UTC
user
pushed 245edf431a04de26c24d87166b4438d6a4804b5a
fix cmake project for linux
2022-05-12 12:18:20 UTC
artthirt@gmail.com
pushed 86d09c3cadb6e227aee82915ee7d6f82fae75df4
added copy dll to binary output (windows)
2022-05-12 11:29:03 UTC
artthirt@gmail.com
pushed 3977e6587856ab5ca7fd8b8b661211261d5ea754
added output directory for executable files
2022-05-12 10:30:08 UTC
user
pushed eba00b4b423c6db0bfaf310298fce0315c0cc94e
updated cmake for compile on ubuntu 18.04
2022-05-12 10:19:46 UTC
artthirt@gmail.com
pushed e94874ad30918d504db1741ed89346d83a28b3d2
added cmake project files (windows)
2022-05-12 07:54:17 UTC
Files
Title Description Format
pylon4NvidiaGPU_03032021_wm This is the pylon4NvidiaGPU prototype itself. 7z
README The README explains how to set up and use the prototype. txt
Info

Project State

Public Project

Licences

Software Licence: BSD-3-Clause
Hardware Licence: Project has no hardware

Project Tags

Admins

mbinev

Does this project pique your interest?

Login or register to join or follow this project.

Comments
Back to top

Ready to join the project?

You'd like to participate ... Show more