Real-time Image Processing on NVIDIA GPU with Basler pylon and Fastvideo

This project provides a prototype that allows for real-time image processing on Nvidia GPUs. The prototype is based on the Basler pylon Camera Software Suite and Fastvideo SDK and enables testing of sophisticated image processing algorithms with Basler USB3 Vision cameras on Windows 7/8.1/10 64bit.

Documentation

Challenges:

Basler’s machine vision cameras use an internal FPGA for certain image processing tasks, e.g., debayering, automatic exposure time, automatic gain, automatic white balance, gamma, noise reduction, sharpness enhancement, vignetting correction, etc. This means that real-time image processing takes place directly on the camera without causing any delays or additional CPU load on the host computer. However, there are a number of applications with requirements exceeding the feature capabilities offered by Basler’s machine vision cameras. For example, customers may need additional features not included in the respective camera firmware, e.g., rotation, flat-field correction, etc., or extended ranges of existing camera features, e.g., more binning factors. Another aspect is that using the camera’s FPGA to perform certain image processing algorithms in some cases may adversely affect camera performance or frame rate. For example, performing debayering on the camera’s FPGA may require more bandwidth to transfer the image data than the given interface supports, e.g., Gigabit Ethernet (115 MByte/s) or USB 3.0 (380 MByte/s). This may lead to reduction in the camera’s frame rate. To overcome the challenges detailed above, performing real-time image processing on a GPU is worth considering as an alternative approach.

What Does the Prototype Support?

The prototype created in this project supports and is based on the following:

  • Basler USB3 Vision cameras

  • Operating systems: Windows 7, 8.1, 10, 64-bit

  • NVIDIA GPUs: 6xx series and above (computing capability ≥ 3.0). The prototype has been tested with:

    • NVIDIA GeForce GT 1030, 2 GB
    • NVIDIA GeForce GTX 1050, 4 GB
  • CUDA Toolkit 10.2

  • Basler pylon Camera Software Suite 5.2 for Windows 64-bit:

    • Copy-deployed version of the pylon Viewer is included in the prototype.
    • pylon programming samples are included in the prototype.
    • For USB3 Vision drivers, a regular pylon 5.2 version has to be installed.
  • Fastvideo SDK for Windows 64-bit with watermark (included within the prototype).

Which Image Processing Algorithms Does the Prototype Offer?

  • Unpacking of raw image data, e.g., Mono12p or Bayer12p packed data
  • Dynamic defect pixel and hot-spot correction
  • Flat-field correction
  • Fixed-pattern noise subtraction
  • Binning: 2x2, 3x3, 4x4 (monochrome only; color and higher binning factors possible on request)
  • 1D LUT (lookup table)
  • White balance correction (color only)
  • Debayering: 5x5, 11x11, 23x23 (color only)
  • Base color correction (color only)
  • Denoising
  • Rotation: 90°, 180°, 270°, flip-flop (additional angles possible on request)
  • Cropping
  • Resizing, e.g., downscaling (upscaling possible on request).
  • Sharpness
  • Gamma correction
  • JPEG image encoding and saving
  • MJPEG video encoding and saving

Which Additional Features Are Possible on Request?

  • GigE Vision camera support
  • CXP camera support
  • Linux x86_64 and Linux ARM aarch64 support
  • J2K lossy and lossless encoding for 8–16 bit
  • H.264/H.265 encoding
  • Low-latency streaming
  • Interoperability with FFmpeg for output encoding and streaming
  • Live video output via OpenGL
  • Decompression for Basler ace 2 cameras
  • Additional rotation angles
  • Support of DCP profiles (Raw to RGB)
  • 3D LUT support (color grading)
  • HDR support
  • Perspective transforms (parallel edge distortion)
  • Undistortion via LCP profile (removal of barrel or pincushion effects)
  • Custom FFmpeg filters and codecs on GPU
  • Others

Performance & Test Results

The following table shows test results that clearly demonstrate that using GPU processing doesn’t reduce camera performance or frame rate. In the case of performing debayering on the GPU, the camera’s frame rate can even be increased.

enter image description here

Performance Issues

The following aspects typically affect the upload and download times possible with the GPU:

  • The PCIe generation itself
  • Number of available/active lanes
  • Other devices using the PCIe bus

Therefore, it’s advisable to test different scenarios in order to determine the PCIe performance.

Next Steps

Have you tested and enjoyed the image processing algorithms implemented in the prototype?

Would you like to use them within your own application without watermark?

Are you looking for even more functionality above what the current prototype offers?

Please refer to the official Fastvideo solution on GitHub (https://github.com/fastvideo/gpu-camera-sample), which provides other/additional functionality. Alternatively, simply get in touch with Fastvideo LLC (https://www.fastcompression.com/) directly.

Code

GitHub Repository

https://github.com/fastvideo/gpu-camera-sample

Commits

AlexVRom
pushed 805459ec0db15a3b94886959f4f209b2d9ffd17a
Update README.md
2021-04-19 11:00:16 UTC
AlexVRom
pushed ee92068a2e09b068236d9a557739644b66c43cec
Merge branch 'master' of https://github.com/fastvideo/gpu-camera-sample
2021-04-19 10:54:06 UTC
AlexVRom
pushed f3b8fc87040b0eab0203dac2129f438cb8f39790
Added FLIR cameras support (tested on USB3 cameras only)
2021-04-19 10:53:21 UTC
Fyodor
pushed d96ea67893394af2a41713b2defaa60efa4267d4
Update README.md
2021-04-14 10:44:35 UTC
Fyodor
pushed 8bcf51a8b5378c578d5ceac2e7fe223f2ace6e9f
Update README.md
2021-03-11 21:09:22 UTC
Fyodor
pushed ba947f9b33e10c69da8649810fb695f0c8b79d62
Update README.md
2021-03-11 21:08:54 UTC
Fyodor
pushed 372999760581497fbc55e1f443e39b4509394f77
Update README.md
2021-03-11 21:08:30 UTC
Fyodor
pushed 639c7475ed385b899795db4991b11fc8306dd8aa
Update README.md
2021-03-11 16:31:24 UTC
Fyodor
pushed ac095e1ceaae933239164c9616a57957b82d15b1
Update README.md
2021-03-11 16:29:17 UTC
Fyodor
pushed fb999559b94c538276fb5c60a2cc7a1451d772fb
Update README.md
2021-02-18 12:19:20 UTC
Fyodor
pushed 90fe89356e1a92ef67cbe25cfe9159b16b6a777f
Update README.md
2021-01-06 06:47:49 UTC
Fyodor
pushed 5f606d88221c856a9e4673b110c56f3e35918bd9
Update README.md
2020-11-10 08:06:10 UTC
Fyodor
pushed 5828dc296ad69bdacd2e560bde4e52b50c570d50
Update README.md
2020-11-10 07:38:25 UTC
Fyodor
pushed ab1dbc44345c9e6363024254534f960dd2559c8e
Update README.md
2020-11-08 19:17:53 UTC
Fyodor
pushed eab34f96760a3327695a0a0af9398ce9a0d36839
Update README.md
2020-10-16 20:28:23 UTC
Fyodor
pushed f29395151f1faf1102042b84f48f17e47318eff4
Update README.md
2020-10-16 20:26:51 UTC
Fyodor
pushed 8b428d908748ae7bb43e03967b220b527ec488b2
Update README.md
2020-10-16 20:22:29 UTC
Fyodor
pushed 458c5687ed50e2b63b6e1607f2866d93efe0b493
Update README.md
2020-10-11 13:37:48 UTC
Fyodor
pushed 60107c46f691cff1ec8c4f60616ca193146e95cc
Update README.md
2020-10-11 13:33:39 UTC
Fyodor
pushed 975dcd6322a5b640fd95f6afbb408391d274a663
Update README.md
2020-10-10 15:18:09 UTC
artthirt
pushed 5041b187a195b483a9970905db9fb6e8d5698bb9
Update README.md
2020-10-10 10:48:04 UTC
artthirt
pushed 206978a486e241c121d7508a21b681067b556296
Update README.md
2020-10-10 08:53:11 UTC
artthirt
pushed 0d0dd5489871d83b9be6d800379776478ef7557d
Update README.md
2020-10-10 08:49:20 UTC
AlexVRom
pushed 11579929d9c700882d464375e76a853096b56fa7
Update README.md
2020-10-09 06:45:51 UTC
AlexVRom
pushed e6365805c231fe60e5e8ffa007510facd8f232ae
Update README.md
2020-10-09 06:44:19 UTC
artthirt
pushed 0dc08c8f0e14d36be7f7b0f31e66df31bad19c57
fix warnings
2020-10-08 14:08:57 UTC
user
pushed c57062e4319d7b20fcb80cb5e14d288ef113532a
minimize delay in main interface. some optimizations
2020-10-08 14:04:42 UTC
user
pushed 427561f354d1bbf72fe042a87f1f6e716b7b9ebc
fix pix fmt
2020-10-08 12:19:31 UTC
user
pushed 9c36eba972af2287356caa53dfd80e8a39271e7a
remove unused copy memory
2020-10-08 12:14:15 UTC
user
pushed 0edcb0ff83717b82763753e624f158cbcc1f0312
remove unused variables
2020-10-08 06:06:44 UTC
Files
Title Description Format
pylon4NvidiaGPU_03032021_wm This is the pylon4NvidiaGPU prototype itself. 7z
README The README explains how to set up and use the prototype. txt
Info

Project State

Public Project

Licences

Software Licence: BSD-3-Clause
Hardware Licence: Project has no hardware

Project Tags

Admins

mbinev

Does this project pique your interest?

Login or register to join or follow this project.

Comments
Back to top

Ready to join the project?

You'd like to participate ... Show more