Real-time Image Processing on NVIDIA GPU with Basler pylon and Fastvideo

Challenges:

Basler’s machine vision cameras use an internal FPGA for certain image processing tasks, e.g., debayering, automatic exposure time, automatic gain, automatic white balance, gamma, noise reduction, sharpness enhancement, vignetting correction, etc. This means that real-time image processing takes place directly on the camera without causing any delays or additional CPU load on the host computer. However, there are a number of applications with requirements exceeding the feature capabilities offered by Basler’s machine vision cameras. For example, customers may need additional features not included in the respective camera firmware, e.g., rotation, flat-field correction, etc., or extended ranges of existing camera features, e.g., more binning factors. Another aspect is that using the camera’s FPGA to perform certain image processing algorithms in some cases may adversely affect camera performance or frame rate. For example, performing debayering on the camera’s FPGA may require more bandwidth to transfer the image data than the given interface supports, e.g., Gigabit Ethernet (115 MByte/s) or USB 3.0 (380 MByte/s). This may lead to reduction in the camera’s frame rate. To overcome the challenges detailed above, performing real-time image processing on a GPU is worth considering as an alternative approach.

What Does the Prototype Support?

The prototype created in this project supports and is based on the following:

• Basler USB3 Vision cameras

• Operating systems: Windows 7, 8.1, 10, 64-bit

• NVIDIA GPUs: 6xx series and above (computing capability ≥ 3.0). The prototype has been tested with:

  • NVIDIA GeForce GT 1030, 2 GB
  • NVIDIA GeForce GTX 1050, 4 GB

• CUDA Toolkit 10.2

• Basler pylon Camera Software Suite 5.2 for Windows 64-bit:

  • Copy-deployed version of the pylon Viewer is included in the prototype.
  • pylon programming samples are included in the prototype.
  • For USB3 Vision drivers, a regular pylon 5.2 version has to be installed.

• Fastvideo SDK for Windows 64-bit with watermark (included within the prototype).

Which Image Processing Algorithms Does the Prototype Offer?

• Unpacking of raw image data, e.g., Mono12p or Bayer12p packed data
• Dynamic defect pixel and hot-spot correction
• Flat-field correction
• Fixed-pattern noise subtraction
• Binning: 2x2, 3x3, 4x4 (monochrome only; color and higher binning factors possible on request)
• 1D LUT (lookup table)
• White balance correction (color only)
• Debayering: 5x5, 11x11, 23x23 (color only)
• Base color correction (color only)
• Denoising
• Rotation: 90°, 180°, 270°, flip-flop (additional angles possible on request)
• Cropping
• Resizing, e.g., downscaling (upscaling possible on request).
• Sharpness
• Gamma correction
• JPEG image encoding and saving
• MJPEG video encoding and saving

Which Additional Features Are Possible on Request?

• GigE Vision camera support
• CXP camera support
• Linux x86_64 and Linux ARM aarch64 support
• J2K lossy and lossless encoding for 8–16 bit
• H.264/H.265 encoding
• Low-latency streaming
• Interoperability with FFmpeg for output encoding and streaming
• Live video output via OpenGL
• Decompression for Basler ace 2 cameras
• Additional rotation angles
• Support of DCP profiles (Raw to RGB)
• 3D LUT support (color grading)
• HDR support
• Perspective transforms (parallel edge distortion)
• Undistortion via LCP profile (removal of barrel or pincushion effects)
• Custom FFmpeg filters and codecs on GPU
• Others

Performance & Test Results

The following table shows test results that clearly demonstrate that using GPU processing doesn’t reduce camera performance or frame rate. In the case of performing debayering on the GPU, the camera’s frame rate can even be increased.

Performance Issues

The following aspects typically affect the upload and download times possible with the GPU:

• The PCIe generation itself
• Number of available/active lanes
• Other devices using the PCIe bus

Therefore, it’s advisable to test different scenarios in order to determine the PCIe performance.

Next Steps

Have you tested and enjoyed the image processing algorithms implemented in the prototype?

Would you like to use them within your own application without watermark?

Are you looking for even more functionality above what the current prototype offers?

Please refer to the official Fastvideo solution on GitHub (https://github.com/fastvideo/gpu-camera-sample), which provides other/additional functionality. Alternatively, simply get in touch with Fastvideo LLC (https://www.fastcompression.com/) directly.