ZED X and NVIDIA Jetson: In-depth review

Intel’s move to focus exclusively on its x86 architecture made sense strategically, as it strengthened their position in systems that already relied on Intel processors. But this left a gap in the market for ARM/Jetson compatibility, which Stereolabs quickly recognized and filled. By ensuring the ZED camera worked seamlessly with NVIDIA’s ARM-based Jetson platform, Stereolabs provided an alternative for users who needed flexibility beyond Intel’s ecosystem. This decision allowed Stereolabs to turn Intel’s limitations into an opportunity, carving out a unique place in the industry.

Stereolabs’ focus on ARM compatibility also aligned perfectly with NVIDIA’s expanding vision. NVIDIA was evolving from its traditional role in GPUs to driving advancements in robotics with its Isaac SDK and Jetson hardware. Their commitment to combining 2D/3D vision with GPU-accelerated image processing created a space that Intel had not yet explored.

This shared emphasis on innovation and adaptability made NVIDIA and Stereolabs natural partners. By the time Stereolabs launched the ZED X camera, they had fully embraced this partnership, dropping support for Intel platforms entirely.

Oh irony!

The exclusive ZED X-NVIDIA combo has made a bet on a few high-profile applications in robotics. We have highlighted here the three most dominantly supported applications for the ZED X.

As intriguing as these applications are, you might be wondering, "How do I actually get started?" The next chapter will delve into the essential building blocks you'll need to bring these applications to life. From setting up the development environment to choosing the right SDKs and hardware, we'll guide you through the foundational steps to ensure a successful project.

Embarking on a project that leverages the ZED X camera and NVIDIA Jetson platforms requires a well-thought-out architecture. In this chapter, we'll outline the key components you'll need to set up your development environment and target hardware and software runtime.

We outlined in this diagram the software and components of a typical setup. You can develop code from any Operating System, using a remote connection to a target (any Jetson device). But you do need to setup this target device with the NVIDIA sdkmanager. Most developers nowadays run Docker instances on the Jetson target in order to keep the complex dependencies equal among all their setups.

Here’s the breakdown:

ZED SDK: The ZED SDK v4 is your primary tool for interacting with the ZED X camera. It provides APIs for capturing images, generating depth maps, and managing the camera devices.

ROS Humble: We recommend to stick to ROS 2 and the Humble release, as it is best supported on NVIDIA hardware and is a requirement if you want to add Isaac ROS later-on, since the Isaac SDK is not required.

Docker: You will run Docker containers both on your target hardware and development environment, even if these are already running the Ubuntu 20.04 Operating System.

NVIDIA SDK Manager: Only required to bootstrap your Jetson device (it will install the correct OS and libraries). Once the bootstrapping is done, you don’t need the SDK manager anymore and any developer can use the Docker containers instead, on any supported Docker OS.

The ZED X camera offers a variety of integration options that cater to different development needs. In this chapter, we'll delve into some of the most popular frameworks in the mobile robotics community: ZED ROS2, ZED PCL, ZED Docker, and ZED OpenCV.

Highlights:

• ZED ROS node: The wrapper provides access to left and right rectified/unrectified images, depth data, colored 3D point clouds, IMU data, VIO pose tracking, detected objects and human body skeleton.
• GNSS Fusion: The integration allows for GNSS data fusion, enabling more accurate robot localization in the presence of a ROS 2 driver for a GNSS sensor.
• 2D Mode: For robots operating on a planar surface, a 2D mode is available, and fixes the Z coordinate for odometry and pose to a constant value.
• Object and Body Tracking: The object detection pipeline offered by the ZED SDK is made available by the ZED ROS node.
• Dynamic Reconfigure: The ZED camera can be configured in rqt using the Dynamic Reconfigure Plugin, allowing you to inspect many of the parameters of the camera.

Limitations:

• Requires ZED SDK v4.0.6 or later
• No Docker container provided for Humble on 20.04.

The ZED ROS2 integration offers a comprehensive solution for those looking to leverage the capabilities of ZED cameras in a ROS2 environment.

Highlights:

• Seamless Conversion: The example program uses a conversion function to map between sl::Mat and cv::Mat, allowing for a seamless transition between ZED SDK and OpenCV.
• Flexible Configuration: The program allows for various configurations, including camera resolution, depth mode, and coordinate units, making it adaptable to different use-cases.
• CUDA Support: For those with CUDA-enabled devices, the program also includes a conversion function for cv::cuda::GpuMat, enabling GPU-accelerated image processing.

Limitations:

• None we could identify.

This ZED OpenCV integration offers a straightforward way to integrate ZED cameras with OpenCV, especially for those who prefer not to use ROS.

With this, we've covered the popular framework integrations for the ZED X camera. Each has its own set of capabilities and limitations, making it crucial to choose the one that aligns best with your project's specific needs.

Hopefully you became a bit wiser upon the ins and outs of the using the ZED X on an NVIDIA target. Intermodalics offers professional support on this and other integrations, don’t hesitate to reach out !