We provide scripts for performing preprocessing operations (e.g., cropping, debayering, rectification) and visualizing the data.
We recommend building and running the preprocessing and visualization code with Docker. This section describes Docker installation, building the nsavp_preprocessing_and_visualization
Docker image, and running the image with multiple interactive bash sessions. The following has been tested with Ubuntu 22.04.
To begin, install Docker Engine and perform this post-installation step.
After installing Docker, a Docker image for running the code described in this page can be built as follows:
cd nsavp_tools/
docker build -t nsavp_preprocessing_and_visualization -f docker/Dockerfile_preprocessing_and_visualization .
The result will be a new Docker image named nsavp_preprocessing_and_visualization
with the preprocessing_and_visualization
package built in the /catkin_ws/
directory within the image.
Unless otherwise specified, all commands in subsequent sections are run through an interactive bash session in a Docker container running the nsavp_preprocessing_and_visualization
image. The following commands will run the image and start an interactive bash session in the running container:
xhost +
docker run -it -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix -v /path/to/nsavp/data/:/nsavp_data nsavp_preprocessing_and_visualization
The arguments -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix
are included to allow display windows to be opened from within the container. The /path/to/nsavp/data/
path should be replaced with the folder storing NSAVP data on your host machine. This folder will appear within the container as the /nsavp_data
folder. Once finished, the interactive session can be exited with Ctrl+D
and the container can be removed by first finding it's name with docker ps
and then removing it with docker rm <container name>
.
Running the examples below requires multiple interactive bash sessions in the same Docker container. To open an additional interactive bash session within a running container, open a new terminal window, identify the name of the container with docker ps
, and finally run:
docker exec -it <container_name> bash
The preprocessing_and_visualization.launch
launch file provides a single interface for a variety of preprocessing operations and visualizations of the data. The launch file runs a series of ROS nodes which process NSAVP data when it is played back as ROS messages. Therefore, to use the launch file with a set of NSAVP data, the data must first be converted to the ROS1 rosbag format as described in the Converting H5 Files to a ROS1 Rosbag File section of the conversion documentation page. An example of performing this conversion in order to preprocess and visualize the R0_FA0 sequence is given in the Examples section below.
The preprocessing_and_visualization.launch
launch file takes the following arguments:
- Calibration and manual measurement filepaths
path_calibration_results
(Required): A filepath to the calibration results YAML file for the sequence to be processed.path_measured_extrinsics
: A filepath to the measured extrinsics YAML file for the sequence to be processed. If provided, the camera housing frames will be visualized with respect to thebase_link
frame in a RViz window. Defaults to an empty string""
.
- Image preprocessing parameters:
debayer
: A boolean specifying whether (true
) or not (false
) to debayer RGB images. Defaults totrue
.crop_parameters_list
: A nested list containing crop parameters. Each element of the list is a list of crop parameters for an individual camera with the following format[<camera namespace>,<x offset>,<y offset>,<width>,<height>]
. For example,[[rgb_left,0,0,600,600],[rgb_right,0,0,600,600]]
crops the left and right RGB images to a 600x600 region in the top left of the images. Any camera not included in thecrop_parameters_list
will not be cropped. Seetopic_image_plot
for a method to determine cropping parameters. Defaults to a set of crop parameters which removes the car's hood in the R0_FA0 sequence (in conjunction with the other default parameter values).crop_raw
: A boolean specifying whether to perform cropping before (true
) or after (false
) rectification. Ifcrop_raw
isfalse
andrectify
isfalse
, then no cropping will be performed. Defaults tofalse
.rectify
: A boolean specifying whether (true
) or not (false
) to rectify all images. Rectification of bayer encoded images is not supported, so ifdebayer
isfalse
,rectify
will be treated asfalse
for RGB images, regardless of what value it has been set to. Defaults totrue
.resolution_rectified
: A two element list specfiying the resolution of all rectified images with the following format:[<width>,<height>]
. In the case of a monocular camera, ifresolution_rectified
is empty, the rectified resolution will match the resolution of the image input to rectification (which may be cropped from the raw image). In the case of a stereo camera, ifresolution_rectified
is empty, the rectified resolution will match the resolution of the first camera's images input to rectification (the 'first' camera here is<camera namespace 0>
in the stereo pair, seestereo_pairs
). Defaults to an empty list[]
.stereo_pairs
: A nested list containing pairs of cameras to rectify together. Each element of the list is a two element list with the following format:[<camera namespace 0>,<camera namespace 1>]
. Any two cameras can be specified as a pair and the script will attempt to rectify them. Rectification may fail if the translation between the two cameras is not predominantly vertical or horizontal (e.g., rectification fails with[rgb_left,mono_left]
because the translation between the cameras is along the image plane diagonal). An example value forstereo_pairs
is[[mono_left,adk_right],[mono_right,adk_left]]
. This would rectify the left monochrome camera with the right thermal camera and the right monochrome camera with the left thermal camera. A camera can be specified in thestereo_pairs
list only once (e.g.[[mono_left,adk_right],[mono_left,adk_left]]
will fail asmono_left
is listed twice). Any camera not included instereo_pairs
will be rectified as a monocular camera. Defaults to creating a stereo pair between the left and right camera of each modality.threshold_thermal
: A boolean specifying whether (true
) or not (false
) to threshold thermal images. Thresholding involves the following stages: (1) values less thanthreshold_thermal_minimum
are set tothreshold_thermal_minimum
, (2) values greater thanthreshold_thermal_maximum
are set tothreshold_thermal_maximum
, (3) the remaining values are normalized between 0 and 255, and (4) the result is converted to an unsigned 8-bit image. Thresholding is performed for visualization because the distribution of values in the raw 16-bit thermal images mainly occupies only a small subset of the 16-bit range and also has long tails. Therefore, linearly mapping the 16-bit range to an 8-bit range or linearly mapping the full range of values present in the image to an 8-bit range results in a low-contrast image. Defaults totrue
.threshold_thermal_minimum
: An integer specifying the bottom of the thresholded range (seethreshold_thermal
). Defaults to22500
.threshold_thermal_maximum
: An integer specifying the top of the thresholded range (seethreshold_thermal
). Defaults to25000
.
- Data visualization parameters:
visualize_all_sensors
: A boolean specifying whether (true
) or not (false
) to display all sensor data (images and ground truth pose) in a RViz window. This display always shows the images after the final stage of preprocessing (i.e., the/<camera namespace>/image_processed
topics, see the Image and CameraInfo Messages section for more details). Defaults totrue
.visualize_calibrated_transforms
: A boolean specifying whether (true
) or not (false
) to display the camera optical frames alongside the camera housing frames in the RViz window that is created when thepath_measured_extrinsics
filepath is given. The display does not appear if thepath_measured_extrinsics
filepath is left empty. The relative transformations between the optical frames are given by the calibration results file. They are displayed by assuming zero translation between theadk_left_housing
andadk_left_optical
frames. Defaults tofalse
.visualize_stereo_rectification
: A boolean specifying whether (true
) or not (false
) to display all stereo rectification pairs with lines drawn to assess the quality of the rectification. Iftrue
a rqt_gui window will be opened to display up to four concatenated images, one for each stereo pair. For each stereo pair, the images are concatenated based on the relative position of the cameras, e.g., for a horizontal stereo pair the images will be concatenated horizontally with the leftmost camera image on the left. Lines are drawn along the concatenation axis to allow the quality of the stereo rectification to be assessed (the lines should cross through the same objects in each image within a pair). Defaults tofalse
.topic_image_plot
: A string specifying a topic to be plot with matplotlib. If given, a matplotlib plot window will be opened displaying (in grayscale) the last image published on thetopic_image_plot
topic before the window was opened. If the window is closed, it will reopen with the most recent image again. This is intended to aid in setting cropping parameters as the matplotlib window allows zooming and displays the cursor's pixel coordinate in the bottom right if it hovers over the image. Defaults to an empty string""
.
Note: In this script, events are first processed into event images using the
dvs_renderer
package included in the rpg_dvs_ros event camera driver. These event images, pubished under the/dvxplorer_left/image_raw
and/dvxplorer_right/image_raw
topics, are input into the subsequent image preprocessing stages. Alternatively, the individual event pixel coordinates could be rectified (e.g., with OpenCV'sundistortPoints()
) and/or cropped, but this option is not demonstrated here.
The examples given in this section assume all files from the R0_FA0 sequence have been downloaded and all the H5 files have been converted into a single rosbag R0_FA0.bag
.
Assuming all files from the R0_FA0 sequence are located in the folder /nsvap_data/R0_FA0/
within a Docker container running the nsavp_conversion
image (see the Setup section of the conversion documentation page) the H5 files can be converted into a single rosbag with the following two commands:
roslaunch conversion h5_combine.launch path_base_folder:=/nsavp_data/R0_FA0/ filepath_h5_out:=R0_FA0.h5
roslaunch conversion h5_to_rosbag.launch path_h5_in:=/nsavp_data/R0_FA0/R0_FA0.h5 path_bag_out:=/nsavp_data/R0_FA0/R0_FA0.bag
The result would be a new rosbag file, /nsavp_data/R0_FA0/R0_FA0.bag
. Note that the rosbag format will be significantly larger than the H5 format. For the R0_FA0 sequence the H5 files sum to 139 GB while the rosbag containing all sensors is 320 GB.
These examples can also work with a rosbag containing only a subset of sensors from a single sequence (e.g., if only a subset of the R0_FA0 H5 files were downloaded before running the h5_combine
and h5_to_rosbag
commands above), although some visualizations will contain blank sections.
Each of the examples given in subsequent sections are run through an interactive bash session in a Docker container running the nsavp_preprocessing_and_visualization
image (note that this differs from the commands above which require the nsavp_conversion
Docker image). It is assumed that the files R0_FA0.bag
, R0_FA0_C1_calibration_results.yaml
, and R0_FA0_M0_measured_extrinsics.yaml
are located in the folder /nsavp_data/R0_FA0/
in this container.
To run the examples, first run the given preprocessing_and_visualization.launch
launch file command in one interactive bash session, and then play back the data with rosbag play
in a separate interactive bash session (see Setup for how to start multiple interactive bash sessions). Specifically, the examples given below can be reproduced by running rosbag play
as follows:
rosbag play /nsavp_data/R0_FA0/R0_FA0.bag -s 616
where the argument -s 616
starts play back 616 seconds into the sequence.
The raw sensor data can be visualized by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml debayer:=false crop_parameters_list:=[] rectify:=false threshold_thermal:=false
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting RViz display is shown below. Note that there are several potential issues with the raw image data: the images are distorted (this is most evident in the curved lamppost on the right side of the images), the RGB images are bayer encoded (note the visible grid pattern), and the hood of the car is visible in the event and RGB images. Additionally, the thermal image is somewhat hard to interpret as it is low contrast.
Note: RViz internally normalizes 16-bit images so the thermal image visualization here is not a direct reflection of their raw values.
In the center, the base_link
pose is visualized in an OpenStreetMap map (using the rviz_satellite ROS package). Note that in all RViz pose visualizations, red corresponds to the x-axis, green corresponds to the y-axis, and blue corresponds to the z-axis (this can be remembered as RGB mapping to xyz).
The default parameters values in the preprocessing_and_visualization.launch
launch file are set to correct the issues described in the previous subsection. Specifically, the RGB images are debayered, all images are rectified, the car's hood is cropped out of the rectified event and RGB images, and the thermal images are thresholded. Note that cropping after rectification preserves slightly more information than cropping before rectification. These image preprocessing operations can be run and the resulting sensor data can be visualized by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting RViz display is shown below.
The default parameters values in the preprocessing_and_visualization.launch
launch file are set such that stereo rectification is performed between the left and right camera of each modality. The stereo rectification of these cameras can be visualized by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml visualize_all_sensors:=false visualize_stereo_rectification:=true
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting rqt_gui display is shown below. Note that within all stereo pairs the horizontal lines cross through the same objects in each image. This indicates the calibration results are accurate and stereo rectification was successful.
The script also supports stereo rectification between cameras of different modalities. This can be demonstrated by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml visualize_all_sensors:=false visualize_stereo_rectification:=true stereo_pairs:=[[mono_left,adk_right],[rgb_left,mono_right],[adk_left,dvxplorer_right],[dvxplorer_left,rgb_right]]
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting rqt_gui display is shown below.
In addition, the script supports stereo rectification with vertical baselines, for example between the left RGB and left event cameras. This can be demonstrated by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml visualize_all_sensors:=false visualize_stereo_rectification:=true stereo_pairs:=[[rgb_left,dvxplorer_left]]
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting rqt_gui display is shown below.
The camera housing and optical frames can be viewed by first running the following command:
roslaunch preprocessing_and_visualization preprocessing_and_visualization.launch path_calibration_results:=/nsavp_data/R0_FA0/R0_FA0_C1_calibration_results.yaml path_measured_extrinsics:=/nsavp_data/R0_FA0/R0_FA0_M0_measured_extrinsics.yaml visualize_all_sensors:=false visualize_calibrated_transforms:=true
and then running rosbag play
in a separate interactive bash session.
A screenshot of the resulting RViz display is shown below. The frames and their IDs overlap, but each frame can be toggled from the RViz display by clicking Panels
then Displays
and clicking the checkboxes next to the frame IDs.
As mentioned above, the optical frames are visualized with respect to the housing frames by assuming zero translation between the adk_left_housing
and adk_left_optical
frames. The origins of these two frames are therefore coincident in the visualization by definition. Small offsets are visible between the right camera housing and optical frames, possibly due to errors in manual measurement.
The image_preprocessing.py
script, run by the preprocessing_and_visualization.launch
launch file, performs image preprocessing operations, publishes Image messages at each stage of processing, and publishes CameraInfo messages containing calibration and rectification information for each camera.
The possible image topics are as follows (using the naming scheme outlined in the CameraInfo documentation where possible):
image_raw
: raw (distorted, full resolution, possibly bayer encoded)image_color
: distorted, debayeredimage_crop
: distorted, cropped (possibly bayer encoded)image_crop_color
: distorted, debayered then croppedimage_rect
: rectifiedimage_rect_color
: debayered, rectifiedimage_crop_rect
: cropped then rectifiedimage_crop_rect_color
: debayered, cropped then rectifiedimage_rect_crop
: rectified then croppedimage_rect_color_crop
: debayered, rectified then croppedimage_raw_thresholded
: distorted, thresholdedimage_crop_thresholded
: distorted, cropped then thresholdedimage_rect_thresholded
: rectified then thresholdedimage_crop_rect_thresholded
: cropped then rectified and thresholdedimage_rect_crop_thresholded
: rectified then cropped and thresholdedimage_processed
: image after the final stage of preprocessing (used for RViz visualization)
For each /<camera namespace>/image_raw
topic a synchronized CameraInfo
message is published under the topic /<camera namespace>/camera_info
. The CameraInfo
message contains the information necessary to rectify the raw image according to the launch file arguments. We set the width
and height
fields of the CameraInfo
messages to that of the rectified resolution. This violates the ROS standard which does not allow for a differing resolution between raw and rectified images (see ROS REP 104), but this is required for rectifying stereo images with different raw resolutions. We also use it here to support arbitrary rectified resolutions (i.e., the resolution_rectified
launch file argument described above).
The pose_visualization.py
script, run by the preprocessing_and_visualization.launch
launch file, broadcasts transforms through TF2 for the base_link
and, optionally, the camera housing frames (if the path_measured_extrinsics
filepath is provided). It also publishes NavSatFix messages for visualization with rviz_satellite
. It is consistent with ROS REP 103 and ROS REP 105, with the exception that we do not use an odom
frame here. Specifically, the coordinate frame tree from parent to child is as follows:
earth
: Earth Centered Earth Fixed (ECEF) framemap
: East North Up (ENU) frame centered at the firstbase_link
position processed bypose_visualization.py
base_link
: origin on the ground at the midpoint of the vehicle's rear wheels with the x-axis pointing forward, y-axis pointing left, and z-axis pointing up<camera namespace>_housing
: camera housing frames (e.g.mono_left_housing
), defined in the measured extrinsics files
The image_preprocessing.py
script, described in the previous subsection, also broadcasts the chain of transforms between the camera optical frames described in the calibration results file. If the visualize_calibrated_transforms
launch file argument is set to true
, it additionally broadcasts a transform between the adk_left_housing
and adk_left_optical
frames.