os_driver activating... followed by process has died

[chadrs2]

When I am running the ouster_ros driver launch file in ros2-humble, it starts up normally connecting to the sensor and then in the terminal when it comes to activating the sensor it shuts down the driver node

[os_driver-2] [INFO] [1727995391.847437646] [ouster.os_driver]: Retrived sensor active config
[os_driver-2] [INFO] [1727995391.847511566] [ouster.os_driver]: Will send UDP data to 10.42.0.1
[os_driver-2] [INFO] [1727995391.847523008] [ouster.os_driver]: Contacting sensor 10.42.0.89 ...
[os_driver-2] [INFO] [1727995391.863511634] [ouster.os_driver]: Sensor 10.42.0.89 configured successfully
[os_driver-2] [INFO] [1727995391.863533340] [ouster.os_driver]: Starting sensor 10.42.0.89 initialization... Using ports: 7502/7503
[os_driver-2] [2024-10-03 22:43:11.863] [ouster::sensor] [info] initializing sensor client: 10.42.0.89 expecting lidar port/imu port: 7502/7503
[os_driver-2] [2024-10-03 22:43:11.937] [ouster::sensor] [info] parsing non-legacy metadata format
[os_driver-2] [INFO] [1727995391.938583205] [ouster.os_driver]: No metadata file was specified, using: 10.42.0-metadata.json
[os_driver-2] [INFO] [1727995391.938716769] [ouster.os_driver]: Wrote sensor metadata to 10.42.0-metadata.json
[os_driver-2] [INFO] [1727995391.938736095] [ouster.os_driver]: ouster client version: 0.11.1+483206f-release
[os_driver-2] product: OS-1-128, sn: 122426000526, firmware rev: v3.1.0
[os_driver-2] lidar mode: 1024x10, lidar udp profile: RNG19_RFL8_SIG16_NIR16
[os_driver-2] [INFO] [1727995391.939190116] [ouster.os_driver]: reset service created
[os_driver-2] [INFO] [1727995391.939500966] [ouster.os_driver]: get_metadata service created
[os_driver-2] [INFO] [1727995391.939704258] [ouster.os_driver]: get_config service created
[os_driver-2] [INFO] [1727995391.939912723] [ouster.os_driver]: set_config service created
[INFO] [launch.user]: os_driver activating...
[ERROR] [os_driver-2]: process has died [pid 9447, exit code -11, cmd '/docker_ros2_ws/install/ouster_ros/lib/ouster_ros/os_driver --ros-args -r __node:=os_driver -r __ns:=/ouster --params-file /docker_ros2_ws/install/ouster_ros/share/ouster_ros/config/driver_params.yaml'].

After colcon building my environment and source install/setup.bash, I simply run
ros2 launch ouster_ros driver.launch.py

I've edited the driver_params.yaml file with the correct IP address and set use_system_default_qos: false. Everything else is the same. I've done this before and it worked.

System
LiDAR: OS1-128
System: Ubuntu 22.04 in Docker

[mgrova]

Same problem here!

If I use the xml version, it works:

ros2 launch ouster_ros sensor.launch.xml sensor_hostname:=10.5.0.10 viz:=false udp_dest:=10.5.0.1 lidar_port:=7502 imu_port:=7503

Any idea about what could be happening?

Bests,
Marco.

[Samahu]

Hi, which version of the driver do you currently use? (note to self add the driver package/version to the log)

[chadrs2]

Where do I find that if it's a ROS2 package?

[chadrs2]

When I run ros2 pkg xml ouster_ros --tag version, I get version 0.13.0

[Samahu]

Can you checkout version 0.13.1 there was a problem reported in this issue which caused the driver to not apply launch file params specifed in the driver_params.yaml.

[chadrs2]

I get the same error with the new version

[os_driver-2] [INFO] [1728326550.047323450] [ouster.os_driver]: Will use automatic UDP destination
[os_driver-2] [INFO] [1728326550.047358267] [ouster.os_driver]: Contacting sensor 10.42.0.89 ...
[os_driver-2] [INFO] [1728326550.085492051] [ouster.os_driver]: Sensor 10.42.0.89 configured successfully
[os_driver-2] [INFO] [1728326550.085542937] [ouster.os_driver]: Starting sensor 10.42.0.89 initialization... Using ports: 0/0
[os_driver-2] [2024-10-07 18:42:30.085] [ouster::sensor] [info] initializing sensor client: 10.42.0.89 expecting lidar port/imu port: 0/0
[os_driver-2] [2024-10-07 18:42:30.085] [ouster::sensor] [info] (0 means a random port will be chosen)
[rviz2-1] [INFO] [1728326551.383241332] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-1] [INFO] [1728326551.383437922] [rviz2]: OpenGl version: 4.5 (GLSL 4.5)
[rviz2-1] [INFO] [1728326551.531269751] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-1] [INFO] [1728326551.937348032] [rviz2]: Stereo is NOT SUPPORTED
[os_driver-2] [2024-10-07 18:42:42.382] [ouster::sensor] [info] parsing non-legacy metadata format
[os_driver-2] [INFO] [1728326562.387811145] [ouster.os_driver]: No metadata file was specified, using: 10.42.0-metadata.json
[os_driver-2] [INFO] [1728326562.388235065] [ouster.os_driver]: Wrote sensor metadata to 10.42.0-metadata.json
[os_driver-2] [INFO] [1728326562.388310183] [ouster.os_driver]: ouster client version: 0.11.1+483206f-release
[os_driver-2] product: OS-1-128, sn: 122426000526, firmware rev: v3.1.0
[os_driver-2] lidar mode: 1024x10, lidar udp profile: RNG19_RFL8_SIG16_NIR16
[os_driver-2] [INFO] [1728326562.403405172] [ouster.os_driver]: reset service created
[os_driver-2] [INFO] [1728326562.405358107] [ouster.os_driver]: get_metadata service created
[os_driver-2] [INFO] [1728326562.407910454] [ouster.os_driver]: get_config service created
[os_driver-2] [INFO] [1728326562.409081627] [ouster.os_driver]: set_config service created
[INFO] [launch.user]: os_driver activating...
[ERROR] [os_driver-2]: process has died [pid 1339, exit code -11, cmd '/docker_ros2_ws/install/ouster_ros/lib/ouster_ros/os_driver --ros-args -r __node:=os_driver -r __ns:=/ouster --params-file /docker_ros2_ws/install/ouster_ros/share/ouster_ros/config/driver_params.yaml'].

and the version I have is 0.13.1: ros2 pkg xml ouster_ros --tag version 0.13.1

[Samahu]

I noticed from the logs that you are spinning rviz in docker, unless you configured the docker properly to spin the GUI application then this is likely to cause a problem. Can you disable rviz from your launch command, and check if you are able to reproduce the problem afterwards. you can do so using:

ros2 launch ouster_ros driver.launch.py params_file:=<...> viz:=false

[chadrs2]

I got the same error when setting viz:=false. It used to work in my docker container and display the live stream LiDAR data in ros2. It just suddenly stopped. I'm still getting the same error.

ros2 launch ouster_ros driver.launch.py viz:=false
[INFO] [launch]: All log files can be found below /root/.ros/log/2024-10-07-20-31-49-834607-CB490-EE11723-18877
[INFO] [launch]: Default logging verbosity is set to INFO
/docker_ros2_ws/install/ouster_ros/share/ouster_ros
/docker_ros2_ws/install/ouster_ros/share/ouster_ros/config/viz.rviz
[INFO] [os_driver-1]: process started with pid [18888]
[os_driver-1] [WARN] [1728333109.937227903] [ouster.os_driver]: lidar port set to zero, the client will assign a random port number!
[os_driver-1] [WARN] [1728333109.937292500] [ouster.os_driver]: imu port set to zero, the client will assign a random port number!
[os_driver-1] [INFO] [1728333110.165197522] [ouster.os_driver]: Will use automatic UDP destination
[os_driver-1] [INFO] [1728333110.165323389] [ouster.os_driver]: Contacting sensor 10.42.0.89 ...
[os_driver-1] [INFO] [1728333110.205681268] [ouster.os_driver]: Sensor 10.42.0.89 configured successfully
[os_driver-1] [INFO] [1728333110.205831471] [ouster.os_driver]: Starting sensor 10.42.0.89 initialization... Using ports: 0/0
[os_driver-1] [2024-10-07 20:31:50.205] [ouster::sensor] [info] initializing sensor client: 10.42.0.89 expecting lidar port/imu port: 0/0
[os_driver-1] [2024-10-07 20:31:50.206] [ouster::sensor] [info] (0 means a random port will be chosen)
[os_driver-1] [2024-10-07 20:32:02.492] [ouster::sensor] [info] parsing non-legacy metadata format
[os_driver-1] [INFO] [1728333122.495792256] [ouster.os_driver]: No metadata file was specified, using: 10.42.0-metadata.json
[os_driver-1] [INFO] [1728333122.495966478] [ouster.os_driver]: Wrote sensor metadata to 10.42.0-metadata.json
[os_driver-1] [INFO] [1728333122.495987888] [ouster.os_driver]: ouster client version: 0.11.1+483206f-release
[os_driver-1] product: OS-1-128, sn: 122426000526, firmware rev: v3.1.0
[os_driver-1] lidar mode: 1024x10, lidar udp profile: RNG19_RFL8_SIG16_NIR16
[os_driver-1] [INFO] [1728333122.496568409] [ouster.os_driver]: reset service created
[os_driver-1] [INFO] [1728333122.496927694] [ouster.os_driver]: get_metadata service created
[os_driver-1] [INFO] [1728333122.497161732] [ouster.os_driver]: get_config service created
[os_driver-1] [INFO] [1728333122.497411519] [ouster.os_driver]: set_config service created
[INFO] [launch.user]: os_driver activating...
[ERROR] [os_driver-1]: process has died [pid 18888, exit code -11, cmd '/docker_ros2_ws/install/ouster_ros/lib/ouster_ros/os_driver --ros-args -r __node:=os_driver -r __ns:=/ouster --params-file /docker_ros2_ws/install/ouster_ros/share/ouster_ros/config/driver_params.yaml'].

[Samahu]

I am unable to re-produce the problem.
Can you check if your sensor has any active errors? Do you have another sensor that you can test with?
Can you try to re-produce without docker? Was this working fine on a previous version with the same sensor (I need to know if this is a sensor problem vs a code problem).

We may need to enable GDB to look into what is causing the failure.

[chadrs2]

I can't re-produce without docker, since my docker is running Ubuntu 22 to allow for ros2 humble, while my host computer is running Ubuntu 20. This exact same launch file was working fine with the same sensor. We do have an OS0 128 beam that I can try this on and see if I get the same errors.

[apric0ts]

Can you try to re-produce without docker?

I am also running into the same problem on Ubuntu 22.04, ros2 humble, no container solution. Using OS1 128

Was this working fine on a previous version with the same sensor

Yes it was

[Samahu]

I am also running into the same problem on Ubuntu 22.04, ros2 humble, no container solution. Using OS1 128

@apric0ts are you running version 0.13.1 of the package?

[apric0ts]

I had tried the ros2 and master branches on the same day this issue was posted, and did not work. I haven't gotten around to trying the 0.13.1 package version.

[chadrs2]

I just ran the same script with our OS0 LiDAR and I got the same issue still using version 0.13.1. This may or may not be related, but my ouster-cli has also stopped working in the docker container. It still works on my main computer. When I run ouster-cli source <ip_address> viz I get this error (for both LiDARs):

Contacting sensor 10.42.0.250...
Initializing connection to sensor 10.42.0.250 on lidar port 37770 with udp dest '10.42.0.1'...
Failed to create scan_source for url ['10.42.0.250']
more details: locale::facet::_S_create_c_locale name not valid

I can still ping the sensors on the IP addresses

[chadrs2]

Nvm, that locale error was something else that I fixed, but the ros driver still has the same error.

[Samahu]

@chadrs2 Can you share the params file used when launching the driver? also could you try what @mgrova suggested with regards to using the xml file and check if this launch configuration has the same problem in your case or not?

[chadrs2]

It works when I run the .xml launch file.

Here's my params file:

ouster/os_driver:
  ros__parameters:
    # sensor_hostname[required]: hostname or IP address of the sensor (IP4 or
    # IP6).
    sensor_hostname: '10.42.0.89'
    # udp_dest[optional]: hostname or multicast group IP where the sensor will
    # send UDP data packets.
    udp_dest: ''
    # mtp_dest[optional]: hostname IP address for receiving data packets via
    # multicast, by default it is INADDR_ANY, so packets will be received on
    # first available network interface.
    mtp_dest: ''
    # mtp_main[optional]: if true, then configure and reinit the sensor,
    # otherwise start client with active configuration of sensor
    mtp_main: false
    # lidar_mode[optional]: resolution and rate; possible values: { 512x10,
    # 512x20, 1024x10, 1024x20, 2048x10, 4096x5 }. Leave empty to remain on
    # current the lidar mode.
    lidar_mode: ''
    # timestamp_mode[optional]: method used to timestamp measurements; possible
    # values:
    # - TIME_FROM_INTERNAL_OSC
    # - TIME_FROM_SYNC_PULSE_IN
    # - TIME_FROM_PTP_1588
    # - TIME_FROM_ROS_TIME: This option uses the time of reception of first
    #                       packet of a LidarScan as the timestamp of the IMU,
    #                       PointCloud2 and LaserScan messages.
    timestamp_mode: ''
    # ptp_utc_tai_offset[optional]: UTC/TAI offset in seconds to apply when
    # TIME_FROM_PTP_1588 timestamp mode is used.
    ptp_utc_tai_offset: -37.0
    # udp_profile_lidar[optional]: lidar packet profile; possible values:
    # - LEGACY: not recommended
    # - RNG19_RFL8_SIG16_NIR16
    # - RNG15_RFL8_NIR8
    # - RNG19_RFL8_SIG16_NIR16_DUAL
    # - FUSA_RNG15_RFL8_NIR8_DUAL
    udp_profile_lidar: ''
    # metadata[optional]: path to save metadata file to, if left empty a file
    # with the sensor hostname or ip will be crearted in ~/.ros folder.
    metadata: ''
    # lidar_port[optional]: port value should be in the range [0, 65535]. If you
    # use 0 as the port value then the first avaliable port number will be
    # assigned.
    lidar_port: 0
    # imu_port[optional]: port value should be in the range [0, 65535]. If you
    # use 0 as the port value then the first avaliable port number will be
    # assigned.
    imu_port: 0
    # sensor_frame[optional]: name to use when referring to the sensor frame.
    sensor_frame: os_sensor
    # lidar_frame[optional]: name to use when referring to the lidar frame.
    lidar_frame: os_lidar
    # imu_frame[optional]: name to use when referring to the imu frame.
    imu_frame: os_imu
    # point_cloud_frame[optional]: which frame of reference to use when
    # generating PointCloud2 or LaserScan messages, select between the values of
    # lidar_frame and sensor_frame.
    point_cloud_frame: os_lidar 
    # proc_mask[optional]: use any combination of the 4 flags IMG, PCL, IMU and
    # SCAN to enable or disable their respective messages.
    proc_mask: IMU|PCL|SCAN|IMG|RAW
    # scan_ring[optional]: use this parameter in conjunction with the SCAN flag
    # to select which beam of the LidarScan to use when producing the LaserScan
    # message. Choose a value the range [0, sensor_beams_count).
    scan_ring: 0
    # use_system_default_qos[optional]: if false, data are published with sensor
    # data QoS. This is preferrable for production but default QoS is needed for
    # rosbag. See: https://github.com/ros2/rosbag2/issues/125
    use_system_default_qos: true
    # point_type[optional]: choose from: {original, native, xyz, xyzi, xyzir}
    # Here is a breif description of each option:
    #  - original: This uses the original point representation ouster_ros::Point
    #          of the ouster-ros driver.
    #  - native: directly maps all fields as published by the sensor to an
    #          equivalent point cloud representation with the additon of ring
    #          and timestamp fields.
    #  - xyz: the simplest point type, only has {x, y, z}
    #  - xyzi: same as xyz point type but adds intensity (signal) field. this
    #          type is not compatible with the low data profile.
    #  - xyzir: same as xyzi type but adds ring (channel) field.
    #          this type is same as Velodyne point cloud type
    #          this type is not compatible with the low data profile.
    # for more details about the fields of each point type and their data refer
    # to the following header files:
    # - ouster_ros/os_point.h
    # - ouster_ros/sensor_point_types.h
    # - ouster_ros/common_point_types.h.
    point_type: original
    # azimuth window start[optional]: values range [0, 360000] millidegrees
    azimuth_window_start: 0
    # azimuth_window_end[optional]: values range [0, 360000] millidegrees
    azimuth_window_end: 360000
    # persist_config[optional]: request the sensor to persist settings
    persist_config: false
    # attempt_config[optional]: attempting to reconnect to the sensor after
    # connection loss or sensor powered down
    attempt_reconnect: false
    # dormant_period_between_reconnects[optional]: wait time in seconds between
    # reconnection attempts
    dormant_period_between_reconnects: 1.0
    # max_failed_reconnect_attempts[optional]: maximum number of attempts trying
    # to communicate with the sensor. Counter resets upon successful connection.
    max_failed_reconnect_attempts: 2147483647
    # organized[optional]: whether to generate an organized point cloud. default
    # is organized.
    organized: true
    # destagger[optional]: enable or disable point cloud destaggering, default enabled.
    destagger: true
    # min_range[optional]: minimum lidar range to consider (meters).
    min_range: 0.0
    # max_range[optional]: maximum lidar range to consider (meters).
    max_range: 1000.0

[chadrs2]

Is this what it should look like if I am on the correct version?

git branch -a
* (HEAD detached at ros2-v0.13.1)
  ros2
  remotes/origin/HEAD -> origin/master
  remotes/origin/ROS-119-driver-reconnection-behavior-ros2
  remotes/origin/ROS-175-reduce-vertical-resolution-for-improving-performance-of-the-driver
  remotes/origin/ROS-250-support-non-truncated-range-image
  remotes/origin/ROS-287-ouster-support-for-appollo
  remotes/origin/ROS-296_fix_spelling_mistakes_FOXY
  remotes/origin/gh-pages
  remotes/origin/iron-devel
  remotes/origin/master
  remotes/origin/rolling-devel
  remotes/origin/ros2
  remotes/origin/ros2-foxy
  remotes/origin/ros2-foxy-sparse-neighbor-culling
  remotes/origin/ros2-humble-sparse-neighbor-culling

[Samahu]

Thanks for sharing I am able to reproduce the problem and looking into it..

[chadrs2]

I just got it fixed! In my driver_params.yaml file I had to change the proc_mask argument from IMU|PCL|SCAN|IMG|RAW to IMU|PCL|SCAN|IMG. So I removed the RAW argument. I don't know why that was there because in the commented text above it says to use any combination of the 4 flags, not saying anything about RAW.

[Samahu]

I just got it fixed! In my driver_params.yaml file I had to change the proc_mask argument from IMU|PCL|SCAN|IMG|RAW to IMU|PCL|SCAN|IMG. So I removed the RAW argument. I don't know why that was there because in the commented text above it says to use any combination of the 4 flags, not saying anything about RAW.

The RAW flag was added recently, it allows users to be able to still obtain raw packets while using the driver config but apparently there is a flow in how this flag implemented .. applogies for any inconvenience .. a hotfix for the problem will be posted today.

[Samahu]

resolved by #384