robotics datasets

Curation effort to document and consolidate as exhaustive as possible a list of datasets that may be useful for robotics research.

• motivation:
  • deep learning/llms for robotics is fundamentally data hungry
  • lots of distributed datasets shared generously by different robotics labs/ institutions
    • not many efforts at consolidation
  • make it easier and more accessible to inspect different robotics datasets
  • multiple heterogenous datasets for different hardware, settings, environment

DLR

• Scientific datasets list

datasets

• Benchmark Maps

  • 2.5D elevation maps of planetary environment that were collected on Mt. Etna during the space-analogous ARCHES mission. In addition to the raw elevation maps, we provide cost maps that encode the traversibility of the terrain.

• CROOS-CV

  • The CROOS-CV dataset is intended to support and benchmark Computer Vision (CV) development for Close Range On-Orbit Servicing (CROOS). It is an representative image dataset for CROOS operations with distances of 2 m between servicer and client satellite that was recorded under illumination conditions similar to a Low Earth Orbit.

• HOWS-CL-25

  • HOWS-CL-25 is a synthetic dataset especially designed for object classification on mobile robots operating in a changing environment (like a household), where it is important to learn new, never seen objects on the fly.

• Long Range Navigation Tests (LRNTs)

  • During the ROBEX demo mission space campaign that took place during June–July 2017 on Mt. Etna, Italy, we performed some Long Range Navigation Tests.

• Morocco Navigation Tests

  • The Institute was part of the PERASPERA Space Robotics Technologies Cluster in the operational grants OG3 and OG6. In that context, the research group participated in the 2018 November/December field test in the Moroccan desert close to the city of Erfoud.

• MMX Navigation Testing Data Set

  • Public collection of test data that is used to test the DLR Autonomous Navigation Experiment on the MMX Rover for Phobos

• Planetary Stereo Solid-State LiDAR Inertial Dataset

  • we release a dataset recorded on the Moon-like environment of Mount Etna, Sicily, with a sensor setup that comprises a stereo camera, a LiDAR and an IMU.

• ReSyRIS

  • The Real-Synthetic Rock Instance Segmentation dataset (ReSyRIS) is created for training and evaluation of rock segmentation, detection and instance segmentation in (quasi-)extra-terrestrial environments.

• STIOS

  • The Stereo Instances on Surfaces Datensatz (STIOS) is created for evaluation of instance-based algorithm and mainly intended for robotic applications, which is why the dataset refers to horizontal surfaces.

• THR Dataset

  • The THR (Top Hat Rail) data set consists of color and depth images from different objects taken from multiple views in different scenes. The data set consists of 9 object classes and can be used e.g. to improve perception algorithms by learning.

Datasets for Autonomous Systems and Robotics

• reference: Datasets for Autonomous Systems and Robotics

• Apollo Scape

• Agarwal etc. al (2020): Ford Multi-AV Seasonal Dataset.

  • arXiv:2003.07969,
  • Ford AV Datasets

• Behley et al. (2019): SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences. arXiv: (https://arxiv.org/abs/1904.01416)1904.01416

• Bosch Small Traffic Lights Dataset

  • Object detection dataset for traffic lights in road recordings

• Braun et al. (2019): The EuroCity Persons Dataset.

  • link

• Caesar et al. (2019): nuScenes: A multimodal dataset for autonomous driving.

  • arXiv:1903.11027 ;
  • nuScenes
    • includes RADAR data as well

• CARLA Imitation learning

• Change et al. (2019): Argoverse:

  • 3D Tracking and Forecasting with Rich Maps. CVPR 2019. PDF

• Chen et al. (2016): Anticipating Accidents in Dashcam Videos. ACCV 2016 Oral.

• Chilean Underground Mine Dataset

• Cityscapes

• Déziel et al. (2021): PixSet : An Opportunity for 3D Computer Vision to Go Beyond Point Clouds With a Full-Waveform LiDAR Dataset.

  • arXiv:2102.12010,
  • Leddar PixSet

• Geyer et al. (2020): A2D2: Audi Autonomous Driving Dataset.

  • arXiv:2004.06320;
  • A2D2

• Kesten et al. (2019): Lyft Level 5 AV Dataset 2019

• KITTI Vision Benchmark Suite

• Maddern et al. (2016): 1 Year, 1000km: The Oxford RobotCar Dataset.

  • The International Journal of Robotics Research (IJRR), 2016. pdf

• Mallios et al. (2017): Underwater caves sonar and vision data set.

  • The International Journal of Robotics Research, 2017 (36), 1247-1251. doi: 10.1177/0278364917732838

• Mapillary Vistas

• PandaSet by Hesai and Scale AI

• Schafer et al. (2018): A Commute in Data: The comma2k19 Dataset. arXiv:1812.05752;

  • comma2k19

• Udacity Self-Driving Car Dataset ;

  • relabeled on roboflow

• Waymo dataset

• Yu et al. (2018): BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling.

  • arXiv:1805.04687;
  • Berkeley DeepDrive

Open X-Embodiment: Google

• Open X-Embodiment: Robotic Learning Datasets and RT-X Models
  • dataset details link
  • code link

awesome-robotics-datasets github list

• https://github.com/mint-lab/awesome-robotics-datasets/blob/master/README.md

Dataset Collections

• Robotics
  • Radish: The Robotics Data Set Repository, Andrew Howard and Nicholas Roy (Not working)
  • Repository of Robotics and Computer Vision Datasets, MRPT
    • 📝 It includes Malaga datasets and some of classic datasets published in Radish.
  • IJRR Data Papers, IJRR
  • Awesome SLAM Datasets, Younggun Cho 👍
• Computer Vision
  • CVonline Image Databases, CVonline
  • Computer Vision Datasets on the Web, CVPapers 👍
  • YACVID: Yet Another Computer Vision Index To Datasets, Hayko Riemenschneider 👍
  • Computer Vision Online Datasets, Computer Vision Online
• Others
  • Machine Learning Repository, UCI
  • Kaggle Datasets, Kaggle
  • IEEE DataPort, IEEE

Place-specific Datasets

Driving Datasets

• KITTI Vision Benchmark Suite and KITTI-360, Andreas Geiger et al. 👍
  • SemanticKITTI, Jens Behley et al.
• Waymo Open Dataset, Waymo
• Cityscapes Dataset
• AppoloScape Dataset
• Berkely DeepDrive Dataset (BDD100K), BAIR at UC Berkely
• nuScenes Dataset, APTIV
• $D^2$-City Dataset, DiDi
• Ford Campus Vision and Lidar Data Set, PeRL at Univ. of Michigan
• MIT DARPA Urban Challenge Dataset, MIT
• KAIST Multi-spectral Recognition Dataset in Day and Night, RCV Lab at KAIST
• KAIST Complex Urban Dataset, IRAP Lab at KAIST
• New College Dataset, MRG at Oxford Univ.
• Chinese Driving from a Bike View (CDBV), CAS
• CULane Dataset, CUHK
• ROMA (ROad MArkings) Image Database, Jean-Philippe Tarel et al.

Flying Datasets

• The Zurich Urban Micro Aerial Vehicle Dataset, RPG at ETHZ
• The UZH-FPV Drone Racing Dataset, RPG at ETHZ
• MultiDrone Public Dataset, MultiDrone Project
• The Blackbird Dataset, AgileDrones Group at MIT

Underwater Datasets

• Marine Robotics Datasets, ACFR

Outdoor Datasets

• The Rawseeds Project
  • 📝 It includes Bovisa dataset is for outdoor and Bicocca dataset is for indoor.
• Planetary Mapping and Navigation Datasets, ASRL at Univ. of Toronto

Indoor Datasets

• Robotics 2D-Laser Datasets, Cyrill Stachniss
  • 📝 It includes some of classic datasets published in Radish.
• Long-Term Mobile Robot Operations, Lincoln Univ.
• MIT Stata Center Data Set, Marine Robotics Group at MIT
• KTH and COLD Database, Andrzej Pronobis
• Shopping Mall Datasets, IRC at ATR
• RGB-D Dataset 7-Scenes, Microsoft

Topic-specific Datasets for Robotics

Localization, Mapping, and SLAM

• SLAM Benchmarking, AIS at Univ. of Freiburg
• Robotic 3D Scan Repository, Univ. of Wurzburg and Univ. of Osnabruck
• 3D Pose Graph Optimization, Luca Carlone
• Landmark-based Localization
  • Range-only Data for Localization, CMU RI
  • Roh's Angulation Dataset, HyunChul Roh
  • Wireless Sensor Network Dataset, Kamin Whitehouse

Path Planning and Navigation

• Pathfinding Benchmarks, Moving AI Lab at Univ. of Denver
• Task and Motion Planner Benchmarking, RSS 2018 Workshop

Topic-specific Datasets for Computer Vision

Features

• Affine Covariant Features Datasets, VGG at Oxford
  • Repeatability Benchmark Tutorial, VLFeat
• A list of feature performance evaluation datasets, maintained by openMVG

Saliency and Foreground

• Saliency
  • MIT Saliency Benchmark, MIT
  • Salient Object Detection: A Benchmark, Ming-Ming Cheng
• Foreground/Change Detection (Background Subtraction)
  • ChangeDetection.NET (a.k.a. CDNET)

Motion and Pose Estimation

• AdelaideRMF: Robust Model Fitting Data Set, Hoi Sim Wong

Structure-from-Motion and 3D Reconstruction

• Objects
  • IVL-SYNTHESFM v2, Davide Marelli et al.
  • Fuji-SfM Dataset, Jordi Gene-Mola et al.
  • Large Geometric Models Archive, Georgia Tech
  • The Stanford 3D Scanning Repository, Stanford Univ.
• Places
  • Photo Tourism Data, UW and Microsoft

Object Tracking

• Visual Object Tracking Challenge (a.k.a. VOT) 👍
• Visual Tracker Benchmark (a.k.a. OTB)

Object, Place, and Event Recognition

• Pedestrians
  • EuroCity Persons Dataset (a.k.a. ECP)
  • Daimler Pedestrian Benchmark Data Sets
  • CrowdHuman
• Objects
  • RGB-D Object Dataset, UW
  • Sweet Pepper and Peduncle 3D Datasets, InKyu Sa
• Places
  • Loop Closure Detection, David Filliat et. al.
• Traffic and Surveillance
  • BEST: Benchmark and Evaluation of Surveillance Task, SJTU
  • VIRAT Video Dataset

Research Groups

• TUM CVG Datasets
  • Tags: Visual(-inertia) odometry, visual SLAM, 3D reconstruction
• Oxford VGG Datasets
  • Tags: Visual features, visual recognition, 3D reconstruction
• QUT CyPhy Datasets
  • Tags: Visual SLAM, LiDAR SLAM
• Univ. of Bonn Univ. Stachniss Lab Datasets
  • Tags: SLAM
• EPFL CVLAB Datasets
  • Tags: 3D reconstruction, local keypoint, optical flow, RGB-D pedestrian
• The Middlebury Computer Vision Pages
  • Tags: Stereo matching, 3D reconstruction, MRF, optical flow, color
• Caltech CVG Datasets
  • Tags: Objects (pedestrian, car, face), 3D reconstruction (on turntables)