Skip to content
Snippets Groups Projects
neeloyc2's avatar
neeloyc2 authored
75044d3d
History

Project Code for Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperation

Note: This repository contains the code for our ICRA 2025 paper. For more details, please refer to the project website and arXiv preprint. For issues, contact Neeloy Chakraborty at neeloyc2@illinois.edu.

Offline evaluation of our method compared with baselines on a middle stage growth video. Real-time online evaluation of our method on an early stage growth video.

Abstract

Teleoperation is an important technology to enable supervisors to control agricultural robots remotely. However, environmental factors in dense crop rows and limitations in network infrastructure hinder the reliability of data streamed to teleoperators. These issues result in delayed and variable frame rate video feeds that often deviate significantly from the robot's actual viewpoint. We propose a modular learning-based vision pipeline to generate delay-compensated images in real-time for supervisors. Our extensive offline evaluations demonstrate that our method generates more accurate images compared to state-of-the-art approaches in our setting. Additionally, ours is one of the few works to evaluate a delay-compensation method in outdoor field environments with complex terrain on data from a real robot in real-time. Resulting videos and code are provided here.

Getting Started

File Structure

Below, we give a high-level overview of the directory structure of our project.

./
└───depth
|   └───collect_eval_metrics - Code to evaluate performance of depth models
|   └───dav2 - Code to train DAv2 models and save depth predictions
|   └───dino - Code to train DINOv2 models and save depth predictions
└───inpainting
|   └───eval_inpaint.py - Script to evaluate performance of inpainting models
|   └───save_resnet_inpaint.py - Script to save inpainted predictions from trained ResNet model
|   └───save_telea_inpaint.py - Script to save inpainted predictions using Telea method
|   └───train_resnet_inpaint.py - Script to train ResNet inpainting model
└───ros/catkin_ws/src
|   └───combined_node - ROS node code to perform real-time delay compensation
|   └───data_collection - Code to generate dataset of frames from recorded SVO files
|   └───rtab_mapping - Code with launch files to play back SVO files and visualize topics in RViz
|   └───zed-ros-wrapper - ROS nodes for Stereolabs ZED SDK
└───weights
    └───depth - Weights for depth models
    └───inpainting - Weights for inpainting models

Installation for Dataset Creation and Real-Time Evaluation with Pulsar

  • Using Ubuntu 20.04 system with CUDA 12.4 GPU (we used NVIDIA GeForce RTX 2080 8GB)
  • Create Conda environment with Python 3.9
    • conda create -n "terra-torch3d" python=3.9
      conda activate terra-torch3d
  • Install required packages for Conda environment
    • Install packages using our environment yaml file here, EXCEPT for PyTorch3D
    • Install our fork of PyTorch3D from source by following instructions here
  • Install CUDA 11.7 user-wide (this installed CUDA may be different from the one referenced in the conda environment, and it is used by the ZED SDK)
    • Follow the instructions here to download and install
    • Add necessary lines to ~/.bashrc to ensure correct paths are sourced (modify depending on where it is installed)
      • export PATH=/usr/local/cuda-11.7/bin${PATH:+:${PATH}}
        export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda-11.7/lib64
  • Install ZED SDK 3.7
    • Download SDK from here
    • Follow instructions here to install the SDK
  • Install ROS Noetic and dependencies
  • Clone repository and setup ROS workspace
    • Ensure GIT LFS is installed
      • To download weights alongside code, run git lfs install
      • To ignore weights and only pull code, run git lfs install --skip-smudge
    • git clone https://gitlab.engr.illinois.edu/hubris/decompteleop.git
      cd decompteleop/ros/catkin_ws
      catkin make

Collecting Offline Dataset

Our dataset is released online and can be downloaded here. We generated our dataset by collecting samples from the Terrasentia Dataset found here.

We also provide instructions for how to curate your own dataset containing RGB, depth, and pose frames. Follow the instructions here to collect your own dataset.

Training and Evaluating Depth Models

Pretrained weights for the depth models are located at weights/depth. Open the following docs to learn how to train your own depth models:

Open the doc here to learn how to evaluate the accuracy of depth predictions.

Training and Evaluating Inpainting Models

Pretrained weights for the inpainting models are located at weights/inpainting. Open the doc here to learn how to train and evaluate your own inpainting models.

Real-time ROS Evaluation

Open the doc here to learn how to run our real-time delay compensation ROS node.

References

We thank the authors of the following projects for making their data/code/models available (in no particular order):

Citation

@INPROCEEDINGS{chakraborty2025towards,
  author={Chakraborty, Neeloy and Fang, Yixiao and Schreiber, Andre and Ji, Tianchen and Huang, Zhe and Mihigo, Aganze and Wall, Cassidy and Almana, Abdulrahman and Driggs-Campbell, Katherine},
  booktitle={2025 International Conference on Robotics and Automation (ICRA)}, 
  title={Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperation}, 
}