News

Selected Publications

All Publications

This work introduces Robust Metric Visual Odometry (RoMeO), a novel monocular RGB visual odometry (VO) method that estimates camera poses from video without requiring IMUs or 3D sensors. RoMeO improves robustness, generalization, and metric-scale pose recovery by leveraging pre-trained depth models, combining monocular depth with multi-view stereo (MVS) for accurate correspondence and optimization, and applying effective training techniques. It reduces trajectory errors by over 50% compared to SOTA across multiple datasets and enhances SLAM pipelines with global bundle adjustment and loop closure.
arXiv’24
Details PDF Bibtex

Mesh2NeRF introduces an analytic method to derive ground-truth radiance fields directly from textured meshes, enhancing 3D generation tasks by providing precise supervision for training generative NeRFs and single scene representations. This approach addresses artifacts from traditional methods and demonstrates significant improvements in view synthesis and mesh extraction across various datasets.
ECCV’24
Details PDF Video Project Bibtex

LiSA introduces semantic awareness into Scene Coordinate Regression (SCR) for LiDAR localization, enhancing robustness and accuracy by addressing challenges posed by dynamic objects and repetitive structures. Through knowledge distillation from a segmentation model, LiSA improves performance without additional computation during inference.
CVPR’24
Details PDF Code Bibtex

L-MAGIC introduces a method that utilizes large language models to guide the generation of coherent 360-degree panoramic scenes from a single input image. By leveraging pre-trained diffusion and language models without fine-tuning, L-MAGIC achieves zero-shot performance, producing panoramic scenes with improved layouts and rendering quality, as demonstrated by extensive experiments and human evaluations.
CVPR’24
Details PDF Video Code Project Bibtex Demo

This study introduces an online evaluation protocol for Test Time Adaptation (TTA) methods, emphasizing the impact of computational speed on adaptation performance. By simulating real-world data streams, the research demonstrates that faster, simpler TTA methods can outperform more complex, slower ones when inference speed is considered, highlighting the need for efficient and effective adaptation techniques.
ICML’24
Details PDF Code Bibtex Poster

OpenBot-Fleet is an open-source cloud robotics system utilizing smartphones and low-cost wheeled robots for navigation. By leveraging cloud storage and computation, it enables the collection and learning of navigation policies, achieving over 80% success in unseen environments, thus advancing scalable and cost-effective deployment of learning robot fleets.
ICRA’24
Details PDF Video Code Project Bibtex

This work introduces SimCS, a parameter-free method that leverages simulated data to regularize continual learning in Online Domain-Incremental Continual Segmentation (ODICS). Experiments demonstrate that SimCS consistently enhances performance when integrated with various continual learning methods.
AAAI’24
Details PDF Slides Bibtex Poster

This work introduces GIM, a self-training framework for generalizable image matching. Utilizing diverse internet videos, GIM generates supervision signals to improve state-of-the-art architectures in zero-shot and cross-domain scenarios. It establishes ZEB, a zero-shot evaluation benchmark, while demonstrating significant gains in 3D reconstruction and visual localization tasks.
ICLR’24
Details PDF Video Code Project Bibtex Demo

E2PNet introduces a learning-based approach for event-to-point cloud registration, leveraging a novel Event-Points-to-Tensor (EP2T) network to encode event data into a 2D grid-shaped feature tensor. This facilitates the application of existing RGB-based frameworks to event data, enhancing robustness under extreme illumination and fast motion conditions.
NeurIPS’23
Details PDF Code Bibtex

A central question in robotics is how to design a control system for an agile mobile robot. This paper studies this question systematically, focusing on a challenging setting: autonomous drone racing. We show that a neural network controller trained with reinforcement learning (RL) outperformed optimal control (OC) methods in this setting. We then investigated which fundamental factors have contributed to the success of RL or have limited OC. Our study indicates that the fundamental advantage of RL over OC is not that it optimizes its objective better but that it optimizes a better objective.
SciRob’23
Details PDF Video Bibtex

Swift is an autonomous system that can race physical vehicles at the level of the human world champions. The system combines deep reinforcement learning (RL) in simulation with data collected in the physical world. Swift competed against three human champions, including the world champions of two international leagues, in real-world head-to-head races. Swift won several races against each of the human champions and demonstrated the fastest recorded race time. This work represents a milestone for mobile robotics and machine intelligence, which may inspire the deployment of hybrid learning-based solutions in other physical systems.
Nature’23
Details PDF Video Bibtex Nature Video

CLNeRF introduces continual learning to Neural Radiance Fields, enabling efficient updates and preventing catastrophic forgetting in scenes with changing appearance and geometry over time. It leverages generative replay and the Instant Neural Graphics Primitives architecture, validated on the proposed World Across Time dataset.
ICCV’23
Details PDF Video Code Dataset Bibtex

This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts. The LDM3D model is fine-tuned on a dataset of tuples containing an RGB image, depth map and caption, and validated through extensive experiments. We also develop an application called DepthFusion, which uses the generated RGB images and depth maps to create immersive and interactive 360-degree-view experiences using TouchDesigner. This technology has the potential to transform a wide range of industries, from entertainment and gaming to architecture and design. Overall, this paper presents a significant contribution to the field of generative AI and computer vision, and showcases the potential of LDM3D and DepthFusion to revolutionize content creation and digital experiences.
CVPRW’23
Details PDF Video Code Bibtex

We present a visual-inertial depth estimation pipeline that integrates monocular depth estimation and visual-inertial odometry to produce dense depth estimates with metric scale. Our approach performs global scale and shift alignment against sparse metric depth, followed by learning-based dense alignment. We evaluate on the TartanAir and VOID datasets, observing up to 30% reduction in inverse RMSE with dense scale alignment relative to performing just global alignment alone. Our approach is especially competitive at low density; with just 150 sparse metric depth points, our dense-to-dense depth alignment method achieves over 50% lower iRMSE over sparse-to-dense depth completion by KBNet, currently the state of the art on VOID. We demonstrate successful zero-shot transfer from synthetic TartanAir to real-world VOID data and perform generalization tests on NYUv2 and VCU-RVI. Our approach is modular and is compatible with a variety of monocular depth estimation models.
ICRA’23
Details PDF Video Code Short Talk Bibtex

This paper introduces ZoeDepth, a novel approach to single-image depth estimation that combines relative and metric depth frameworks. By pre-training on multiple datasets and fine-tuning on specific ones, ZoeDepth achieves state-of-the-art performance and exceptional zero-shot generalization across diverse indoor and outdoor datasets.
arXiv’23
Details PDF Code Model BibTeX

In this paper we train a contact-exploiting manipulation policy in simulation for the contact-rich household task of loading plates into a slotted holder, which transfers without any fine-tuning to the real robot. We investigate various factors necessary for this zero-shot transfer, like time delay modeling, memory representation, and domain randomization. Our policy transfers with minimal sim-to-real gap and significantly outperforms heuristic and learnt baselines. It also generalizes to plates of different sizes and weights.
ICRA’23
Details PDF Video Code Project Short Talk Bibtex

We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. We address training instabilities that frequently occur with APG through curriculum learning and experiment on a widely used controls benchmark, the CartPole, and two common aerial robots, a quadrotor and a fixed-wing drone. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics.
ICRA’23
Details PDF Code Short Talk Bibtex

​We propose an end-to-end approach that can autonomously fly quadrotors through complex natural and human-made environments at high speeds with purely onboard sensing and computation. The key principle is to directly map noisy sensory observations to collision-free trajectories in a receding-horizon fashion. This direct mapping drastically reduces processing latency and increases robustness to noisy and incomplete perception. The sensorimotor mapping is performed by a convolutional network that is trained exclusively in simulation via privileged learning: imitating an expert with access to privileged information. By simulating realistic sensor noise, our approach achieves zero-shot transfer from simulation to challenging real-world environments that were never experienced during training: dense forests, snow-covered terrain, derailed trains, and collapsed buildings. Our work demonstrates that end-to-end policies trained in simulation enable high-speed autonomous flight through challenging environments, outperforming traditional obstacle avoidance pipelines.
ScienceRobotics’21
Details PDF Video Code Project Bibtex

​In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of 87.74 ± 0.13 and 88.24 ± 0.15 on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude.
ICML’21
Details PDF Slides Code Project Bibtex

​Current robots are either expensive or make significant compromises on sensory richness, computational power, and communication capabilities. We propose to leverage smartphones to equip robots with extensive sensor suites, powerful computational abilities, state-of-the-art communication channels, and access to a thriving software ecosystem. We design a small electric vehicle that costs $50 and serves as a robot body for standard Android smartphones. We develop a software stack that allows smartphones to use this body for mobile operation and demonstrate that the system is sufficiently powerful to support advanced robotics workloads such as person following and real-time autonomous navigation in unstructured environments. Controlled experiments demonstrate that the presented approach is robust across different smartphones and robot bodies.
ICRA’21
Details PDF Video Code Project Bibtex

​We propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation. We train the policy entirely in simulation by leveraging demonstrations from an optimal controller that has access to privileged information. We use appropriate abstractions of the visual input to enable transfer to a real quadrotor. We show that the resulting policy can be directly deployed in the physical world without any fine-tuning on real data. Our methodology has several favorable properties: it does not require a human expert to provide demonstrations, it cannot harm the physical system during training, and it can be used to learn maneuvers that are challenging even for the best human pilots. Our approach enables a physical quadrotor to fly maneuvers such as the Power Loop, the Barrel Roll, and the Matty Flip, during which it incurs accelerations of up to 3g.
RSS’20
Details PDF Slides Video Code Project Bibtex

​We introduce sequential greedy architecture search (SGAS), an efficient method for neural architecture search. By dividing the search procedure into sub-problems, SGAS chooses and prunes candidate operations in a greedy fashion. We apply SGAS to search architectures for Convolutional Neural Networks (CNN) and Graph Convolutional Networks (GCN). Extensive experiments show that SGAS is able to find state-of-the-art architectures for tasks such as image classification, point cloud classification and node classification in protein-protein interaction graphs with minimal computational cost.
CVPR’20
Details PDF Slides Video Code Bibtex

We present a general framework for adversarial attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks. To do this, we re-frame the adversarial attack problem as learning a distribution of parameters that always fools the agent. In the semantic case, our proposed adversary (denoted as BBGAN) is trained to sample parameters that describe the environment with which the black-box agent interacts, such that the agent performs its dedicated task poorly in this environment. We apply BBGAN on three different tasks, primarily targeting aspects of autonomous navigation: object detection, self-driving, and autonomous UAV racing. On these tasks, BBGAN can generate failure cases that consistently fool a trained agent.
Poster at AAAI’20
Details PDF Bibtex

GCNs show promising results, but they are limited to very shallow models due to the vanishing gradient problem. As a result most state-of-the-art GCN algorithms are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We borrow concepts from CNNs, mainly residual/dense connections and dilated convolutions, and adapt them to GCN architectures. Through extensive experiments, we show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation.
Oral at ICCV’19
Details PDF Video Code Project Bibtex

We propose Observational Imitation Learning (OIL), a novel imitation learning variant that supports online training and automatic selection of optimal behavior by observing multiple imperfect teachers. We apply our proposed methodology to the challenging problems of autonomous driving and UAV racing. For both tasks, we utilize the Sim4CV simulator that enables the generation of large amounts of synthetic training data and also allows for online learning and evaluation. We train a perception network to predict waypoints from raw image data and use OIL to train another network to predict controls from these waypoints. Extensive experiments demonstrate that our trained network outperforms its teachers, conventional imitation learning (IL) and reinforcement learning (RL) baselines and even humans in simulation.
RSS’19
Details PDF Video Project Bibtex

In this paper, we propose learning an optimized controller using a DNN that fuses multiple controllers. The network learns a robust controller with online trajectory filtering, which suppresses noisy trajectories and imperfections of individual controllers. The result is a network that is able to learn a good fusion of filtered trajectories from different controllers leading to significant improvements in overall performance. We compare our trained network to controllers it has learned from, end-to-end baselines and human pilots in a realistic simulation; our network beats all baselines in extensive experiments and approaches the performance of a professional human pilot.
CVPRW’19 - UAVision’19
Details PDF Video Bibtex

We present TrackingNet, the first large-scale dataset and benchmark for object tracking in the wild. We provide more than 30K videos with more than 14 million dense bounding box annotations. Our dataset covers a wide selection of object classes in broad and diverse context. By releasing such a large-scale dataset, we expect deep trackers to further improve and generalize. In addition, we introduce a new benchmark composed of 500 novel videos, modeled with a distribution similar to our training dataset. By sequestering the annotation of the test set and providing an online evaluation server, we provide a fair benchmark for future development of object trackers. Deep trackers fine-tuned on a fraction of our dataset improve their performance by up to 1.6% on OTB100 and up to 1.7% on TrackingNet Test. We provide an extensive benchmark on TrackingNet by evaluating more than 20 trackers. Our results suggest that object tracking in the wild is far from being solved.
ECCV’18
Details PDF Code Project Bibtex

We present an approach for transferring driving policies from simulation to reality via modularity and abstraction. Our approach is inspired by classic driving systems and aims to combine the benefits of modular architectures and end-to-end deep learning approaches. The key idea is to encapsulate the driving policy such that it is not directly exposed to raw perceptual input or low-level vehicle dynamics. We evaluate the presented approach in simulated urban environments and in the real world. In particular, we transfer a driving policy trained in simulation to a 1/5-scale robotic truck that is deployed in a variety of conditions, with no finetuning, on two continents.
CoRL’18
Details PDF Video Bibtex

We present a photo-realistic training and evaluation simulator [Sim4CV](https://sim4cv.org/) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network (DNN) architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool.
IJCV’18
Details PDF Video Project Bibtex

A vehicle trained end-to-end to imitate an expert cannot be guided to take a specific turn at an upcoming intersection. This limits the utility of such systems. We propose to condition imitation learning on high-level command input. At test time, the learned driving policy functions as a chauffeur that handles sensorimotor coordination but continues to respond to navigational commands. We evaluate different architectures for conditional imitation learning in vision-based driving. We conduct experiments in realistic three-dimensional simulations of urban driving and on a 1/5 scale robotic truck that is trained to drive in a residential area. Both systems drive based on visual input yet remain responsive to high-level navigational commands. Experimental results demonstrate that the presented approach significantly outperforms a number of baselines.
ICRA’18
Details PDF Video Bibtex

In this paper, we train a deep neural network to predict UAV controls from raw image data for the task of autonomous UAV racing in a photo-realistic simulation. Training is done through imitation learning with data augmentation to allow for the correction of navigation mistakes. Extensive experiments demonstrate that our trained network (when sufficient data augmentation is used) outperforms state-of-the-art methods and flies more consistently than many human pilots.
ECCVW’18 - UAVision’18
Details PDF Project Bibtex

In this paper, we present a framework that allows the explicit incorporation of global context within CF trackers. We reformulate the original optimization problem and provide a closed form solution for single and multi-dimensional features in the primal and dual domain. We demonstrate with extensive experiments that this framework can significantly improve the performance of many CF trackers with only a modest impact on their frame rate.​
Oral at CVPR’17
Details PDF Video Code Project Supplement Bibtex

In order to evaluate object trackers for aerial tracking applications, we have generated 123 new video sequences from a UAV and annotated them with upright bounding boxes and attributes relevant for tracking. With more than 110,000 frames this is currently the largest data set for aerial tracking by a long shot and the second largest data set for generic tracking. In addition, we have developed a photo-realistic simulator within the UE4 framework that can be used for numerous vision tasks. As an example, we have integrated several state-of-the-art object trackers to control an UAV inside the simulator based on live feedback. In addition our simulator can be used to generate large amounts of realistic vision data.
Poster at ECCV’16
Details PDF Dataset Project Supplement Simulator Bibtex

We have developed a persistent, robust and autonomous object tracking system for unmanned aerial vehicles (UAVs). Our computer vision and control strategy integrates multiple UAVs with a stabilized RGB camera and can be applied to a diverse set of moving objects (e.g. humans, animals, cars, boats, etc.). A novel strategy is employed to successfully track objects over a long period, by ’handing over the camera’ from one UAV to another. The popular object tracker Struck was optimized for both speed and performance and integrated into the proposed system.
Interactive Presentation at IROS’16
Details PDF Video Project Bibtex

Contact