This project enabled networks of co-robots to collaborate among themselves and with humans to understand their surrounding environment at a semantic level. This allowed co-robots to work alongside human operators in a wide range of environments and applications. This project advanced the state of the art in robust semantic mapping from multiple robots as it introduced a new framework that can handle large, dynamic, and uncertain environments that explicitly allowed the study of interactions and information exchanges with humans; also, the framework allows an intelligent use and sharing of the limited computational resources possessed by the network of co-robots as a whole. These developments were driven by two case studies: a job-shop (small factory) scenario, where robots and fixed cameras track and assist human workers during production and assembly of parts; and a classic search-and-rescue scenario, where operators use a heterogeneous team of robots to quickly assess damages and discover survivors.  

In terms of intellectual merit, this project achieved three goals: 1) robust, scalable, semantic mapping of static environments, 2) semantic mapping of dynamic environments with multiple robots, and 3) computation control. By semantic mapping, we intend solutions that, using computer-vision algorithms, detect and classify meaningful entities, such as objects and geometric primitives (lines and planes) and incorporate them in a geometric map by following semantically meaningful constraints (e.g., objects should be on top of horizontal planes). The complexity of non-linear measurements from multiple robots and the combinatorial nature of the semantic constraints prevented previous solutions from being robust, flexible, and scaling to large problems. Our work started from where previous geometric SLAM algorithms ended, and provided novel, robust and scalable algorithms, while characterizing uncertainties and ambiguities in the produced maps. 

Besides supporting MS and PhD students, this project exposed undergraduate students to research and allowed them to contribute building, debugging, and testing a testbed that consists of three main parts: 1) CPU-based computational units, 2) GPU-based computational units, and 3) Robotic platforms. The testbed was used to verify the feasibility of our approach on a single CPU-based computational unit such as a Raspberry Pi and then helped us validate our approach to a team of Raspberry Pis. We then extended our solution to GPU-based computational units. We carefully profiled the resource utilization, both in terms of memory and processing power of the different computational units under different locations, input sizes, application deadline, and different conditions such as illumination and background clutter.

Specific achievements accomplished during this project are summarized below:

-- Implementation of end-to-end object detection algorithm on Raspberry Pi. This involved implementing traditional computer-vision algorithms and convolution neural network-based algorithms. Both exact and approximate versions of these algorithms were implemented. Furthermore, a Markov Decision Process (MDP)-based decision framework for adaptive selection of computer vision algorithms was implemented.

-- Establishing connection between Raspberry Pi and the quadcopter using the Robot Operating System (ROS) and running experiments in different locations and conditions to study the performance of the proposed solutions.

-- Implementation of a distributed computing framework to establish communication between multiple quadcopters. This activity involved building a resource-task mapper to decide the different tasks of the object-detection algorithm executed by quadcopters in the team based on the resources available.

-- Finally, we extended our analysis to GPUs, whose parallel structure makes them more effective than general-purpose CPUs for algorithms like deep learning where the processing of large blocks of visual data is done in parallel. We showed that it is possible to bring a many-fold increase in the overall processing speed by running an application in parallel (e.g., by exploiting the inherent parallel structure of the application or executing computer-vision algorithm on multiple images in parallel) along with approximation.

Last Modified: 01/29/2021
Modified by: Dario Pompili