Award Abstract # 1715195
RI: Small: Collaborative Research: Seeing Surfaces: Actionable Surface Properties from Vision

NSF Org: IIS
Div Of Information & Intelligent Systems
Recipient: RUTGERS, THE STATE UNIVERSITY
Initial Amendment Date: August 9, 2017
Latest Amendment Date: April 29, 2022
Award Number: 1715195
Award Instrument: Standard Grant
Program Manager: Jie Yang
jyang@nsf.gov
 (703)292-4768
IIS
 Div Of Information & Intelligent Systems
CSE
 Direct For Computer & Info Scie & Enginr
Start Date: September 1, 2017
End Date: August 31, 2023 (Estimated)
Total Intended Award Amount: $249,931.00
Total Awarded Amount to Date: $249,931.00
Funds Obligated to Date: FY 2017 = $249,931.00
History of Investigator:
  • Kristin Dana (Principal Investigator)
    kristin.dana@rutgers.edu
Recipient Sponsored Research Office: Rutgers University New Brunswick
3 RUTGERS PLZ
NEW BRUNSWICK
NJ  US  08901-8559
(848)932-0150
Sponsor Congressional District: 12
Primary Place of Performance: Rutgers University New Brunswick
94 Brett Road
Piscataway
NJ  US  08854-8058
Primary Place of Performance
Congressional District:
06
Unique Entity Identifier (UEI): M1LVPE5GLSD9
Parent UEI:
NSF Program(s): Robust Intelligence,
NRI-National Robotics Initiati
Primary Program Source: 01001718DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 7495, 7923
Program Element Code(s): 7495, 8013
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

This project is to enable computers and robots the capability of estimating actionable, physical properties of surfaces (the feel) from their appearance (the looks). The key idea is to leverage the deeply interwoven relation between radiometric and physical surface characteristics. By learning models that make explicit the physical surface properties encoded in full and partial measurements of radiometric appearance properties, computers can estimate crucial physical properties of real-world surfaces from passive observations with novel camera systems. This project paves the path for integrating these models and estimation algorithms into scene understanding, robotic action planning, and efficient visual sensing. The research results provide a currently missing but fundamental capability to computer vision that benefits a number of applications in areas of computer vision, robotics, and computer graphics. The project provides hands-on research opportunities for both undergraduate and graduate students and are integrated in the PIs' undergraduate and graduate courses taught at Drexel and Rutgers. They are also used as a backdrop for K-12 outreach activities including high school and middle school mentorship programs. The data collection activities provide an ideal platform to expose K-12 students to physics and computer science.

This research investigates the methods to infer actionable surface properties from images and detailed surface reflectance measurements. The research activities are centered on four specific aims: 1) controlled and uncontrolled large-scale data collection of actionable physical properties and appearance measurements of everyday surfaces, 2) derivation of prediction models for deducing physical properties from local surface appearance, 3) integration of global semantic context including object and scene information, and 4) development of efficient appearance capture and its use for novel physics-from-appearance sensing. These research thrusts collectively answer the fundamental question of how computer vision can anticipate the physical properties of a surface without touching it and knowing what it is, laying the foundation for computational vision-for-action.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

Ichikawa, T. "Shape From Sky: Polarimetric Normal Recovery Under the Sky" IEEE Conference on Computer Vision and Pattern Recognition , 2021 Citation Details
Purri, Matthew and "Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces from Images" European Conference on Computer Vision , 2020 https://doi.org/10.1007/978-3-030-58583-9_1 Citation Details

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

Project Outcomes:

 

Three main project outcomes for this grant are  methods for: ?Teaching Cameras to Feel?, ?Shape from Sky? and ?Ground Terrain Recognition?. 

 

Teaching Cameras to Feel:

The connection between visual input and tactile sensing is critical for object manipulation tasks such as grasping and pushing. In this work, we introduce the challenging task of estimating a set of tactile physical properties from visual information. We aim to build a model that learns the complex mapping between visual information and tactile physical properties. We construct a first of its kind image-tactile dataset with over 400 multiview image sequences and the corresponding tactile properties. A total of fifteen tactile physical properties across categories including friction, compliance, adhesion, texture, and thermal conductance are measured and then estimated by our models. We develop a cross-modal framework comprised of an adversarial objective and a novel visuo-tactile joint classification loss. Additionally, we introduce a neural architecture search framework capable of selecting optimal combinations of viewing angles for estimating a given physical property.

 

Shape from Sky

The sky exhibits a unique spatial polarization pattern by scattering the unpolarized sun light. Just like insects use this unique angular pattern to navigate, we use it to map pixels to directions on the sky. That is, we show that the unique polarization pattern encoded in the polarimetric appearance of an object captured under the sky can be decoded to reveal the surface normal at each pixel. We derive a polarimetric reflection model of a diffuse plus mirror surface lit by the sun and a clear sky. This model is used to recover the per-pixel surface normal of an object from a single polarimetric image or from multiple polarimetric images captured under the sky at different times of the day. We experimentally evaluate the accuracy of our shape-from-sky method on a number of real objects of different surface compositions. The results clearly show that this passive approach to fine-geometry recovery that fully leverages the unique illumination made by nature is a viable option for 3D sensing. With the advent of quad-Bayer polarization chips, we believe the implications of our method span a wide range of domains.

 

Ground Terrain Material Recognition

Computational surface modeling that underlies material recognition has transitioned from reflectance modeling using in-lab controlled radiometric measurements to image-based representations based on internet-mined single-view images captured in the scene. We take a middle-ground approach for material recognition that takes advantage of both rich radiometric cues and flexible image capture. A key concept is differential angular imaging, where small angular variations in image capture enables angular-gradient features for an enhanced appearance representation that improves recognition. We build a large-scale material database, Ground Terrain in Outdoor Scenes (GTOS) database, to support ground terrain recognition for applications such as autonomous driving and robot navigation. The database consists of over 30,000 images covering 40 classes of outdoor ground terrain under varying weather and lighting conditions. We develop a novel approach for material recognition called texture-encoded angular network (TEAN) that combines deep encoding pooling of RGB information and differential angular images for angular-gradient features to fully leverage this large dataset. With this novel network architecture, we extract characteristics of materials encoded in the angular and spatial gradients of their appearance. Our results show that TEAN achieves recognition performance that surpasses single view performance and standard (non-differential/large-angle sampling) multiview performance.

We have expanded our work in this area to two application domains: 1) Agriculture and 2) Remote Sensing in this final year of the project. Our students had expertise in material recognition pipelines and we applied these methods to these two domains. In agriculture, we have developed a method for comparing the ripening characteristics of different crops via photometrically calibrated images. In Remote Sensing we have developed a way to use simulated change to train change detection networks. 


Last Modified: 11/02/2023
Modified by: Kristin J Dana

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page