Skip navigation and jump directly to page content

 IU Trident Indiana University

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions

Project Lead: David Crandall, IU School of Informatics and Computing

Research made possible by: High Performance Systems (HPS), Scientific Applications and Performance Tuning (SciAPT), Big Red II supercomputer

Funded in part by the National Science Foundation

Figure 1. We present a CNN-based technique for detecting, identifying, and segmenting hands in egocentric video that includes multiple people interacting with each other. To illustrate one specific application, we show that hand segments alone can be used for accurate activity recognition. 

Hands appear very often in egocentric video, and their appearance and pose give important cues about what people are doing and to what they are paying attention. Existing work in hand detection has made strong assumptions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Convolutional Neural Networks, and introduce a simple candidate region generation approach that outperforms existing techniques at a fraction of the computational cost.

We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an application, we investigate the extent to which hand segmentation alone can distinguish between different activities. We evaluate these techniques on a new dataset of 48 first-person videos (along with pixel-level ground truth for over 15,000 hand instances) of people interacting in realistic environments.

Other collaborators include: Sven Bambach and Stefan Lee: School of Informatics and Computing Indiana University, Chen Yu: Psychological and Brain Sciences Indiana University.

For more information about this project:

The High Performance Systems (HPS) group implements, operates, and supports some of the fastest supercomputers in the world – IU’s Big Red II, the Quarry cluster, Karst, and the large memory Mason system – in order to advance Indiana University's mission in research, training, and engagement in the state.  HPS also supports databases and database engines used by the IU community.

The mission of the Scientific Applications and Performance Tuning (SciAPT) group is to deliver and support software tools that promote effective and efficient use of IU's advanced cyberinfrastructure which, in turn, improves research and enables discoveries.

NSF GSS Codes:

Primary Field: Computer Science (401) Artificial Intelligence