Marcus Wallenberg

In this paper, we study object recognition in the embodied setting. More specifically, we study the problem of whether the recognition system will benefit from acquiring another observation of the object under study, or whether it is time to give up, and report the observed object as unknown. We describe the hardware and software of a system that implements recognition and object permanence as two nested perception-action cycles. We have collected three data sets of observation sequences that allow us to perform controlled evaluation of the system behavior. Our recognition system uses a KNN classifier with bag-of-features prototypes. For this classifier, we have designed and compared three different uncertainty measures for target observation. These measures allow the system to (a) decide whether to continue to observe an object or to move on, and to (b) decide whether the observed object is previously seen or novel. The system is able to successfully reject all novel objects as “unknown”, while still recognizing most of the previously seen objects.

Segmentation is an important preprocessing step in many applications. Purely colour-based segmentation is often problematic. For this reason, we here investigate fusion of depth and colour information, to facilitate robust segmentation of single images. We evaluate segmentation results on data collected using the Microsoft Kinect peripheral for Xbox 360, using superparamagnetic clustering. We also propose a method for aligning and encoding colour and depth information from the Kinect device. As we show in the paper, the fusion of depth and colour information produces more semantically relevant segments for scene analysis than either depth or colour separately.

We present in this paper a research platform for development and evaluation of embodied visual object recognition strategies. The platform uses a stereoscopic peripheral-foveal camera system and a fast pan-tilt unit to perform saliency-based visual search. This is combined with a classification framework based on the bag-of-features paradigm with the aim of targeting, classifying and recognising objects. Interaction with the system is done via typed commands and speech synthesis. We also report the current classification performance of the system.