Publications by Per-Erik Forssén

This paper presents a method that combines shadow detection and a 3D box model including shadow simulation, for estimation of size and position of vehicles. We define a similarity measure between a simulated image of a 3D box, including the box shadow, and a captured image that is classified into background/foreground/shadow. The similarity Measure is used in all optimization procedure to find the optimal box state. It is shown in a number of experiments and examples how the combination shadow detection/simulation improves the estimation compared to just using detection or simulation, especially when the shadow detection or the simulation is inaccurate. We also describe a tracking system that utilizes the estimated 3D boxes, including highlight detection, spatial window instead of a time based window for predicting heading, and refined box size estimates by weighting accumulated estimates depending oil view. Finally, we show example results.

A runtime system for implementation of image processing operations is presented. It is designed for working in a flexible and distributed environment related to the software architecture of a newly developed UAV system. The software architecture can be characterized at a coarse scale as a layered system, with a deliberative layer at the top, a reactive layer in the middle, and a processing layer at the bottom. At a finer scale each of the three levels is decomposed into sets of modules which communicate using CORBA, allowing system development and deployment on the UAV to be made in a highly flexible way. Image processing takes place in a dedicated module located in the process layer, and is the main focus of the paper. This module has been designed as a runtime system for data flow graphs, allowing various processing operations to be created online and on demand by the higher levels of the system. The runtime system is implemented in Java, which allows development and deployment to be made on a wide range of hardware/software configurations. Optimizations for particular hardware platforms have been made using Java's native interface.

This paper presents a method for detection of homogeneous regions in grey-scale images, representing them as blobs. In order to be fast, and not to favour one scale over others, the method uses a scale pyramid. In contrast to most multi-scale methods this one is non-linear, since it employs robust estimation rather than averaging to move through scale-space. This has the advantage that adjacent and partially overlapping clusters only affect each other's shape, not each other's values. It even allows blobs within blobs, to provide a pyramid blob structure of the image.

This article describes an essential step towards what is called a view centered representation of the low-level structure in an image. Instead of representing low-level structure (lines and edges) in one compact feature map, we will separate structural information into several feature maps, each signifying features at a characteristic phase, in a specific scale. By characteristic phase we mean the phases 0, pi, and +/-pi/2, corresponding to bright, and dark lines, and edges between different intensity levels, or colours. A lateral inhibition mechanism selects the strongest feature within each local region of scale represented. The scale representation is limited to maps one octave apart, but can be interpolated to provide a continous representation. The resultant image representation is sparse, and thus well suited for further processing, such as pattern detection.

This Chapter presents a vision-based method for unmanned aerial vehicle (UAV) motion estimation that uses as input an image motion field obtained from matches of point-like features. The Chapter enhances visionbased techniques developed for single UAV localization and demonstrates how they can be modified to deal with the problem of multi-UAV relative position estimation. The proposed approach is built upon the assumption that if different UAVs identify, using their cameras, common objects in a scene, the relative pose displacement between the UAVs can be computed from these correspondences under certain assumptions. However, although point-like features are suitable for local UAV motion estimation, finding matches between images collected using different cameras is a difficult task that may be overcome using blob features. Results justify the proposed approach.

Recent years have seen advances in the estimation of full 6 degree-of-freedom object pose from a single 2D image. These advances have often been presented as a result of, or together with, a new local image descriptor. This paper examines how the performance for such a system varies with choice of local descriptor. This is done by comparing the performance of a full 6 degree-of-freedom pose estimation system for fourteen types of local descriptors. The evaluation is done on a database with photos of complex objects with simple and complex backgrounds and varying lighting conditions. From the experiments we can conclude that duplet features, that use pairs of interest points, improve pose estimation accuracy, and that affine covariant features do not work well in current pose estimation frameworks. The data sets and their ground truth is available on the web to allow future comparison with novel algorithms.

Fourier descriptors (FDs) is a classical but still popular method for contour matching. The key idea is to apply the Fourier transform to a periodic representation of the contour, which results in a shape descriptor in the frequency domain. Fourier descriptors have mostly been used to compare object silhouettes and object contours; we instead use this well established machinery to describe local regions to be used in an object recognition framework. We extract local regions using the Maximally Stable Extremal Regions (MSER) detector and represent the external contour by FDs. Many approaches to matching FDs are based on the magnitude of each FD component, thus ignoring the information contained in the phase. Keeping the phase information requires us to take into account the global rotation of the contour and shifting of the contour samples. We show that the sum-of-squared differences of FDs can be computed without explicitly de-rotating the contours. We compare our correlation based matching against affine-invariant Fourier descriptors (AFDs) and demonstrate that our correlation based approach outperforms AFDs on real world data.

 The motionfield from image sequences of a dynamic 3D scene is in generalpiecewise continuous. Since two neighbouring regions may havecompletely different motions, motion estimation at thediscontinuities is problematic. In particular spatial averaging ofmotion vectors is inappropriate at such positions. We avoid thisproblem by channel encoding brightness change constraint equations(BCCE) for each spatial position into a channel matrix. By spatialaveraging of this channel representation and subsequently decodingwe are able to estimate all significantly different motionsoccurring at the discontinuity, as well as their covariances. Thispaper extends and improves this multiple motion estimation schemeby locally selecting the appropriate scale for the spatialaveraging.  

In this paper we make use of the idea that a robot can autonomously discover objects and learn their appearances by poking and prodding at interesting parts of a scene. In order to make the resultant object recognition ability more robust, and discriminative, we replace earlier used colour histogram features with an invariant texture-patch method. The texture patches are extracted in a similarity invariant frame which is constructed from short colour contour segments. We demonstrate the robustness of our invariant frames with a repeatability test under general homography transformations of a planar scene. Through the repeatability test, we find that defining the frame using using ellipse segments instead of lines where this is appropriate improves repeatability. We also apply the developed features to autonomous learning of object appearances, and show how the learned objects can be recognised under out-of-plane rotation and scale changes.

 Epipolar geometry is the geometry situation of two cameras depicting the same scene. For un-calibrated cameras epipolar geometry is compactly described by the fundamental matrix. Estimation of the fundamental matrix is trivial if we have a set of corresponding points in the two images. Corresponding points are often found using e.g. the Harris interest point detector, but there are several advantages with using richer features instead. In this paper we will use blob features. Blobs are homogeneous regions which are compactly described by their colour, area, centroid and inertia matrix. Using blobs to establish correspondences is fast, and the extra information besides position, allows us to reject false matches more accurately.

This paper presents and overview of the basic and applied research carried out by the Computer Vision Laboratory, Linköping University, in the WITAS UAV Project. This work includes customizing and redesigning vision methods to fit the particular needs and restrictions imposed by the UAV platform, e.g., for low-level vision, motion estimation, navigation, and tracking. It also includes a new learning structure for association of perception-action activations, and a runtime system for implementation and execution of vision algorithms. The paper contains also a brief introduction to the WITAS UAV Project.

This thesis introduces and explores a new type of representation for low and medium level vision operations called channel representation. The channel representation is a more general way to represent information than e.g. as numerical values, since it allows incorporation of uncertainty, and simultaneous representation of several hypotheses. More importantly it also allows the representation of “no information” when no statement can be given. A channel representation of a scalar value is a vector of channel values, which are generated by passing the original scalar value through a set of kernel functions. The resultant representation is sparse and monopolar. The word sparse signifies that information is not necessarily present in all channels. On the contrary, most channel values will be zero. The word monopolar signifies that all channel values have the same sign, e.g. they are either positive or zero. A zero channel value denotes “no information”, and for non-zero values, the magnitude signifies the relevance.

In the thesis, a framework for channel encoding and local decoding of scalar values is presented. Averaging in the channel representation is identified as a regularised sampling of a probability density function. A subsequent decoding is thus a mode estimation technique.'

The mode estimation property of channel averaging is exploited in the channel smoothing technique for image noise removal. We introduce an improvement to channel smoothing, called alpha synthesis, which deals with the problem of jagged edges present in the original method. Channel smoothing with alpha synthesis is compared to mean-shift filtering, bilateral filtering, median filtering, and normalized averaging with favourable results.

A fast and robust blob-feature extraction method for vector fields is developed. The method is also extended to cluster constant slopes instead of constant regions. The method is intended for view-based object recognition and wide baseline matching. It is demonstrated on a wide baseline matching problem.

A sparse scale-space representation of lines and edges is implemented and described. The representation keeps line and edge statements separate, and ensures that they are localised by inhibition from coarser scales. The result is however still locally continuous, in contrast to non-max-suppression approaches, which introduce a binary threshold.

The channel representation is well suited to learning, which is demonstrated by applying it in an associative network. An analysis of representational properties of associative networks using the channel representation is made.

Finally, a reactive system design using the channel representation is proposed. The system is similar in idea to recursive Bayesian techniques using particle filters, but the present formulation allows learning using the associative networks.

This report introduces a robust contour descriptor for view-based object recognition. In recent years great progress has been made in the field of view based object recognition mainly due to the introduction of texture based features such as SIFT and MSER. Although these are remarkably successful for textured objects, they have problems with man-made objects with little or no texture. For such objects, either explicit geometrical models, or contour and shading based features are also needed. This report introduces a robust contour descriptor which we hope can be combined with texture based features to obtain object recognition systems that work in a wider range of situations. Each detected contour is described as a sequence of line and ellipse segments, both which have well defined geometrical transformations to other views. The feature detector is also quite fast, this is mainly due to the idea of first detecting chains of contour points, these chains are then split into line segments, which are later either kept, grouped into ellipses or discarded. We demonstrate the robustness of the feature detector with a repeatability test under general homography transformations of a planar scene. Through the repeatability test, we find that using ellipse segments instead of lines, where this is appropriate improves repeatability. We also apply the features in a robotic setting where object appearances are learned by manipulating the objects.

This report describes how blob features can be used for automatic estimation of the fundamental matrix from two perspective projections of a 3D scene. Blobs are perceptually salient, homogeneous, compact image regions. They are represented by their average colour, area, centre of gravity and inertia matrix. Coarse blob correspondences are found by voting using colour and local similarity transform matching on blob pairs. We then do RANSAC sampling of the coarse correspondences, and weight each estimate according to how well the approximating conics and colours of two blobs correspond. The initial voting significantly reduces the number of RANSAC samples required, and the extra information besides position, allows us to reject false matches more accurately than in RANSAC using point features.

In this paper we present a new and efficient method to implement robust smoothing of low-level signal features: B-spline channel smoothing. This method consists of three steps: encoding of the signal features into channels, averaging of the channels, and decoding of the channels. We show that linear smoothing of channels is equivalent to robust smoothing of the signal features, where we make use of quadratic B-splines to generate the channels. The linear decoding from B-spline channels allows to derive a robust error norm which is very similar to Tukey's biweight error norm. Channel smoothing is superior to iterative robust smoothing implementations like non-linear diffusion, bilateral filtering, and mean-shift approaches for four reasons: it has lower computational complexity, it is easy to implement, it chooses the global minimum error instead of the nearest local minimum, and it can also be used on non-linear spaces, such as orientation space. In the experimental part of the paper we compare channel smoothing and the previously mentioned three other approaches for 2D orientation data.

In this report we describe how an RGB component colour image may be expanded into a set of channel images, and how the original colour image may be reconstructed from these. We also demonstrate the effect of averaging on the channel images and how it differs from conventional averaging. Finally we demonstrate how boundaries can be detected as a change in the confidence of colour state.

This report describes a novel window matching technique. We perform window matching by transforming image data into sparse features, and apply a computationally efficient matching technique in the sparse feature space. The gain in execution time for the matching is roughly 10 times compared to full window matching techniques such as SSD, but the total execution time for the matching also involves an edge filtering step. Since the edge responses may be used for matching of several regions, the proposed matching technique is increasingly advantageous when the number of regions to keep track of increases, and when the size of the search window increases. The technique is used in a real-time ego-motion estimation system in the WITAS project. Ego-motion is estimated by tracking of a set of structure points, i.e. regions that do not have the aperture problem. Comparisons with SSD, with regard to speed and accuracy are made.

This report describes an experimental still image coder that grew out of a project in the graduate course ``Advanced Video Coding'' in spring 2000. The project has investigated the idea to use local orientation histograms in fractal coding. Instead of performing a correlation-like grey-level matching of image regions, the block search is made by matching feature histograms of the block contents. The feature investigated in this report is local orientation, but in principle other features could be used as well. In its current state the coder does not outperform state of the art still image coders, but the block-search strategy seems promising, and will probably prove useful in several other applications.

In this report, the principles of man-made object detection in satellite images is investigated. An overview of terminology and of how the detection problem is usually solved today is given. A three level system to solve the detection problem is proposed. The main branches of this system handle road, and city detection respectively. To achieve data source flexibility, the Logical Sensor notion is used to model the low level system components. Three Logical Sensors have been implemented and tested on Landsat TM and SPOT XS scenes. These are: BDT (Background Discriminant Transformation) to construct a man-made object property field; Local-orientation for texture estimation and road tracking; Texture estimation using local variance and variance of local orientation. A gradient magnitude measure for road seed generation has also been tested.