The Complex Acoustic Surveillance and Tracking (COAST2) Project

• Funding agency: CENIIT
• Project leader: Isaac Skog
• Duration: 2019 - onward

Background

Historically, Sweden holds a strong position in the area of underwater technology, with world-class submarines and autonomous underwater systems. An important component needed to ensure that Sweden remains at the technological forefront within this area, and maintains its industrial leadership and defense capabilities, is the development of new technologies for underwater monitoring and surveillance. During the last decades — driven by the development in sensing, computational, and communication capabilities — a range of new signal processing theories and methods for target detection and tracking, such as random finite set filters, adaptive waveform design, and track-before-detect filters, have been developed. Along the same line of research, methods for sensor management and distributed sensor fusion in sensor networks subjected to energy, computational, and communication constraints, have been developed. These methods have successfully been applied in various terrestrial radar systems and sensor networks. However, due to the complex acoustic environment, the application of these theories and methods in sub-surface monitoring and surveillance systems remains a challenge. This is especially the case for systems that are to operate in littoral waters, such as the Swedish coast. The shallow depths, surface heating, and cooling, salinity changes, dense civil traffic, etc., result in an ever-changing acoustic environment; an environment to which the theories and methods must be adapted and tuned. Generally, the complex acoustic environment also limits the communication capabilities within underwater sensor networks to a point beyond that imposed on terrestrial sensor networks. This implies that the employed distributed target detection and tracking methods have to function with a minimum of information exchange, and new communication strategies that make dual use of the communication signals both for sensing and information exchange, are necessary.

In addition to the mentioned challenges, generally comes the problem of not knowing the exact location of the sensors within the surveillance system, which can severely degrade the performance of the system. The uncertainties in the location of the sensors are caused by the need to, in many situations, quickly and quietly deploy the system. This implies that it is not feasible to apply active calibration methods to infer the location of the sensors. Neither can, due to the poor propagation of radio waves in water, standard radio-based positioning tools commonly employed in terrestrial systems, such as the Global Navigation Satellite Systems (GNSS), be used to infer the location of the sensors. Further, other sensing platforms, such as hydrophone arrays, may be towed behind a vessel and constantly change location and geometry. Therefore, new methods for passive calibration, a.k.a self-calibration, of the sensor locations, must be developed.

Goal

The goal of the Complex Acoustic Surveillance and Tracking (COAST) project is to, in cooperation with the Dept. of Naval Systems & Underwater Technology, Swedish Defence Research Agency (FOI), and Saab Dynamics, research and develop methods to adapt current state-of-the-art target detection and tracking methods to the complex acoustic environment encountered by sub-surface systems. To obtain results of long-lasting impact and relevance to the Swedish industry and defense, experimentally driven research where theories and methods are developed, verified, and adapted based on sea trial data, will constitute a core part of the project activities. The focus will be on combining the theoretical excellence of the Dept. of Automatic Control, Linköping University (LiU), with the excellent knowledge about the underwater domain and unique experimental resources, of the Dept. of Naval Systems & Underwater Technology, FOI, and the knowledge about system integration and operational requirements at Saab Dynamics.

Project description and work packages

As a first step in the development and adaption of the current state-of-the-art target detection and tracking methods to the complex acoustic environment encountered by sub-surface systems, the following three work packages (WPs) have been framed.

WP1: Sensor Location Calibration

For a sensor network or sensor array to reliably detect and track signal sources, the location of the sensors within the system must be known with high accuracy. Although the location of the sensors is generally known with high accuracy when released at the surface, due to their drift in the water, the exact locations of the sensors, when they have sunk to the bottom, are unknown. Needless to say, for sensor systems towed behind vessels, the sensor locations are constantly changing and are therefore unknown. Thus, various active calibration methods for estimating the location of the sensors have been developed. However, in many situations, it is due to time and cost constraints, and the risk of revealing the existence of the surveillance system or disturbing marine wildlife, desirable to avoid active calibration methods. Therefore, methods that may be used to either track the motions of the sensors while they sink or to passively estimate their locations once they are on the bottom, need to be developed.

Aim: To study the feasibility of using: (i) inertial sensors to track the sensors while sinking or towed behind a vessel, and (ii) signals of opportunity, or in the case of towed system the self-noise of the vessel, for estimating the locations of the sensors.

WP2: Track-Before-Detect in Non-Stationary Disturbances

Track-before-detect is a powerful technique for simultaneous target detection and tracking within the same stochastic filtering framework. By coherently integrating the measured signals along possible target trajectories, and thereby increasing the signal-to-disturbances ratio, it enables targets to be detected and tracked at low a signal-to-disturbance ratio. Fundamental to the successful use of the track-before-detect technique is access to realistic target motion models and disturbance models. Realistic motion models are generally readily available for the considered targets. However, reliable disturbance models are generally not available. The disturbances include ambient noise caused by water turbulence, shipping, wave action, rainfall, etc., and reverberations due to bottom, surface, and in-volume reflections. The properties of these disturbances vary with the location, observation direction, time, etc. Therefore, even though well-established models for individual disturbance sources under different environmental conditions exist, it is difficult to create generic applicable models. Further, many of the models capture only the temporal behavior of the disturbances and not for track-before-detect applications important spatial behavior. Hence, new methods for online learning of disturbance models suitable for track-before-detect applications are needed.

Aim: To study and develop methods for online learning of disturbance models suitable for track-before-detect applications in underwater surveillance systems.

WP3: Sensor Management for Joint Sensing and Communication

To optimize the detection and tracking performance of underwater surveillance systems, various sensor management strategies for selecting the optimal sonar waveforms, sensor locations, and paths for reconnaissance vessels, have been proposed. In distributed underwater surveillance networks, one of the main bottlenecks when implementing a detection and tracking framework is the, in general, limited bandwidth available for sensing and communication; the limited bandwidth generally implies that suboptimal distributed information fusion strategies have to be used, with a loss in detection and tracking performance as a result. One possible way to better use the available bandwidth and improve the performance is to use the, within the radar community, recently proposed idea of joint sensing and communication. That is, the communication signals are designed so that they can simultaneously be used both for information exchange and active sensing. To maximize the benefits of the joint sensing and communication concept new distributed sensor management and information strategies must be developed; strategies that, based upon currently available information decides on which sources that are to communicate, when to communicate, and what waveforms to be used.

Aim: To research and develop sensor management methods for joint sensing and communication in underwater sensor networks.

Project Status and Results

Summary of Activities Conducted Within The Project 2022-2023

• The Ph.D. student Koul Ashwani was recruited for the COAST connected project Joint Sensing, Localization, and Communication for Next Generation Autonomous Underwater Systems
• The conference paper [2] was awarded the "Second best paper award" at the IEEE International Conference on Multisensor Fusion and Integration (MFI 2022).
• A 24-month research sponsoring contract with FOI has been signed.
• A 3-month research internship at Saab Dynamics was arranged for the CENIIT-funded Ph.D. student Daniel Bosser.
• The conference paper [1] was published at IEEE OCEANS. The research presented in this paper was conducted together with researchers at Saab Dynamics and FOI.
• A Vinnova "Avancerad och innovativ digitalisering 2022" project application about autonomous systems for surveillance of underwater infrastructure was submitted together with Saab Dynamics and Signals & Systems at Uppsala University. Unfortunately, the application was rejected. Discussion about a potential new application has been initiated.

Summary of Activities Conducted Within The Project 2021-2022

• PI's collaboration contract with FOI has been renewed.
• One conference paper [2] and one journal paper [3] were published. One more journal paper has been revised [6].
• Guest lectures on Array signal processing in TSRT78 Digital Signal Processing, Magnus Lundberg Nordenvad, FOI.

Further the project Joint Sensing, Localization, and Communication for Next Generation Autonomous Underwater Systems has been granted and the process of recruiting a Ph.D. student for the project is ongoing. The project is funded by Wallenberg AI, Autonomous Systems, and Software Program (WASP), and will be conducted in cooperation with Saab Dynamics and FOI.

Summary of Activities Conducted Within The Project 2020-2021

• Sea-trial where inertial and pressure data was recorded from sinking underwater sensor nodes.
• Sea-trial where data to investigate the possibility of using communication signals as sonar pulses were recorded.
• Guest lectures on Array signal processing in TSRT78 Digital Signal Processing, Magnus Lundberg Nordenvad, FOI.
• MSc thesis: Tracking of sinking underwater node using inertial navigation, Kristoffer Lindve, FOI/MH.
• Two conference papers (see [4] and [13]) and three journal papers (see [16,17,19]) were published.
• Development of software toolbox for array calibration during sea trials; based upon [5] and successfully used during several sea trials.

Further the project Tensor-field based localization has been started. The project is funded by the Swedish Research Council and will study how magnetic field variations and sensor arrays can be used for localization in GNSS-denied environments, such as underwater environments.

Summary of Activities Conducted Within The Project 2019-2020

• PhD student Daniel Bosser recruited to work in the project.
• Seminaries on Underwater acoustics in marine environments for the benefit of defense and security , by Lena Lund, FOI.
• Seminaries on Signal processing in marine bioacoustics , by Mathias Andersson, FOI.
• Guest lectures on Array signal processing in TSRT78 Digital Signal Processing, Magnus Lundberg Nordenvad, FOI.
• MSc thesis: Trajectory and pulse optimization for active towed array sonar using MPC and information measures, , Fabian Ekdahl Filipsson, FOI/UU.
• MSc thesis: A Study of Using Communication Signals as Sonar Pulses in Underwater Sensor Systems , Erica Svensson, FOI/UU.
• One conference paper (see [5]) was published and one journal paper (see [6]) was submitted.

Results WP1: Sensor Location Calibration

Signals of opportunit based calibration

A simultaneous localization and mapping (SLAM) method to calibrate the geometries of hydrophone arrays using the sound emitted from nearby ships has been developed. The performance of the proposed calibration method has been evaluated using data from two PASS-2447 Omnitech Electronics Inc. 56-element hydrophone arrays. Tests with four data sets have shown that array geometries in the northeast plane can be consistently estimated. Further, the calibration of the array geometries has been shown to increase the source localization accuracy significantly. The findings are presented in the conference paper:

• The Signals of Opportunity based Geometry Calibration of Hydrophone Arrays

A video illustrating the calibration method in action is shown below.

Inertial tracking of sinking sensor node

The possibility of using inertial and pressure sensors to track the location and orientation of a sinking sensor node has been investigated. Sensor data was collected during a sea trial and then processed using an extended Kalman filter-based depth-aided inertial navigation system. The results are presented in the MSc thesis Tracking of sinking underwater node using inertial navigation by K. Lindve.

Results WP2: Track-Before-Detect

Together with Robin Forsling and Gustaf Hendeby, a study on passive multi-sonar track-before-detect has been initiated. Experimental data from a test with an unmanned underwater vehicle moving in littoral water has been collected by the Underwater Department, FOI. The initial results show that the use of track-before-detect enables earlier detection at a lower false alarm rate compared to conventional methods. The current focus is on enhancing the performance further by learning the background disturbances. A short video of the algorithm in action is shown below.

A statistically motivated likelihood function for track-before-detect using passive sonar has also been developed. The paper [2] presents the method. The paper received the "second best paper" award at the MFI 2022 conference. A video illustrating the behavior of a passive track-before-detect system using the proposed likelihood is shown below.

Results WP3: Sensor Management for Joint Sensing and Communication

A pre-study about the feasibility of using existing communication signals for sensing was conducted by the MSc student Erica Svensson. The results presented in her master thesis report A Study of Using Communication Signals as Sonar Pulses in Underwater Sensor Systems indicate that a high-range resolution can be obtained.

Project Collaborations

FOI Swedish Defense Research Agency

The PI has a close collaboration with the Underwater Department at FOI Swedish Defense Research Agency, and the PI currently holds an adjunct position at FOI. The collaboration has resulted in:

• The joint publications [1,2,3,5,6,14].
• Three MSc thesis projects.
• A 24-month research sponsor contract.
• Multiple guest lectures at LiU by researchers from FOI.
• M. L. Nordenvaad, FOI, becoming a co-supervisor for PhD student D. Bossér.

Beyond the collaboration within the COAST project, I and FOI have received joint funding from Security Link for a project on localization in GNSS denied environments, such as underwater environments. See project description about Tensor-field based localization project.

Saab Dynamics

A collaboration with J. Olofsson and A. Gällström at Saab Dynamics has been initiated. So far the collaboration has resulted in:

• The Wallenberg AI, Autonomous Systems, and Software Program (WASP) project Joint Sensing, Localization, and Communication for Next Generation Autonomous Underwater Systems; the Ph.D. Ashwani Koul has been recruited to work on the project.
• A 3-month internship at Saab Dynamics for the COAST-project Ph.D. student Daniel Bosser.
• The joint publication [1] about background modeling for passive track-before-detect systems.

Publications

Project publications

The research within the COAST project has resulted in the following publications:

1. D. Bosser, R. Forsling, G. Hendeby, M. L. Nordenvaad, and I. Skog, Underwater Environment Modeling for Passive Sonar Track-Before-Detect , Proc. OCEANS 2019, Ireland, Jun. 2023.
2. D. Bosser, G. Hendeby, M.L. Nordenvaad, and I. Skog, A Statistically Motivated Likelihood for Track-Before-Detect , IEEE Int. Conf. on Multisensor Fusion and Integration (MFI), Sep. 2022
3. R.L. Nordström, E. Lalander, I. Skog, and M. Andersson, Maximum likelihood separation of anthropogenic and wind-generated underwater noise , The Journal of the Acoustical Society of America, Sep. 2021.
4. A. Kullberg, I. Skog, and G. Hendeby, Learning Motion Patterns in AIS Data and Detecting Anomalous Vessel Behavior, Proc. FUSION 2021, South Africa, Nov. 2021.
5. I. Skog and E. Gudmundson, Signals of Opportunity based Geometry Calibration of Hydrophone Arrays , Proc. OCEANS 2019, Seattle, WA, Oct. 2019.
6. I. Skog and M.L. Nordenvaad, Signals of Opportunity Based Hydrophone Array Shape and Orientation Estimation , In review: IEEE Journal of Oceanic Engineering, 2022.

Related publications

Research conducted in projects closely connected to the COAST project has resulted in the following publications:

7. I. Skog, G. Hendeby, and M. Kok, Tightly Integrated Motion Classification and State Estimation in Foot-Mounted Navigation Systems, Int. Conf on Indoor Positioning and Indoor Navigation, Germany. Sep. 2023.
8. A. Kullberg, I. Skog, and G. Hendeby. Iterated Filters for Nonlinear Transition Models, Int. Conf on Information Fusion, Charleston, SC. Jul. 2023.
9. A. Kullberg, I. Skog, and G. Hendeby, Unified Linearization-based Nonlinear Filtering , In review: IEEE Signal Processing Letters, 2023.
10. A. Kullberg, M. A. Skoglund, I. Skog, and G. Hendeby, On the Relationship Between Iterated Statistical Linearization and Quasi–Newton Methods , In review: IEEE Signal Processing Letters, 2023.
11. M. Malmström, A. Kullberg, I. Skog, D. Axehill, and F. Gustafsson, "Extended target tracking utilizing machine-learning software – with application to animal classification,” In review: IEEE Signal Processing Letters, 2023.
12. M. Malmström, I. Skog, D. Axehill, and F. Gustafsson, Uncertainty quantification in neural network classifiers – a local linear approach ,” In review: Automatica, 2023.
13. H. Carlsson, I. Skog, G. Hendeby, and J. Jalden Inertial Navigation Using an Inertial Sensor Array, Submitted to IEEE Trans. Signal Processing, 2022.
14. C Huang, G. Hendeby, and I. Skog A Tightly-Integrated Magnetic-Field aided Inertial Navigation System , IEEE Int. Conf. Information Fusion (FUSION), Jul. 2022.
15. I. Skog, G. Hendeby, and F. Trulsson, Magnetic-field Based Odometry – An Optical Flow Inspired Approach , IEEE Int. Conf. Indoor Positioning and Indoor Navigation (IPIN), Nov. 2021.
16. H. Carlsson, I. Skog, T.B. Schön, and J. Jaldén, Quantifying the Uncertainty of the Relative Geometry in Inertial Sensors Arrays, IEEE Sensors Journal, vol. 21, no. 17, 2021
17. H. Carlsson, I. Skog, and J. Jaldén, Self-Calibration of Inertial Sensor Arrays, IEEE Sensors Journal, vol. 21, no. 6, 2021
18. A. Kullberg, I. Skog, and G. Hendeby, Online Joint State Inference and Learning of Partially Unknown State-Space Models, IEEE Trans. Signal Processing, DOI: 10.1109/TSP.2021.3095709, 2021.
19. A. Kullberg, I. Skog, and G. Hendeby, Learning driver behaviors using a Gaussian process augmented state-space models, Proc. Int. Conf on Information Fusion, Online, Jul. 2020.
20. J. Wahlström and I. Skog, Fifteen Years of Progress at Zero Velocity: A Review, IEEE Sensors Journal, DOI: 10.1109/JSEN.2020.3018880, Aug. 2020.