Project leader: Gustaf Hendeby
Applied Research Platform for
Sensor and information fusion is a mature area with its own society (ISIF) and conference (FUSION). LiU has a strong position in developing both theory and applications. The aim of this proposal is to create a research group that complements the existing sensor fusion competence with knowledge in computer science and hardware. The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students.
Background and Industrial Relevance
Sensor and information fusion is a mature research area with its own society (International Society of Information Fusion, ISIF) and an annual conference (International Conference on Information Fusion, FUSION). It has also been a trademark of LiU and an explicit research area since 1995 in large research projects such as ISIS, MOVII, LINC-SIC, CADICS, and Security Link, and more than 200 MSc projects have been preformed in the area. Since 2010 there is even an undergraduate course called Sensor Fusion with approximately 40 students attending each year.
In the 1960's, an era when sensor fusion first gained momentum. An important driving force for the development was the Apollo program and aerospace applications. This led to the development of algorithms such as the Kalman filter (KF) and extended Kalman filter (EKF), which are still used as important corner stones in the field. The development made a next jump forwards in the 1990's, with a number of methods for better handling of nonlinear problem formulations. The enabler was to a large extent the availability of more computing power. The unscented Kalman filter (UKF), a development of the Kalman filter methodology, and the simulation based particle filter (PF), are examples of methods developed during this time. At the same time the driving applications started to shift as more and more sensors made their way into automotive industry, which in many ways can be considered the driving force in the early 2000's. As a result, almost all new cars today are equipped with an anti-lock braking system (ABS), electronic stability programme (ESP), and increasingly advanced active safety systems.
LiU, and the Automatic Control division in particular, has a strong position in the field of sensor and information fusion. One success factor has been the ability to combine high quality theoretical work with relevant and successful applications, \eg, in the aerospace and automotive settings. The result has been a number of well cited journal articles and successful spinnoffs, such as NIRA Dynamics (market leading in indirect tire pressure monitoring) and Senion (indoor localization solutions).
In the last decade or so advances in sensor and computers have resulted in an abundance of inexpensive sensors and computational resources entering the consumer market and everyday life of common people. One such example is the smartphone, which is a device full of senors used by almost everyone. This development has led to a surge in new areas of applications, such as:
- health and activity monitoring, where bracelets and smartphone apps are used to estimate the users activity and other vital signs;
- the basis of Internet of Things (IoT) and Industry 4.0;
- navigation in GPS denied environments, where other sensors are used to compensate for the lack of GPS coverage; and
- the rapid development of small affordable quadcopters and drones, used both as toys and commercially, relying on the advances in hardware to work.
Algorithms targeting this type of applications are required to be efficient, as computing power and batteries are limited, and robust, to compensate for imperfections associated with the generally low-cost sensors being used. Hence, many of the solutions appearing in the field right now are rather ad hoc, designed to address a specific problem using a specific sensor.
The aim of this proposal is to initialize the creation of a research group that performs basic research and in parallel demonstrates the results using smartphone sensors. This is important in order not to miss out on the potential of new technology, and as a reaction to the increasing importance of being able to demonstrate results in terms of demonstrators to attract funding. Currently, the research group is almost exclusively restricted to working in MATLAB, which is unsuitable for real-time applications such as demonstrators. For the group's previous success stories, close collaboration with spin-off companies to handle low-level details has often been necessary. Not having more of this competence in-house has been identified as a weakness. The expertise in the suggested research group will address this.
The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students. The Sensor Fusion app (SFapp), described in more detail below, is a key tool to achieve this. The following areas have been identified as of high interest:
- Medical applications: On-body sensors can be
used to estimate anything ranging from body motion and gait to vital
signs of the wearer. This is seen as an important component in
future home care solutions. The project has previously experience
of both activity classification using machine learning and motion
analysis aided by biomechanical models.
Current related activities include:
- A PhD student cooperating with the University Hospital in Linköping using acoustic analysis to provide early indication of heart pump failure. The prospect of this type of approach is considered promising.
- A method for magnetic tracking that has gained interest from Sectra as a way to facilitate digital pathology.
- The company StepIt developing a product to avoid blood clogs during long flights see the proposed project as a way to provide added value. The laboratory exercise developed around the SFapp involves similar techniques and can be seen as a starting point.
- Airborn systems: The recent development of drones has opened up a whole new need for the use of lightweight and low-power sensors and computing power in a difficult environment. A number of MSc theses have been performed where smartphones have been used as an airborn sensor platform in quadcopters. Proper utilization of available hardware can expand the applicability of this type of equipment in new directions.
- Condition monitoring in robotics and transportation
systems: Sounds and vibrations in machinery and transportation
systems can tell a lot about the integrity of it. Sensors measuring
vibrations and sound can therefore be used for condition monitoring.
Current activities include:
- Since June 2016, a smartphone with the SFapp has been attached to a train in the new Wildfire roller coaster in Kolmården Wildlife Park. It collects information about vibrations and movements and the idea is to detect problems before they become safety risks.
- In a PhD project, vibration data from cars has been collected and shown to provide speed information. This work has attracted interest by NIRA Dynamics as part of a GPS-free navigation solution.
- Robust localization: An area with several
on-going research projects where a network of smartphones can be
used to replace costly hardware and to provide a more flexible
demonstrator environment, e.g.:
- Network of smartphones using the magnetometers to localize magnetic objects.
- Network of microphones to localize sounding objects such as motorized vehicles and animals.
- Network of smartphones to measure radio signal strength from cellular networks, WiFi, and/or Bluetooth to localize transmitters, such as victims of avalanches and earthquakes as has been illustrated with the SFapp in an MSc thesis.
The Sensor Fusion App
An important tool needed to achive the goals of the project described above is the Sensor Fusion App (SFapp). The SFapp serves dual purposes as both a teaching and research platform.
The app was originally developed by the applicant for a lab in the Sensor Fusion course [1, 2], giving students easy access to actual measurements in real-time, using MATLAB or any other high level language. In its current state, the SFapp can handle inertial measurements, as well as some other sensors found in most all smartphones. The app was introduced in 2013 and is now used as a lab-platform both nationally and internationally, and as indicated above, it is also used in MSc theses and research projects.
Through its spread, the app has provided the department with good publicity. The SFapp (Available free of charge from Google Play Store) has been installed by almost 6500 unique users, has more than 2000 current installs, and is rated 4.2/5. This shows the adoption of the app, and its potential to promote research results. New results can with the click of a button be distributed to thousands of users.
From above it is clear that there are already many on-going projects at LiU where the app has been used or can be used. However, the potential of more applications and demonstrators is huge. What is needed for this is to rethink the platform; refactoring the Android code, integrate new sensors, data compression, integration with cloud solutions, databases and other framework such as ROS, and not the least a toolkit of the standard methods in sensor fusion that fit into the SFapp framework. We intend to improve the support for radio networks (WiFi, cellular, Bluetooth), add ability to collect and stream audio and video, as well as more hardware specific equipment such as thermal imaging (available with the CAT S60 smartphone) and 3D-imaging (available from Google project Tango units) as needed.
Visions and Plans
The vision is to establish a research group that performs basic research and in parallel demonstrate the results using smartphone sensors. This way ensuring that LiU keeps its strong position in sensor and information fusion also in emerging applications with low-cost hardware becoming increasingly common in everyday situations. A key component is working with real problems and data, which inevitably reveals bottlenecks of existing theory and sparkles new research directions.
As a concrete start for a PhD student, we plan to start analyzing the data collected from the roller coaster in Kolmården. This requires synchronized frequency analysis along the track, which needs high position accuracy in a challenging environment at high speeds posing a nontrivial problem. We also consider to add at least acoustics and possibly also video to the collected data. This will require both new theory to be developed and improvements of the SFapp.
The short-term plan is to then extend the work and address more of the problems indicated as interesting based on experiences made and in the process improve the SFapp as we go along and identify the most urgent needs. The latter we hope to do in cooperation with other projects with similar needs, e.g., the project TRAX which already use the SFapp for some of its data collection.
Dissemination will include papers in the FUSION conference for basic research, application orientated conferences for demonstrators, journal articles for the basic theory with applications, and not the least apps that are spread using Google Play.
Research Environment and Industrial Cooperation
The proposed research project based on relevant industrial problem formulations as indicated above. Additional to the indicated collaboration with StepIt and NIRA Dynamics and connections to Kolmården Wildlife Park and the University Hospital which will provide data and problem formulations. Companies such as Ericsson and Saab should also be interested, as shown, e.g., by performed MSc theses. The proposal is also well in line with ongoing research initiatives, such as WASP, where demonstrating research results is a key component and where mobile solutions are promoted. The European training network TRAX is another benefactor, as previously stated.
Altogether, the proposal project addresses an identified weakness in the current research group by introducing a new research platform for sensor fusion utilizing the applicants previous experiences for both theoretical development and industry relevant practical results.
The project has received a planning grant for 2017.
|||G. Hendeby, F. Gustafsson, and N. Wahlström. Teaching sensor fusion and Kalman filtering using a smartphone. In Proceedings of 19th IFAC World Congress, Cape Town, South Africa, Aug. 2014. (Predating the project, but relevant for it.)|
|||G. Hendeby, F. Gustafsson, N. Wahlström, and S. Gunnarsson. Platform for teaching sensor fusion using a smartphone. International Journal of Engineering Education, N(N), 2017. Special issue on: "Engineering behind technology-based educational innovations". Accepted for publication. (Based on work prior to the project, but releant to it.)|
Associate Professor and Docent in Automatic Control
(Swedish: Universitetslektor och docent i reglerteknik)
- +46 13 28 58 15
Informationsansvarig: Gustaf Hendeby
Senast uppdaterad: 2016-12-12