Beacon - site visit 2
Behaviour modelling in context-aware systems, a project of the Intelligent Systems Group (Infotech Oulu) at the University of Oulu.
Site visit in Oulu on 17 May 2004
The project develops methods for learning behaviour models of both users and devices, which will enable the devices to adapt to their environment. The project's hypotheses are that behavioural modelling can be constructed with the help of observed user behaviour and data from devices acting in the environment and that these models can be used efficiently in proactive systems. The project will validate these hypotheses by developing methods for both model learning and signal analysis transforming data into parameters required by the models. The project will also build prototypes based on these methods.
For testing and building applications, the project has invested in a Smart Living Room of 140 square meters. The room is provided with a pressure-sensitive floor based on EMFi sensor stripes that have been mounted under the carpet in the room. The EMFi sensor (electromechanical film) is a plastic material that converts mechanical energy into an electrical signal and vice versa. Changes in the pressure on the film generate a charge on the film's surfaces that can be measured as a current or a voltage signal. The pressure-sensitive sensor is constructed of a grid of strips of sensitive plastic material (EMFi). Each strip out of 64 produces a continuous signal that must be analysed in order to detect and recognize the events.
A person walking in the room will produce force to the floor and thereby a voltage that can be measured. A walker's footstep hitting a stripe sensor can be recognised when looking at the measured signal. The different phases of a step, such as the heal or toe hitting the floor, are clearly recognisable. By combining data from vertical and horizontal stripes, also the location of the walker can be computed.
Each walker has a different style of walking and the shape of the footstep patterns may vary. Other reasons for a varying pattern may be gait style or footwear, or when footsteps are divided between adjacent sensor stripes. To detect footsteps from the data, the project has applied a statistical pattern matching method based on segmental semi-Markov models (SSMM). When using an SSMM in footstep detection, both shape variability and prior knowledge about the waveform are used to build an example footstep. Pattern matching begins when a sudden increase in energy of the EMFi signal is detected. A Viterbi-like algorithm is then used to find similar occurrences to the created footstep model. The first results are encouraging and the next step will be to improve the detection method to allow a combination of data from several stripes.
The project uses a two-level classifier system to identify footsteps. At the first level learning Vector Quantization (LVQ) is used for rejecting or accepting a single footstep. Then a sequence of three footsteps is accepted or rejected at the second level to make final decision. The system has been tested and produced a 90% success rate with a 20% rejection rate in a test with eleven walkers. With as many walkers, when the LVQ and Hidden Markov Model (HMM) classifiers where used separately to identify single steps, their success rates were 67% and 52%, respectively. Several simultaneous walkers still pose a problem which will be solved in the future.
Robot detection and recognition is probably easier as the robot's "footsteps" seems to differ from a human's ones considerably. Miniatures mobile robots are developed in the project as part of the smart environment. The robots will have powerful audio-video processing capabilities and sensors for proactive applications.
Footstep detection and identification is part of the context data that can be used to learn routines. A routine is a temporal context sequence that occurs often. Routine learning is used, e.g., to learn about important places of the user. The learning system notices that the user passes more time in certain locations and may ask the user to name these. The user is encouraged to use general names, like 'home' or 'office', in which case also all other systems that understand these concepts can start to utilize this information.
In a few tests, the project has used a Compaq iPAC PDA enhanced with a sensor box to collect context data in different scenarios. In the first tests, the context has been manually tagged, but later on real data will be used. Important locations were recognised where the user stayed more than five minutes at least twice during the scenario. By using association rules, other contexts or actions can be associated with these locations. Known contexts were acceleration (walking, sitting, walking upstairs/downstairs, in an elevator), illumination (inside, outside), location (office, library, lobby, stairway, restaurant). Also a PDA profile was used (general, silent, outdoors, meeting). In a usage situation, a rule may be presented to the user who can then choose to apply it permanently. Such a rule could switch off the mobile phone when the user enters a meeting or raise the volume of the ringing tone when the user steps outdoors.
Some recent publications related to the project are
- K. Koho, J. Suutala, T. Seppänen, J. Röning (2004): Footstep Pattern Matching from Pressure Signals Using Segmental Semi-Markov Models, 12th European Signal Processing Conference (EUSIPCO 2004), September 6-10, 2004, Vienna, Austria.
- S. Pirttikangas, J. Riekki, S. Porspakka, J. Röning. Know Your Whereabouts. 2004 Communication Networks and Distributed Systems Modeling and Simulation Conference (CNDS'04), January 19-22, San Diego, California, USA.
- S. Pirttikangas, J. Riekki, J. Röning: Routine learning: analyzing your whereabouts. Proc. International Conference on Information Technology: Coding and Computing (ITCC 2004), Volume: 2, April 5-7, 2004, 208 - 212.
- S. Pirttikangas, J. Suutala, J. Riekki. Röning (2003): Footstep Identification from Pressure Signals Using Hidden Markov Models, Finnish Signal Processing Symposium (FINSIG'03), pp. 124-128, May, 2003. Tampere, Finland.
- S. Pirttikangas, J. Suutala, J. Riekki, J. Röning: Learning vector quantization in footstep identification, 3rd IASTED International Conference on Artificial Intelligence and Applications (AIA 2003), September 8-10, 2003, Benalmadena, Spain, 413-417.
- J. Suutala: Methods for Person Identification from Pressure Signal of Walking Steps. Diploma Thesis. University of Oulu, Department of Electrical and Information Engineering, Oulu, Finland, 2004, 90 p., (in Finnish).
- J. Suutala, S. Pirttikangas, J. Riekki, J. Röning: Reject-Optional LVQ-Based Two-Level Classifier to Improve Reliability in Footstep Identification. Proc. 2nd International Conference on Pervasive Computing (Pervasive 2004), 21-23 April, 2004, Linz/Vienna, Austria, 182-187.
- J. Suutala and J. Röning (2004): Towards the Adaptive Identification of Walkers: Automated Feature Selection of Footsteps Using Distinction Sensitive LVQ , Int. Workshop on Processing Sensory Information for Proactive Systems (PSIPS 2004), June 14-15, Oulu, Finland.
More information
For more information, please contact Professor Röning at the University of Oulu or see the home page of the group at http://www.ee.oulu.fi/research/isg/ .
Research group
University of Oulu, Infotech Oulu and Department of Electrical and Information Engineering, Computing Engineering Laboratory, Intelligent Systems Group:
- Professors Juha Röning, Jukka Riekki, Tapio Seppänen
- Researchers Janne Haverinen, Kalle Koho, Susanna Pirttikangas, Jaakko Suutala, Antti Tikanmäki