Prima - site visit 2

Proactive Information Retrieval by Adaptive Models of Users' Attention and Interests (PRIMA), a joint project between the Helsinki University of Technology, the Helsinki Institute for Information Technology HIIT and the Helsinki School of Economics.

Site visit on 22 June 2004 

The project performs basic research on models underlying personal assistants: By monitoring users, the project tries to build models of users' interests and intentions. These models can be used for disambiguating commands, anticipating actions and enhancing abilities. In this project, the researchers focus on a specific application namely proactive information retrieval. Explicit feedback is replaced by implicit feedback inferred from the eye movements of the user. The eye movements are incorporated into a comprehensive information retrieval model as the basis for proactive retrieval. The problem settings will, however, be abstracted to make solutions more general-purpose.

Eye movements provide one kind of data that can be used for inferring if a document is relevant and which parts of the document that are relevant. In a more advanced setting, the eye movement data may be combined with explicit feedback, information about the document and document collection in question as well as collaborative filtering, the user's general interest profile and the history of the search session. As a first approach, the project has studied if eye movements alone can be used to predict relevance. 

The first results show that relevance can be inferred from eye movements to some extent. In the first information search experiment (2003), subjects were asked to find correct answers from a given set of text. 500 news story titles representing three topic areas were used as the domain. The subjects were asked 1) to pick out certain words, 2) to find the correct answer (title) to a question, and 3) to find an interesting title. For example, when the subjects were asked to find the correct answer, they were first shown the question for seven seconds, and then asked to find the correct answer out of ten possible titles (out of which one was correct, four others related to the same theme and five randomly picked).

The researchers were able to extract 21 different features from the eye movement data. The best distinguishing features were found to be 1. the total number of fixations on a word, 2. the total duration of a fixation on a word, 3. the sum of durations of all regressions initiating from the word, and 4. the standard deviation of the pupil diameter.  The results showed that relevance could be predicted and that the data was not too noisy for time series modelling. The researchers used hidden Markov models (HMMs) and discriminate hidden Markov models to predict the relevancy from the movements. Even the simple HMM structures modelling time series of the movements improved on the results compared with non-sequence models. The project will continue to develop enhanced models. 

A second experiment will be performed during 2004. Subjects will read movie synopses (from the www.allmovie.com database) and rate how interesting they find the movie. Revealing details such as director and actor names have been excluded from the texts. During the experiment, the eye movements of the readers will be recorded. The researchers hope to compare the given ratings with the movements to see of high (or low) interest can be inferred from the movement data.

In a related experiment, the research investigated if the textual content of a relevant document can be used when estimating the relevance of a new document. In the test setting, 1398 movie synopses were gathered together with their ratings by more than 70,000 users from the EachMovie database. Given a set of synopses, known to be either relevant or irrelevant to a certain user, the researchers estimate if an additional movie synopsis could be relevant to this particular user. (The estimation result is of course compared with the correct answer from the movie database). The results show that even sophisticated unsupervised methods such as multinomial PCA cannot help much. But if feature extraction supervised by relevant auxiliary data (such as the genre of the movies in this case) is used, the performance was improved.

One of the problems was that only single user data was used. Such data was scarce and not enough for constructing reliable user-specific models. Better results are expected by instead employing user groups that generalize over users and document clusters that generalize over documents as well as explicit ratings. In the next experiments such data will be combined with implicit feedback inferred from eye movements to improve relevance estimation. Combination of such data is quite difficult and the resulting models (as well as the estimation results) are expected to go beyond state of the art in this area.

As on offspring from the experiments and collecting eye movement data, the project's relevance prediction contest was accepted as a challenge competition of the Pascal network-of-excellence. The competition will be open between 1 March and 1 September 2005. The competition consists of two parts. In the first part, the competitors will be given time series of feature vectors (with 21 features as found above). The task is to classify the sequence to one of the classes 'correct answer', 'relevant', and 'irrelevant'. In the second task, the data is the unprocessed sequences of two-dimensional coordinates telling where the user's gaze was pointed. The data is complemented with the locations of the words on the screen. The task is to classify the trajectory into the same classes as above. For more information about the contest, see www.pascal-netwrk.org/Challenges/IREM/

The project intends to develop their first simple version of a proactive information retrieval system in 2004 and the next extended and enhanced version will be released in 2005...

Recent publications related to the project

• Jaana Simola, Nelli Salminen, and Ilpo Kojo: Eye movements in an information search task. European Conference on Visual Perception (ECVP) 2003, Paris, France, 1-5 September 2003.
• Jarkko Salojärvi, Ilpo Kojo, Jaana Simola, and Samuel Kaski: Can relevance be inferred from eye movements in information retrieval? In Proceedings of the Workshop on Self-Organizing Maps (WSOM'03), Hibikino, Kitakyushu, Japan, September 2003. pp. 261-266.
• Jarkko Salojärvi, Kai Puolamäki, and Samuel Kaski: Relevance feedback from eye movements for proactive information retrieval. In Workshop on Processing Sensory Information for Proactive Systems (PSIPS 2004), OuluFinland, June 14-15, 2004, pp. 37-42. Also published as a poster at the Machine Learning Meets the User Interface workshop at NIPS'03.
• Eerika Savia, Samuel Kaski, Ville Tuulos and Petri Myllymäki: On Text-Based Estimation of Document Relevance. In Proceedings of the International Joint Conference on Neural Networks (IJCNN'04), Budapest, Hungary, July 2004.

Research consortium

• HelsinkiUniversity of Technology, Laboratory of Computer and Information Science, Neural Networks Research Centre (NNRC), Research Group on Learning Methods for Data Exploration:
• Professor Samuel Kaski, researchers Jarkko Salojärvi, Eerika Savia, affiliated researchers Kai Puolamäki, Janne Sinkkonen
• Helsinki School of Economics, Center for Knowledge and Innovative Research, Vision Team:
• Docent Ilpo Kojo, researcher Jaana Simola, Mikko Berg
• Helsinki Institute for Information Technology, Complex Systems Computation Group:
• Professor Petri Myllymäki, researcher Miikka Miettinen, Ville Tuulos, affiliated researchers Hannes Wettig

More information 

For more information, please see the web pages of the project at http://www.cis.hut.fi/projects/mi/prima/ or contact the coordinator of the project, Professor Sami Kaski at the University of Helsinki.