Automatic new topic identification in search engine transaction logs

Abstract

Purpose - Content analysis of search engine user queries is an important task, since successful exploitation of the content of queries can result in the design of efficient information retrieval algorithms of search engines, which can offer custom-tailored services to the web user. Identification of topic changes within a user search session is a key issue in content analysis of search engine user queries. The purpose of this study is to address these issues. Design/methodology/approach - This study applies genetic algorithms and Dempster-Shafer theory, proposed by He et al., to automatically identify topic changes in a user session by using statistical characteristics of queries, such as time intervals and query reformulation patterns. A sample data log from the Norwegian search engine FAST (currently owned by overture) is selected to apply Dempster-Shafer theory and genetic algorithms for identifying topic changes in the data log. Findings - As a result, 97.7 percent of topic shifts and 87.2 percent of topic continuations were estimated correctly. The findings are consistent with the previous application of the Dempster-Shafer theory and genetic algorithms on a different search engine data log. This finding could be implied as an indication that content-ignorant topic identification, using query patterns and time intervals, is a promising line of research. Originality/value - Studies an important dimension of user behavior in information retrieval.