Application of Variable Length N-gram Vectors to Monolingual and Bilingual Information Retrieval

Abstract

Our group in the Department of Informatics at the University of Oviedo has participated, for the first time, in two tasks from CLEF: monolingual (Russian) and bilingual (Spanish-to-English) information retrieval. Our main goal was to test the application to IR of a modified version of n-gram vector space model (codenamed blindLight). This new approach has been successfully applied to other NLP tasks such as language identification or text summarization and the results achieved at CLEF'04, although not exceptional, are encouraging. Major differences between the blindLight approach and classical techniques are two: (1) relative frequencies are no more used as vector weights but replaced by n-gram significances, and (2) cosine distance is abandoned in favor of a new metric inspired by sequence alignment techniques although not so computationally expensive. In order to perform cross-language IR we have developed a naive n-gram pseudo-translator similar to those described by McNamee and Mayfield or Pirkola et al.