Analysis and evaluation of visual information systems performance

Abstract

This dissertation investigates the system-centred evaluation of visual information retrieval from generic photographic collections. The development of visual information retrieval systems has long been hindered by the lack of standardised benchmarks. Researchers have proposed numerous systems and techniques, and although different systems clearly have their particular strength, there is a tendency by researchers to use different means of showing retrieval performance to highlight the own algorithm’s benefits. For the field of visual information search to advance, however, objective evaluation to identify, compare and validate the strengths and merits of different systems is therefore essential. Benchmarks to carry out such evaluation have recently been developed, and evaluation events have also been organised for several domains. Yet, no efforts have considered the evaluation of retrieval from generic photographic collections (i.e. containing everyday real-world photographs akin to those that can frequently be found in private photographic collections as well, e.g. pictures of holidays and events). We therefore first analyse a multitude of variables and factors with respect to the performance and requirements of visual information systems, and we then design and implement the framework and resources necessary to carry out such an evaluation. These resources include: a parametric image collection, representative search requests, relevance assessments and a set of performance measures. In addition, we organise the first evaluation event for retrieval from generic photographic collections and report on its realisation. Finally, we present an analysis and the evaluation of the participating retrieval systems as well as of the evaluation event itself. Filling this particular gap by making possible a systematic calibration and comparison of system performance for retrieval from generic photographic collections constitutes the main scientific contribution of this research. This dissertation thereby enables a deeper understanding of the complex conditions and constraints associated with visual information identification, the accurate capturing of user requirements, the appropriate specification and complexity of user queries, the execution of searches, and the reliability of performance indicators.