p337-wong - Application of Aboutness to Functional...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon
Application of Aboutness to Functional Benchmarking in Information Retrieval KAM-FAI WONG The Chinese University of Hong Kong DAWEI SONG, PETER BRUZA University of Queensland and CHUN-HUNG CHENG The Chinese University of Hong Kong Experimental approaches are widely employed to benchmark the performance of an information retrieval (IR) system. Measurements in terms of recall and precision are computed as performance indicators. Although they are good at assessing the retrieval effectiveness of an IR system, they fail to explore deeper aspects such as its underlying functionality and explain why the system shows such performance. Recently, inductive (i.e., theoretical) evaluation of IR systems has been proposed to circumvent the controversies of the experimental methods. Several studies have adopted the inductive approach, but they mostly focus on theoretical modeling of IR properties by using some metalogic. In this article, we propose to use inductive evaluation for functional benchmarking of IR models as a complement of the traditional experiment-based performance benchmarking. We define a functional benchmark suite in two stages: the evaluation criteria based on the notion of “aboutness,” and the formal evaluation methodology using the criteria. The proposed benchmark has been successfully applied to evaluate various well-known classical and logic-based IR models. The functional benchmarking results allow us to compare and analyze the functionality of the different IR models. Categories and Subject Descriptors: H.1.1 [ Models and Principles ]: Systems and Information Theory; H.3.3 [ Information Storage and Retrieval ]: Information Search and Retrieval— re- trieval models ; search process ; selection process ; H.3.4 [ Information Storage and Retrieval ]: Systems and Software— performance evaluation ( efficiency and effectiveness ); H.3.m [ Information Storage and Retrieval ]: Theoretical Study of Information Retrieval General Terms: Measurement, Performance, Theory Additional Key Words and Phrases: Aboutness, functional benchmarking, inductive evaluation, logic-based information retrieval This project is partially supported by the Chinese University of Hong Kong’s strategic grant (project ID: 44M5007), and by the Cooperative Research Centres Program through the Department of the Prime Minister and Cabinet of Australia. Authors’ addresses: K.-F. Wong, C.-H. Cheng, Department of Systems Engineering & Engineering Management, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong; email: { kfwong, ccheng } @se.cuhk.edu.hk; D. Song, P. Bruza, Distributed Systems Technology Centre, Building 78, Staff House Road,University of Queensland,Qld4072 Australia;email: { dsong, bruza } @dstc.edu.au. Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers,
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 05/02/2011 for the course SWE 1120 taught by Professor Miles during the Spring '11 term at California Baptist University.

Page1 / 34

p337-wong - Application of Aboutness to Functional...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online