Papers
The Role of Subjective Factors in the Information Search Process
Co-authored with Irene Lopatovska. 2009 Journal of the American Society for Information Science and Technology (JASIS&T) 60(12)
We investigated the role of subjective factors in the information search process. Forty eight participants each conducted six web searches in a controlled setting. We examined relationships between subjective factors (happiness levels, satisfaction with and confidence in the search results, feeling lost during search, familiarity with and interest in the search topic, estimation of task difficulty), and objective factors (search behavior, search outcomes and search task characteristics). Data analysis was conducted using a multivariate statistical test (Canonical Correlations Analysis). The findings confirmed existence of several relationships suggested by prior research, including relationships between objective search task difficulty and the perception of task difficulty; between subjective states and search behaviors and outcomes. One of the original findings suggests that higher happiness levels before the search and during the search correlate with better feelings after the search, but also correlates with worse search outcomes and lower satisfaction, suggesting that, perhaps, it pays off to feel some ‘pain’ during the search in order to ‘gain’ quality outcomes.
Early view of this paper is available on Wiley InterScience: My & Irene Lopatovska's article on subjective factors in search is officially published in JASIST http://bit.ly/subj_factors_in_search
Like (2)Assessing Cognitive Load on Web Search Tasks
To appear in Open Ergonomics Journal.
Assessing cognitive load on web search is useful in examining which search system features, search tasks types and phases are more cognitively difficult for the searchers. It is also helpful in examining how individual differences among searchers (e.g. cognitive abilities) affect the search process. This investigation examined cognitive load from the perspective of primary and secondary task performance. A controlled web search study was conducted with 48 participants. The primary task performance components were found to be significantly related to both the objective and the subjective task difficulty. However, the relationship between objective and subjective task difficulty and the secondary task performance measures was weaker than expected. The results indicate that the dual-task approach needs to be used with caution.
What a Difference a Tag Cloud Makes: Effects of Tasks and Cognitive Abilities on Search Results Interface Use
2009. In Information Research, 14(4), paper 414.
The goal of this study is to expand our understanding of the relationships between selected tasks, cognitive abilities and search result interfaces. The underlying objective is to understand how to select search results presentation for tasks and user contexts.
Twenty three participants conducted four search tasks of two types and used two interfaces (List and Overview) to refine and examine search results. Clickthrough data were recorded. This controlled study employed a mixed model design with two within-subject factors (task and interface) and two between-subject factors (two cognitive abilities: memory span and verbal closure).
Quantitative analyses were carried out by means of the statistical package SPSS. Specifically, multivariate analysis of variance with repeated measures and non-parametric tests were performed on the collected data.
The overview of search results appeared to have benefited searchers in several ways. It made them faster; it facilitated formulation of more effective queries and helped to assess search results. Searchers with higher cognitive abilities were faster in the Overview interface and in less demanding situations (on simple tasks), while at the same time they issued about the same number of queries as lower-ability searchers. In more demanding situations (on complex tasks and in the List interface), the higher ability searchers expended more search effort, although they were not significantly slower than the lower ability people in these situations. The higher search effort, however, did not result in a measurable improvement of task outcomes for high-ability searchers.
These findings have implications for the design of search interfaces. They suggest benefits of providing result overviews. They also suggest the importance of considering cognitive abilities in the design of search results’ presentation and interaction.
Tag Trails: Navigating with Context and History
Co-authored with Philip Bakelaar, 2009. In CHI ’09 Extended Abstracts on Human Factors in Computing Systems (pp. 4579-4584).
We describe a technique for preserving and presenting context and history while navigating web resources described by keywords. We use tagging and tag clouds as an application area for our technique. The technique is illustrated by employing it in a prototype that interfaces data from a social tagging website used to bookmark academic articles. The prototype displays a “tag trail” which can reveal contextual connections between web resources and the associated tags. We argue that the user’s understanding of web resources is aided by making such connections explicit.
PDF: http://cli.gs/ngDRTM
Revisiting Search Task Difficulty: Behavioral and Individual Difference Measures
2008. In Proceedings of the 71st Annual Meeting of the American Society for Information Science and Technology (ASIS&T).
Search task characteristics are recognized as important factors that affect search process and its outcomes. We examine the relationships among operational measures of searcher’s behavior, individual cognitive differences, subjective task difficulty and mental effort assessed by dual-task performance. A web-based information study was conduced in a controlled experimental setting. Forty eight study participants performed six search tasks of varying type and structure. Subjective task difficulty was found to be influenced by the searcher’s effort measured as the number of result pages and individual documents visited, the number of documents marked as relevant, as well as by individual cognitive differences, and mental effort assessed by performance on the secondary task. In contrast to previous studies, no strong effects of user navigation graph structure were found.
PDF: http://cli.gs/tsd7Y7
Implicit Measures of Lostness and Success in Web Navigation
Co-authored with Ian Spence, 2007. Interacting with Computers, 19(3), 357-369.
In two studies, we investigated the ability of a variety of structural and temporal measures computed from a web navigation path to predict lostness and task success. The user’s task was to find requested target information on specified websites. The web navigation measures were based on counts of visits to web pages and other statistical properties of the web usage graph (such as compactness, stratum, and similarity to the optimal path). Subjective lostness was best predicted by similarity to the optimal path and time on task. The best overall predictor of success on individual tasks was similarity to the optimal path, but other predictors were sometimes superior depending on the particular web navigation task. These measures can be used to diagnose user navigational problems and to help identify problems in website design.
PDF (author version): http://cli.gs/66yWpa
PDF (Elsevier): http://cli.gs/WAt095
Everything through Email
Co-authored with Steve Whittaker and Victoria Bellotti, 2007. In W. Jones and J. Teevan (Eds.), Personal Information Management (pp. 167-189). Seattle: University of Washington Press.
Email serves as an information conduit—acting as a delivery channel for different types of information, including documents, slide, contact information, and schedules. Its use as a conduit naturally leads to its being used for key PIM functions. People use their inboxes as to-do lists to manage current tasks, their email folders as a repository for archival information, and their email address books to find contacts. They even use email to find and schedule calendar appointments.
Despite its success, there are significant problems with email. Users complain about feeling overwhelmed by the volume of messages they receive, and are concerned about processing incoming messages effectively (Bälter & Sidner 2002; Bellotti et al. 2005; Venolia, Gupta, Cadiz & Dabbish 2001; Whittaker & Sidner 1996). They have difficulties, too, in organizing and managing archives (Bälter & Sidner 2002; Boardman & Sasse 2004; Whittaker & Sidner 1996). They experience severe problems using email to manage tasks, leading them to forget tasks and obligations (Bellotti, Ducheneaut, Howard & Smith 2003; Bellotti et al. 2005; Whittaker 2005; Whittaker, Jones & Terveen 2002). Email's failure to address these PIM problems threatens to seriously reduce individual and corporate productivity.
Why is email so hard to process and manage and why does it generate such serious personal information management problems? And yet why is it still so popular and so essential in PIM? And what can be done to make it better able to support this essential role? This chapter will illustrate how many of email's PIM problems ste from its conduit function. Email is incessant, because it is often the primary delivery channel for work and information (Ducheneaut & Bellotti 2001; Venolia et al. 2001; Whittaker 2005). It is hard to process and organize because it is a mixture of different types of information (tasks, documents, FYIs, meeting scheduling), some of which are important (work tasks) and others unimportant (jokes). And most email is generated by others—making it harder to understand, evaluate, and organize than personally generated information (Boardman & Sasse 2004). These problems are exacerbated by the fact that most email systems have no inbuilt support for PIM aside from folders, so that users have to devise ad hoc ways to manage tasks, find contacts, and organize useful information (Bellotti et al. 2003, 2005; Whittaker 2005; Whittaker & Sidner 1996).
This chapter discusses how people use email to manage their personal information and describes the tools that can support such behavior. We look at important aspects of PIM through the lens of email: allocating attention, deciding actions, managing tasks, and organizing messages into folders.
Individual Differences
Co-authored with Mark H. Chignell, 2007. In W. Jones and J. Teevan (Eds.), Personal Information Management (pp. 206-220). Seattle: University of Washington Press.
In an increasingly complex world where people routinely handle large amounts of information, individuals are constantly challenged to manage and effectively use the information that they are responsible for. While email is the canonical example of an information overloading application, other well known PIM applications and tasks cited in earlier chapters of this book include maintaining addresses and contacts, scheduling, and organizing the various documents and bookmarks that one is interested in. Not surprisingly, there are individual differences (ID) in how, and how well, people cope with the challenge of personal information management. This greatly complicates any scientific analysis of PIM behavior. Thus, in addition to the evaluation methods discussed in the previous chapter, researchers and designers need to consider when and how individual differences should be included within parsimonious interpretations and explanations of PIM behavior. In this chapter we propose an approach where differences between individuals are considered last, after the influences of the environment and the task context have first been considered, and after group difference (e.g., between job classifications) have been investigated. We believe that this is a logical way to proceed, since like observing an ant walking over sand-dunes (cf. Simon, 1996) we should not ascribe complexities to an individual if they can instead be explained as due to properties of the environment.
The goal of this chapter will be to review and synthesize some of the key findings in how PIM behavior differs between individuals. Some of the reasons why these differences occur and what can be done about them will also be discussed.
Finding It on Google, Finding It on del.icio.us
Co-authored with Michael J. Cole, 2007. In L. Kovács, N. Fuhr, & C. Meghini (Eds.), Lecture Notes in Computer Science: Vol. 4765. Research and advanced technology for digital libraries (pp. 559-562). London: Springer-Verlag.
We consider search engines and collaborative tagging systems from the perspective of resource discovery and re-finding on the Web. We performed repeated searches over nine-months on Google and del.icio.us for web pages related to three topics selected to have different dynamic characteristics. The results show differences in the resources they provide to the searcher. The resources tagged on del.icio.us differ strongly from the top results returned by Google. The results also suggest the changes in the most recently tagged web pages may be associated with the level of activity in user communities and, indirectly, with external events.
Email in Interpersonal Information Management
Co-authored with Steve Whittaker and Victoria Bellotti, 2006. Communications of the ACM, 49(1), 68-73.
For many of us, work is interpersonal rather than solitary, and email is the main conduit through which that work and its related information are distributed [1]. We tend to live in our email, as reflected in the amount of time we spend using it and our evaluation of its importance in everyday work. Email’s role as conduit naturally leads to it being used for three key functions in personal information management (PIM): task management, personal archiving, and contact management.
What Can Searching Behavior Tell Us About the Difficulty of Information Tasks? A Study of Web Navigation
Co-authored with Ian Spence, 2006. In Proceedings of the 69th Annual Meeting of the American Society for Information Science and Technology (ASIS&T).
Task has been recognized as an influential factor in information seeking behavior. An increasing number of studies are concentrating on the specific characteristics of the task as independent variables to explain associated information-seeking activities. This paper examines the relationships between operational measures of information search behavior, subjectively perceived post-task difficulty and objective task complexity in the context of factual information-seeking tasks on the web. A question-driven, web-based information-finding study was conducted in a controlled experimental setting. The study participants performed nine search tasks of varying complexity. Subjective task difficulty was found to be correlated with many measures that characterize the searcher’s activities. Four of those measures, the number of the unique web pages visited, the time spent on each page, the degree of deviation from the optimal path and the degree of the navigation path’s linearity, were found to be good predictors of subjective task difficulty. Objective task complexity was found to affect the relative importance of those predictors and to affect subjective assessment of task difficulty
PDF: http://cli.gs/8hTLp3
Predicting Outcomes of Web Navigation
Co-authored with Ian Spence, 2005. In Special Interest Tracks and Posters of the 14th International World Wide Web Conference (pp. 892-893).
Two exploratory studies examined the relationships among web navigation metrics, measures of lostness, and success on web navigation tasks. The web metrics were based on counts of visits to web pages, properties of the web usage graph, and similarity to an optimal path. Metrics based on similarity to an optimal path were good predictors of lostness and task success.
Indirect Assessment of Web Navigation Success
Co-authored with Ian Spence, 2005. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (pp. 1427-1430).
Despite much research on hypertext and web navigation, relatively little is known about the relationship between web navigation strategies and success. We present two exploratory studies designed to explore the relationships between several web navigation metrics that are based on similarity to an optimal path to predict task success. The data suggest that the relationships between these measures depend on the particular web navigation task.
Individual Differences and Task-Based User Interface Evaluation: A Case Study of Pending Tasks in Email
Co-authored with Mark H. Chignell, 2004. Interacting with Computers, 16(4), 769-797.
This paper addresses issues raised by the ever-expanding role of email as a multi-faceted application that combines communication, collaboration, and task management. Individual differences analysis was used to contrast two email user interfaces in terms of their demands on users. The results of this analysis were then interpreted in terms of their implications for designing more inclusive interfaces that meet the needs of users with widely ranging abilities.
The specific target of this research is the development of a new type of email message representation that makes pending tasks more visible. We describe a study that compared a new way of representing tasks in an email inbox, with a more standard representation (the Microsoft Outlook inbox). The study consisted of an experiment that examined how people with different levels of three specific cognitive capabilities (flexibility of closure, visual memory, and working memory) perform when using these representations. We then identified combinations of representation and task that are disadvantageous for people with low levels of the measured capabilities.
Email Task Management Styles: The Cleaners and the Keepers
2004. In CHI ’04 Extended Abstracts on Human Factors in Computing Systems (pp. 1235-1238).
Email has become overloaded as users make use of email tools for performing a wide range of activities. Previous studies have demonstrated the different strategies employed by email users to manage messages. However, we have little information regarding how to explain those differences between users. The research described in this paper seeks to gain understanding of individual differences in email behaviour. We present results from a questionnaire-based study, which focused on how email users dealt with messages that relate to future tasks or events. We identified two types of user, defined by how they dealt with such messages: the cleaners and the keepers. The difference between these two groups can be attributed to differences in email experience and requirements for flexibility of closure. The ultimate goal of such research is to be able to predict differences in email use and to inform email user interface design and we discuss possible ways in which this could be done.
The Privacy Attitude Questionnaire (PAQ): Initial Development and Validation
Co-authored with Mark H. Chignell and Anabel Quan-Haase, 2003. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting: Vol. 47. Internet (pp. 1326-1330).
Privacy has been identified as a key issue in a variety of domains, including electronic commerce and public policy. While there are many discussions of privacy issues from a legal and policy perspective, there is little information on the structure of privacy as a psychometric construct. Our goal is to develop a method for measuring attitudes towards privacy that can guide the design and personalization of services. This paper reports on the development of an initial version of the PAQ. Four privacy attitudes are identified based on the factor structure of the PAQ. Cluster analysis is used to identify potential stereotypes with respect to attitudes towards privacy amongst different groups of people. Version 1.0 of the PAQ is presented in an Appendix as a 36 item questionnaire that measures the four privacy attitudes of personal information, monitoring, exposure and protection.
The Privacy Attitude Questionnaire (PAQ) and Internet Use
Co-authored with Mark H. Chignell and Anabel Quan-Haase, 2003. Presented at the annual conference of the Association of Internet Researchers, Toronto, Ontario, Canada.
The goals of this research include a cataloguing of the different privacy attitudes that exist, and the use of associated privacy measures to predict Internet behaviours and willingness to accept privacy policy statements made by websites. It is our expectation that this type of empirical research can contribute greatly to discussions of attitudes towards privacy and their impact on Internet behaviour. This work should lead to an empirically-based and multidimensional construct of privacy, plus a more detailed understanding of how different groups of people differ according to their attitudes toward the different components of privacy.
TaskView: Design and Evaluation of a Task-Based Email Interface
2002. In J.H. Johnson & D.A. Stewart (Eds.), Proceedings of the 2002 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON)(pp. 136-145).
Email was originally designed as a tool for asynchronous communication. However, as the number of messages increased, so did their variety. A wide range of new and unforeseen email tasks reflects this variety. One of the most commonly performed activities in email is management of pending tasks. This research focuses on how to support this activity in email and explores solutions that use different external representations of messages and associated tasks. Central to this research is understanding the role of both external artifacts in managing multiple pending tasks, as well as internal representations and processes and how they can be linked to external representations. In a recent study we compared the effects of two email interfaces (Microsoft Outlook Inbox and TimeStore-TaskView) on efficiency and effectiveness of information finding in email messages. We found that TimeStore-TaskView interface was overall faster for finding information related to task dates, time and task overviews, while the Inbox interface was faster for finding information from subject lines, senders or from the message body. Based on the results from the study, we are in the process of designing a modified email prototype and a follow-up user study.
Notable: At the Intersection of Annotations and Handheld Technology
Co-authored with Michelle Baldonado, Steve Cousins, and Andreas Paepcke, 2000. In P. Thomas & H. Gellersen (Eds.), Lecture Notes in Computer Science: Vol. 1927. Proceedings of the 2nd International Symposium on Handheld and Ubiquitous Computing (pp. 333-343). London: Springer-Verlag.
The Notable annotation system enables users to annotate paper documents using handheld devices in a mobile environment. This paper describes the design issues and solutions that arose in creating Notable, with a particular focus on design challenges at the intersection of annotations and handheld technology. Novel design strategies include separating the annotation writing platform from the document viewing platform, providing search as the method for document selection, offering context-sensitive phrase completion and icon-based graphical pinning for fine-granularity annotation anchoring, and including some support for coordinating group annotation activity.
Discriminating Meta-Search: A Framework for Evaluation
Co-authored with Mark H. Chignell and Richard C. Bodner, 1999. Information Processing & Management, 35(3), 339-364.
There was a proliferation of electronic information sources and search engines in the 1990s. Many of these information sources became available through the ubiquitous interface of the Web browser. Diverse information sources became accessible to information professionals and casual end users alike. Much of the information was also hyperlinked, so that information could be explored by browsing as well as searching. While vast amounts of information were now just a few keystrokes and mouseclicks away, as the choices multiplied, so did the complexity of choosing where and how to look for the electronic information. Much of the complexity in information exploration at the turn of the twenty-first century arose because there was no common cataloguing and control system across the various electronic information sources. In addition, the many search engines available differed widely in terms of their domain coverage, query methods and efficiency.
Meta-search engines were developed to improve search performance by querying multiple search engines at once. In principle, meta-search engines could greatly simplify the search for electronic information by selecting a subset of first-level search engines and digital libraries to submit a query to based on the characteristics of the user, the query/topic, and the search strategy. This selection would be guided by diagnostic knowledge about which of the first-level search engines works best under what circumstances. Programmatic research is required to develop this diagnostic knowledge about first-level search engine performance.
This paper introduces an evaluative framework for this type of research and illustrates its use in two experiments. The experimental results obtained are used to characterize some properties of leading search engines (as of 1998). Significant interactions were observed between search engine and two other factors (time of day and Web domain). These findings supplement those of earlier studies, providing preliminary information about the complex relationship between search engine functionality and performance in different contexts. While the specific results obtained represent a time-dependent snapshot of search engine performance in 1998, the evaluative framework proposed should be generally applicable in the future.
FotoFile: A Consumer Multimedia Organization and Retrieval System
Co-authored with Allan Kuchinsky, Celine Pering, Michael L. Creech, Dennis Freeze, and Bill Serra, 1999. In M.G. Williams & M.W. Altom (Eds.), Proceedings of the SIGCHI Conference Summary on Human Factors in Computing Systems: The CHI is the Limit (pp. 496-503).
FotoFile is an experimental system for multimedia organization and retrieval, based upon the design goal of making multimedia content accessible to non-expert users. Search and retrieval are done in terms that are natural to the task. The system blends human and automatic annotation methods. It extends textual search, browsing, and retrieval technologies to support multimedia data types.
Categorization is Difficult: Use of an Electronic Notebook for Organizing Design Meeting Notes
1998. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting: Vol. 42. Computer Systems (pp. 516-520).
The electronic engineering notebook is a pen-based computer tool designed to capture engineering notes and to assist in structuring them to facilitate subsequent information access. Experiments were conducted in which free-form and fixed-form note-taking interfaces were compared with respect to capturing and structuring notes from a design meeting. Suitability of domain-based and user-defined terminologies for semantic structuring was also tested. Results from the experiment confirmed that a free-form interface was easier to use for note-taking and structuring notes than were forms with fixed structure. However, difficulties with applying semantic categorizations were observed in both types of interface. The results indicated that terminology should be chosen appropriately to a task, and that experienced users have less problems with applying it. The experiment also demonstrated high diversity in note-taking styles, thus highlighting the need to accommodate individual preferences in electronic notebooks.
EEN: A Pen-Based Electronic Notebook for Non-Intrusive Acquisition of Engineering Knowledge
Co-authored with Jim Louie and Mark S. Fox, 1996. In WET-ICE ‘96: Proceedings of the IEEE Workshop on Enabling Technologies: Vol 5. Infrastructure for Collaborative Enterprises (pp. 40-46).
Capturing design information and decisions is critical to supporting the re-use of designs and coordination in engineering teams. Most of this information is currently not recorded at all. In this paper, we focus on the role of electronic engineering notebooks in the capture, storage and dissemination of design information and decisions, and on their role in integrating and managing design decisions and processes. We propose EEN (Electronic Engineering Notebook) as a tool for assisting engineers in information-related tasks, and ultimately leading to the replacement of traditional paper engineering notebooks. This paper presents the requirements, design and implementation of EEN, as well as initial experimental results.
Add Comment