Competency E
Design, query, and evaluate information retrieval systems.
Introduction
Before we discuss designing, querying, and evaluating information retrieval systems, we have to define two key terms: documents and information retrieval systems(IRSs).
Weedman (2018) defines documents as information-bearing entities that provide information. Users access information retrieval systems to retrieve documents or satisfy an information need.
The traditional model of an IRS is as follows, and its main goals, as detailed in Tucker (2020a), are to aggregate (find all of the desired documents) and discriminate (find only the desired documents)
Image 1.0: A diagram detailing the components of a traditional IRS model. (Adapted from Tucker, 2020a)
Information professionals should understand the three main spheres of the information service field- design, querying, and evaluation. More specifically (Weedman, 2018):
outlining the problem to be solved, identifying the users who will be using the information system
how the individual elements will be represented so that they are both findable and usable
The fundamentals of how information retrieval systems work
Evaluation at every stage of development
Design
The design of information systems is similar to how you would organize anything in your house; your kitchenware, shoes, clothes, and books. There are several things you have to consider to make the collection easier to search:
What are the main categories that things should be grouped in?
Are subcategories necessary?
What do you do with the things that don’t fit into these defined categories?
Design is more than simply organizing items; it is also problem-solving. A solution's effectiveness depends on whether the problem can be found and then defined, meaning that observed problems may not be the true problems (Weedman, 2018). Another consideration is that design does not necessarily define finite solutions; it is up to designers to evaluate which solution will best solve the problem. The defined solutions include solutions used for other problems and adapted for this specific case to entirely original solutions.
The Initial Problem Definition Phase
There are various ways to define a problem (and the ultimate user group) entirely. The one picked depends on the resources (fiscal, temporal, and human) available to the project team. The research from this phase should generate a requirements analysis report informing the next steps of information systems design. There are a variety of ways in which this research can be conducted (Schmidt, 2021):
surveys
informal or formal usability testing
paper prototyping
ranking
ethnography (direct observation of users)
using software to track user's actions (heat mapping)
The Problem is Defined, Now What?
Once a problem has been defined, next is figuring out how to represent and store the information in the system. This is where metadata comes in. Metadata is “structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use, or manage an information resource” (Weedman, 2018).
Metadata is created by using a combination of the following approaches- a standardized set of searchable terms (controlled vocabularies), the natural language of the document, or categorical classification to represent a document. A new approach slowly being incorporated into metadata creation is social tagging. Social tagging is a blend of controlled vocabularies and natural language approaches, where social network users can choose their labels (usually hashtags) for their documents (Weedman, 2019a).
Aggregation vs. Discrimination- Why is it Important to Good IRS Design?
Aggregation and discrimination are related to recall and precision. Aggregation or recall is a percentage measurement of relevant documents retrieved. Discrimination or precision is a percentage measurement of the retrieved relevant documents. There is always a trade-off between the two (Aggregation/ Recall vs. Discrimination/ Precision), which is important for IRS designers and users to recognize.
Query
Fundamental to a user’s query is that their information need is a lack of knowledge (we don’t know what we don’t know). According to Belkin’s anomalous state of knowledge, they are then required to describe the information they don’t know about (Tucker, 2020b).
The search process is complex. These are the main phases in the iterative process.
Recognize the information need.
Outline the need.
Identify an entry point (this step highly depends on the user’s experience with seeking information).
Create and execute a query.
Assess the results and extract what can be used (We refer to this process as iterative because the user can be either satisfied with their search results or return to any of the previous steps to refine the search. The query will evolve as the user develops a better understanding of what they are looking for and how the information in the system is organized (or indexed).
Query vs. Browsing: What is the Difference?
As exemplified in the process above, the queries are more analytical, goal-driven, and planned. Different from when a user browses, the goal needs to be defined. This does not mean that browsing is second to a query. Analytical searches and browsing exist on a continuum (searching is a push and pull between the two).
Browsing depends more on scanning and recognizing relevant information when it appears. There are several types of browsing: scanning, observing, navigating, and monitoring. This type of searching might take longer than a query, take place over various information systems and sometimes help users flesh out their information needs. Overall the process is one of exploration and serendipity.
What Querying Means for IRS
A traditional IRS assumes that the search is one where the user needs to know what they don’t know (pull). As the information landscape and technology continue to grow and evolve, IRSs start to push searches- offering users unsolicited information (Tucker, 2019b).
Evaluate
Evaluation is something that connects design and querying. Information professionals constantly evaluate which information systems to use when answering patrons’ questions (Tucker, 2019a). Information professionals can use the answers to the following three main questions to evaluate an information retrieval system:
What happens when the IRS is used?
What results did the query generate?
What do we know about the IRS?
What Do We Measure When an IRS is Used?
Several variables can be evaluated when an IRS is used. They include the documents in the IRS, the quality of the surrogates (representatives of the documents within the IRS), the indexing language, how the search engine functions, the user interface, the user’s proficiency in developing appropriate queries, and finally, user satisfaction with the experience and results generated (relevance of the results)(Tucker, 2020c)(Weedman, 2018). When we speak of the results' relevance, we ask how well the system aggregates (recalls) all documents relevant to the query and how well the system discriminates (precision) between relevant and irrelevant documents (Tucker, 2020c).
The Future of IRS Research
While the measurements discussed in the previous section help determine an IRS’s effectiveness, it is also essential to understand that they are highly dynamic (linked to users’ information needs, assuming that we know certain variables, such as how many relevant documents exist in the database). This is why there has been a shift toward research that focuses on IRS users and their information-seeking behaviours (Tucker, 2019a).
Evidence
Gummy Candy Database
INFO 202 with Dr. Virginia Tucker
Group Members: Lisa Danes, Brayden Kelley, Lydia Lopez, Sabrina Weegar & Bailey Wells
Description
In a group, we were tasked with designing a data structure for a collection of non-traditional items; we chose Gummy Candies. All the work for this assignment was completed online. The first part of the assignment included the following tasks:
Developing a data structure for a database of records
Creating and manipulating database files using a database management system
Writing a statement of purpose for a specific user group and a set of rules that indexers would use to create database records
For the second part, we were asked to beta test and evaluate the database designs of other groups.
My role in this project was the leader and the techie. As the leader, I was responsible for organizing meetings, keeping our group accountable to our set completion schedule, and editing the final project materials. I also took on the techie role and created the database according to the group’s design notes because I was the most comfortable using the WebDataPro application. I also used this role as an opportunity to work on my website design and coding skills, taking the time to think about the user experience of our database.
Justification
I chose this assignment for Competency E because it demonstrates that while completing this project, I gained valuable insight into the fundamentals and process of how a database is formed. While we only went as far as the beta testing phase, I learned that creating and maintaining a user-friendly database is often an ongoing process where the designer adds, subtracts, or modifies database elements based on user feedback. I also learned that documentation is essential to maintain consistency and enables the team to plan out system capabilities and features properly.
Design Challenge- Redesigning Book Trucks
INFO 282 Design Thinking with Steven Bell
Partner: Amanda Fairchild
Description
For this design challenge assignment, we were placed into pairs and tasked with creating a revised book truck design. The first step involved using empathic design principles and ethnographic research to generate background information on the book trucks we saw around our local libraries. This phase was mostly done remotely as we were still in the grips of the COVID-19 pandemic. We then used this information to identify a problem with the book trucks. After identifying the problem (How we might make book trucks safer and more multifunctional?), we brainstormed, prototyped, and created implementation instructions and evaluation parameters.
Justification
I chose this assignment because it is an excellent example of my design skills. My partner and I effectively used ethnographic research to identify the primary users of book trucks and then comprehensively define the problem many of these users faced with cede book trucks.
Usability Testing Assignment
INFO 287 UX with Aaron Schmidt
Description
This assignment was a practical exercise in conducting usability studies of a library website. First, we were to set up a usability test that we would ultimately administer to 2 test subjects. This usability test consisted of an activity that the testers would have to complete during the test, a script that I would use as the test administrator.
Justification
This assignment was selected as evidence for Competency E because it shows that I can design and administer an evaluation of an information system. It demonstrates that I can synthesize all the information gathered during the test into a concise and informative report for clients and understand the metrics contributing to an information system (website)’s usability.
Conclusion
An information professional is better equipped to work with clients and their information needs by understanding information retrieval systems' design, use, and evaluation. They can understand how IRSs are structured so that they can more effectively use queries and identify resources that fit a user’s information needs. They understand the trade-off between recall and precision, understanding that you can have a list of returned results that is high in both. Finally, they can evaluate which IRSs will be more useful for their users and their information needs and therefore know to direct resources toward these systems.
References
Krug, S. (2013). Don’t Make Me Think, Revisited: A Common Sense Approach to Web Usability (3rd ed.) [EBook]. New Riders.
Schmidt, A. (n.d.). Lecture #3: Usability Testing [Slide show; Powerpoint]. Lecture, United States of America.
Tucker, V. M. (2020a). Lecture #1: Fundamental Concepts [Slide show; Powerpoint]. Lecture, United States of America.
Tucker, V. M. (2020b). Lecture #11: Searching and Querying Systems [Slide show; Powerpoint]. Lecture, United States of America.
Tucker, V. M. (2020c). Lecture #12: Evaluation of Information Retrieval Systems [Slide show; Powerpoint]. Lecture, United States of America.
Tucker, V. M. (2019a). Evaluation [EBook]. In V. M. Tucker (Ed.), Information Retrieval System Design: Principles & Practice (6th ed., pp. 349–357). Academic Pub.
Tucker, V. M. (2019b). Search [EBook]. In V. M. Tucker (Ed.), Information Retrieval System Design: Principles & Practice (6th ed., pp. 317–326). Academic Pub.
Weedman, J. (2018). Information Retrieval: Designing, Querying, and Evaluating Information Systems [Print]. In K. Haycock & M.-J. Romaniuk (Eds.), The Portable MLIS (2nd ed., pp. 171–186). Libraries Unlimited.
Weedman, J. (2019a). Designing for Search [EBook]. In V. M. Tucker (Ed.), Information Retrieval System Design: Principles & Practice (6th ed., pp. 119–139). Academic Pub.
Weedman, J. (2019b). The Design Process [EBook]. In V. M. Tucker (Ed.), Information Retrieval System Design: Principles & Practice (6th ed., pp. 220–232). Academic Pub.