Elements for defining relevance – what the heck could they be?

Relevance

Julie Arie, Walter Lewis, Michael Vandenburg, Ron Davies, Kathleen Matthews, William Denton

The Problem - #20 Relevant Relevance.

Proposed by Mark Jordan

• A whiteboard project (so, no code required).
  
• The outcomes: a list of definitions of “relevance” within the context of search and retrieval systems like OPACs, digital collections, shallow and deep web searching, etc.
• Task for the day: come up with a list of meaningful approaches to [re-]define relevance for library search and retrieval applications that “do better” than conventional relevance-ranking techniques.
• For example, for items in a digital collection, the amount of time a user spends reading one item (“page visit time”) may be a useful relevance indicator. 
• Another example: the number of comments users make on an item.

Our Interpretation

Spend a day discussing issues surrounding relevance and determining what'sactually relevant to the discussion

Our discussion framework

We imagined a "sound board" or something like your old stereo's equalizer with the ability to control elements relevant to relevancy ranking in the following 3 areas...

Elements at time of searching: [filtering & personalization options at point of search?]

Elements that are library supplied impacting rankings of resuts: [better use of bibliographic metadata by the relevancy algorithm & ability of library to customize?]

Elements that are user supplied impacting ranking of results: [using metadata like usage, tagging, reviews, recalls, reserve data, reading lists,...]

Needs identified:

There is a need to have a method to control local weightings for library systems

1. starting place is for matching items to searches

2. build internal text-based relevancy criteria

3. build standards and weighting for external relevancy inputs

Searching – bringing back a good result set

Some of our tasks are to deal with:

• what happens if the search term is wrong
  

• How to guide users to find better results

• fuzzy logic
  

• spelling options

• “did you mean” options

• Synonyms – are they equivalent or something different
  

• Merging external and internal data sources

Result set

ordered in “relevance”

Some internal controls/inputs

Some considerations in determining where an item sits in the result set:

• Where the search term was (title, subject, notes – what is more important)
• How often was the search term included (total word count, percentage ranking)
• Does nearness matter?
• Levels of authority – was this in a reserve collection? Did a staff person pull this information together?
• Length of time an object is viewed? Is this useful?

Do we need to pull the data out of the catalogue in order to make more out of what we have?

Some data sources used by these controls

should be able to use all the metadata associated with the item:

• MARC
• Usage – including circulations, holds, recalls, reshelving counts
• OPAC stats
  
• Table of contents
• Bibliographies
• Full text
• abstracts
  
• Syndetics
• etc
  

Some external controls/inputs

Giving back some control to the user/patron - (more inclusion of social software)

• tagging, comments, trackbacks, rating
• allow users to 'remove' irrellevant results from their searches and have that removal reflected in future result rankings
• allow users to drag and drop results up or down the results list. Store and use that data to present more relevant results in future searches.
• look at results from other sources like Amazon and Google and use their positions to affect the order of our results lists.

Disparate data makes this challenging

When the various types of data are mixed, relevancy may not be able to be applied across the various items.

relevancy and aggregation

 

Is the library community large enough? Or is there a need to aggregate this data? Are we large enough individually? If not, we need to find a common ground for relevancy and then create methods for sharing. Particularly important to link usage data amongst libraries.

Why should libraries aggregate? What does this sort of information provide for libraries?

Engaging the patrons!

Collection development

 

Where this is starting within libraries:

Where this exists outside of libraries:

 

Amazon.com – personalized services, results based on popularity, additional information includes who bought this bought these other titles, customers share reading lists and create a community of “readers” advisory

Google Books – using bibliographies of books to rank relevance of results in searching

"Mets"