A Framework for Measuring Relevancy in Discovery Environments

Increasing Scalability and Reproducibility





Discovery Systems, Resource Discovery & Access


Institutional discovery environments now serve as central resource databases for researchers in the academic environment. Over the last several decades, there have been numerous discovery layer research inquiries centering primarily on user satisfaction measures of discovery system effectiveness. This study focuses on the creation of a largely automated method for evaluating discovery layer quality, utilizing the bibliographic sources from student research projects. Building on past research, the current study replaces a semiautomated Excel Fuzzy Lookup Add-In process witha fully scripted R-based approach, which employs the stringdist R package and applies the Jaro-Winkler distance metric as the matching evaluator. The researchers consider the error rate incurred by relying solely on an automated matching metric. They also use Open Refine for normalization processes and package the tools together on an OSF site for other institutions to use. Since the R-based approach does not require special processing or time and can be reproduced with minimal effort, it will allow future studies and users of our method to capture larger sample sizes, boosting validity. While the assessment process has been streamlined and shows promise, there remain issues in establishing solid connections between research paper bibliographies and discovery layer use. Subsequent research will focus on creating alternatives to paper titles as search proxies that better resemble genuine information-seeking behavior and comparing undergraduate and graduate student interactions within discovery environments.

Author Biographies

Blake Galbreath, Washington State University

Blake L. Galbreath is Core Services Librarian, Washington State University.

Alex Merrill, Washington State University

Alex Merrill is Head of Library Systems and Technical Operations, Washington State University

Corey M. Johnson, Washington State University

Corey M. Johnson is Instruction & Assessment Librarian, Washington State University.


Alex Merrill and Blake Galbreath, “A Framework for Measuring Relevancy in Discovery Environments,” Open Science Framework (OSF), 2023, https://osf.io/wafbx?view_only=edf1715850e7474b90e6c521f7d82349.

Alexandra Hamlett and Helen Georgas, “In the Wake of Discovery: Student Perceptions, Integration, and Instructional Design,” Journal of Web Librarianship 13, no. 3 (2019): 237, https://doi.org/10.1080/19322909.2019.1598919.

Anita K. Foster and Jean B. MacDonald, “A Tale of Two Discoveries: Comparing the Usability of Summon and EBSCO Discovery Service,” Journal of Web Librarianship 7, no. 1 (2013): 14, http://doi.org/10.1080/19322909.2013.757936.

Athena Hoeppner, “The Ins and Outs of Evaluating Web-Scale Discovery Services,” Computers in Libraries 32, no. 3 (2012), https://www.infotoday.com/cilmag/apr12/Hoeppner-Web-Scale-Discovery-Services.shtml.

Barbara Valentine and Beth West, “Improving Primo Usability and Teachability with Help from the Users,” Journal of Web Librarianship 10, no. 3 (2016): 176–96, https://doi.org/10.1080/19322909.2016.1190678.

Blake L. Galbreath, Alex Merrill, and Corey M. Johnson, “A Framework for Measuring Relevancy in Discovery Environment,” Information Technology and Libraries 40, no. 2 (2021): 11–12, https://doi.org/10.6017/ital.v40i2.12835.

Christine Rigda, Margaret Hoogland, and Jessica Morales, “‘But I Just Want a Book!’ Is Your Discovery Layer Meeting Your Users’ Needs?” Journal of Web Librarianship 12, no. 4 (2018): 253, 254, 257, https://doi.org/10.1080/19322909.2018.1518183.

“Configuring Newspaper Search for Primo VE,” ExLibris Knowledge Center (Part of Clarivate), 2023, https://knowledge.exlibrisgroup.com/Primo/Product_Documentation/020Primo_VE/Primo_VE_(English)/130Configuring_Advanced_Search_Interfaces_for_Primo_VE/Configuring_Newspaper_Search_for_Primo_VE.

Corey M. Johnson and Jennifer Saulnier Lange, “Source Seeking Database Usage Patterns for Roots of Contemporary Issues Students” (poster presentation at Washington State University Showcase), March 30, 2023, https://doi.org/10.7273/000004754.

Courtney Lundrigan, Kevin Manuel, and May Yan, “‘Pretty Rad’: Explorations in User Satisfaction with a Discovery Layer at Ryerson University,” College & Research Libraries 76, no. 1 (2015): 47, https://doi.org/10.5860/crl.76.1.43.

Diane Cmor and Xin Li, “Beyond Boolean, Towards Thinking: Discovery Systems and Information Literacy,” 2012 IATUL Proceedings, paper 7, https://docs.lib.purdue.edu/iatul/2012/papers/7/.

Elena Azadbakht, John Blair, and Lisa Jones, “Everyone’s Invited: A Website Usability Study Involving Multiple Library Stakeholders,” Information Technology and Libraries 36, no. 4 (2017): 43, https://doi.org/10.6017/ital.v36i4.9959.

Ellen Safley and Debbie Montgomery, “Oasis or Quicksand: Implementing a Catalog Discovery Layer to Maximize Access to Electronic Resources,” The Serials Librarian 60, no. 1–4 (2011): 164–68, https://doi.org/10.1080/0361526X.2011.556028.

Garrett Grolemund, “Introduction to R Markdown,” R Markdown from Rstudio, July 16, 2014, https://rmarkdown.rstudio.com/articles_intro.html.

Greta Kliewer, Amalia Monroe-Gulick, Stephanie Gamble, and Erik Radio. “Using Primo for Undergraduate Research: A Usability Study,” Library Hi Tech 34, no. 4 (2016): 572, https://doi.org/10.1108/LHT-05-2016-0052.

Hans Rutger Bosker, “Using Fuzzy String Matching for Automated Assessment of Listener Transcripts in Speech Intelligibility Studies,” Behavior Research Methods 53 (2021): 1945–53, https://doi.org/10.3758/s13428-021-01542-4.

J. K. Lippincott, “Net Generation Students & Libraries,” EDUCAUSE Review 40, no. 2 (2005): 57, http://er.educause.edu/-/media/files/article-downloads/erm0523.pdf.

Jenny S. Bossaller and Heather Moulaison Sandy, “Documenting the Conversation: A Systematic Review of Library Discovery Layers,” College & Research Libraries 78, no. 5 (2017): 615–6, https://doi.org/10.5860/crl.78.5.602.

Jessica Mussell and Rosie Croft, “Discovery Layers and the Distance Student: Online Search Habits of Students,” Journal of Library & Information Services in Distance Learning 7, no. 1–2 (2013): 26, https://doi.org/10.1080/1533290X.2012.705561.

Jodi Pierre, “Discovery Services: Continuous Improvement with Ongoing Usability Testing,” Information Today (March 2023): 4–8.

Joy Marie Perrin et. al., “Usability Testing for Greater Impact: A Primo Case Study,” Information Technology and Libraries 33, no. 4 (2014): 59, https://doi.org/10.6017/ital.v33i4.5174.

Kelsey Brett, Ashley Lierman, and Cherie Turner,” Lessons Learned: A Primo Usability Study,” Information Technology and Libraries 35, no. 1 (2016): 19, https://doi.org/10.6017/ital.v35i1.8965.

Kerry Walton, Gary M. Childs, and Laurie Palumbo, “Testing Two Discovery Systems: A Usability Study Comparing Student Perceptions of EDS and Primo,” Journal of Web Librarianship 16, no. 4 (2022): 200–221. https://doi.org/10.1080/19322909.2022.2125478.

Kim Durante and Zheng Wang, “Creating an Actionable Assessment Framework for Discovery Services in Academic Libraries,” College & Research Libraries 19, no. 2–4 (2012): 217, https://doi.org/10.1080/10691316.2012.693358.

Lorcan Dempsey, “Discovery Layers—Top Tech Trends 2,” LorcanDempsey.net, 2012, http://orweblog.oclc.org/archives/002116.html.

Lorcan Dempsey, “Thirteen Ways of Looking at Libraries, Discovery, and the Catalog: Scale, Workflow, Attention,” EDUCAUSE Review (December 10, 2012), https://er.educause.edu/articles/2012/12/thirteen-ways-of-looking-at-libraries-discovery-and-the-catalog-scale-workflow-attention.

Mark P. J. Van Der Loo, “The Stringdist Package for Approximate String Matching,” The R Journal 6, no. 1 (2014): 111–22, https://doi.org/10.32614/rj-2014-011.

Marshall Breeding, “Major Discovery Product Profiles,” Library Technology Reports 50, no. 1 (2014): 33–52, https://web.p.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=0&sid=13d50467-13bb-465a-ab85-f9a0b41249f5%40redis.

Marshall Breeding, “The Ongoing Challenges of Academic Library Discovery Services,” Computers in Libraries 40, no. 1 (2020): 11, https://www.infotoday.com/cilmag/jan20/index.shtml.

Merrill and Galbreath, “A Framework for Measuring Relevancy in Discovery Environment,” Open Science Framework.

Nimisha Singla and Deepak Garg, “String Matching Algorithms and Their Applicability in Various Applications,” International Journal of Soft Computing and Engineering 1, no. 6 (2012): 218–22, https://www.ijsce.org/wp-content/uploads/papers/v1i6/F0304111611.pdf.

“Primo VE 2023 Release Notes,” ExLibris Knowledge Center (Part of Clarivate), 2023, https://knowledge.exlibrisgroup.com/Primo/Release_Notes/002Primo_VE/2023/010Primo_VE_2023_Release_Notes.

Rachel K. Fischer, Aubrey Iglesias, Alice L. Daugherty, and Zhehan Jiang, “A Transaction Log Analysis of EBSCO Discovery Service Using Google Analytics: The Methodology,” Library Hi Tech 39, no. 1 (2021): 249–62, https://doi.org/10.1108/LHT-09-2019-0199.

“Ranking: Deliver the Most Relevant Search Results,” ExLibris Knowledge Center (Part of Clarivate), 2024, https://exlibrisgroup.com/products/primo-discovery-service/relevance-ranking/.

“Resource Types in CDI,” ExLibris Knowledge Center (Part of Clarivate), 2024, https://knowledge.exlibrisgroup.com/Primo/Content_Corner/Central_Discovery_Index/Documentation_and_Training/Documentation_and_Training_(English)/CDI_-_The_Central_Discovery_Index/070Resource_Types_in_CDI.

Sarah P. C. Dahlen and Kathlene Hanson, “Preference vs. Authority: A Comparison of Student Searching in a Subject-Specific Indexing and Abstracting Database and a Customized Discovery Layer,” College & Research Libraries 78, no. 7 (2017): 878–79, https://doi.org/10.5860/crl.78.7.878.

Scott Hanrath and Miloche Kottman, “Use and Usability of a Discovery Tool in an Academic Library,” Journal of Web Librarianship 9, no. 1 (2015): 17–18, https://doi.org/10.1080/19322909.2014.983259.

Scott Uhl, “Applying User-Centered Design to Discovery Layer Evaluation in the Law Library,” Legal Reference Services Quarterly 38, no. 1–2 (2019): 32, https://doi.org/10.1080/0270319X.2019.1614373.

Sean P. Kennedy, “Uncovering Discovery Layer Services,” Public Services Quarterly 10 (2014): 55, https://doi.org/10.1080/15228959.2014.875788.

“Search It,” Washington State University Libraries, 2020, https://searchit.libraries.wsu.edu/.

“The Information Search Process,” University of Illinois Libraries, 2023, https://www.library.illinois.edu/staff/wp-content/uploads/sites/24/2017/07/Information-Search-Process.pdf.

“The UCORE Curriculum,” Washington State University Common Requirements, 2018, https://ucore.wsu.edu/faculty/curriculum/.

“Washington State University Learning Goals,” Washington State University Common Requirements, 2018, https://ucore.wsu.edu/about/learning-goals.

W. Jacobs, Mike Demars, and J. M. Kimmitt, “A Multi-Campus Usability Testing Study of the New Primo Interface,” College & Undergraduate Libraries 27, no. 1 (2020): 1–16, https://doi.org/10.1080/10671316.2019.1695161.

“Welcome to the Roots of Contemporary Issues,” Washington State University Department of History, 2017, https://ucore.wsu.edu/faculty/curriculum/root/.




How to Cite

Galbreath, B., Merrill, A., & Johnson, C. M. (2024). A Framework for Measuring Relevancy in Discovery Environments: Increasing Scalability and Reproducibility. Information Technology and Libraries, 43(2). https://doi.org/10.5860/ital.v43i2.16915