Irrelevant Discovery Layers? An Evidence-Based Evaluation of Three Common Library Search Tools

Ruth Szpunar; Eric Bradley; Erin Gabrielson; Catherine Pellegrino

doi:10.5860/ital.v44i2.17266

Authors

Ruth Szpunar PALNI https://orcid.org/0000-0002-2922-4388
Eric Bradley Goshen College https://orcid.org/0000-0002-4353-995X
Erin Gabrielson Franklin College
Catherine Pellegrino Saint Mary's College

DOI:

https://doi.org/10.5860/ital.v44i2.17266

Abstract

Over the past fifteen years, most academic librarians have implemented one-stop search tools, commonly referred to as discovery layers, to accommodate contemporary user expectations. In more recent years these tools have come under criticism due to their limitations and shortcomings. We set out to evaluate if a discovery layer, when prompted with typical user keyword search strings, produced the most relevant search results when compared with two other widely accessible academic search tools. We compared search results from a discovery layer with a central index (WorldCat Discovery) with search results from a subscription interdisciplinary index and abstract database (Academic Search Complete) and a freely accessible academic web search engine (Google Scholar). We created a rubric detailed enough for multiple evaluators, who were the authors, to judge search results for currency, relevancy, proximity, and authority, as well as to assign appropriate penalties. Academic Search Complete search results received the highest overall scores, while WorldCat Discovery search results received the lowest overall scores. When considering individual pieces of the rubric, Academic Search Complete provided the most current and authoritative sources, while Google Scholar provided the most relevant sources. This article provides recommendations for libraries moving forward to consider the benefits and costs of discovery tools.

References

“About Google Scholar,” Google, accessed May 10, 2024, https://scholar.google.com/intl/en/scholar/about.html.

“Academic Search,” Library of Congress Catalog, accessed June 28, 2024, https://lccn.loc.gov/sn97001287.

“Google Scholar Help,” Google, accessed April 26, 2024, https://scholar.google.com/intl/en%20/scholar/inclusion.html.

“How Is Relevance Ranking Determined in EBSCO Discovery Service (EDS)?”, EBSCO Connect, January 13, 2020, https://connect.ebsco.com/s/article/How-is-relevance-ranking-determined-in-EBSCO-Discovery-Service-EDS?language=en_US.

“Perceptions of Libraries and Information Resources” (Dublin, Ohio: OCLC Online Computer Library Center, Inc., 2005), https://www.oclc.org/content/dam/oclc/reports/pdfs/Percept_all.pdf.

“Relevancy and Scoping,” OCLC Support, January 12, 2024, https://help.oclc.org/Librarian_Toolbox/OCLC_Service_Configuration/WorldCat_Discovery_and_WorldCat_Local/020Relevancy_and_Scoping.

“WorldCat Discovery Resources,” OCLC, 2024, https://www.oclc.org/en/worldcat-discovery/resources.html.

Alexa L. Pearce, “Discovery and the Disciplines: An Inquiry into the Role of Subject Databases through Citation Analysis,” College & Research Libraries 80, no. 2 (March 1, 2019): 195–214, https://doi.org/10.5860/crl.80.2.195.

Andrew D. Asher, Lynda M. Duke, and Suzanne Wilson, “Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources,” College & Research Libraries 74, no. 5 (2013), https://doi.org/10.5860/crl-374.

Arkadiusz Pulikowski and Anna Matysek, “Searching for LIS Scholarly Publications: A Comparison of Search Results from Google, Google Scholar, EDS, and LISA,” The Journal of Academic Librarianship 47, no. 5 (September 2021): 102417, https://doi.org/10.1016/j.acalib.2021.102417.

Blake L. Galbreath, Corey Michael Johnson, and Alexander N. Merrill, “A Framework for Measuring Relevancy in Discovery Environments,” Information Technology and Libraries 40, no. 2 (2021): 1–17, https://doi.org/10.6017/ital.v40i2.12835.

Boram Lee and EunKyung Chung, “An Analysis of Web-Scale Discovery Services from the Perspective of User’s Relevance Judgment,” The Journal of Academic Librarianship 42, no. 5 (September 1, 2016): 529–34, https://doi.org/10.1016/j.acalib.2016.06.016.

Burton Callicott and Debbie Vaughn, “Google Scholar vs. Library Scholar: Testing the Performance of Schoogle,” Internet Reference Services Quarterly 10, no. 3–4 (2005): 71–88, https://doi.org/10.1300/J136v10n03_08.

Casey D. Hoeve and Christina Geuther, “The Silent Battle on the Budget: The Effect of Centralized Indexing on Collection Analysis in Primo and EBSCO,” Collection Management 46, no. 1 (December 5, 2020): 3–13, https://doi.org/10.1080/01462679.2020.1790458.

Cristífol Rovira, Frederic Guerrero-Solé, and Lluís Codina, “Received Citations as a Main SEO Factor of Google Scholar Results Ranking,” Profesional de la información 27, no. 3 (June 18, 2018): 559–69, https://doi.org/10.3145/epi.2018.may.09.

Cristòfol Rovira, Lluís Codina, and Carlos Lopezosa, “Language Bias in the Google Scholar Ranking Algorithm,” Future Internet 13, no. 2 (February 2021): 31, https://doi.org/10.3390/fi13020031.

D. Yvonne Jones, “Biology Article Retrieval from Various Databases: Making Good Choices with Limited Resources,” Issues in Science and Technology Librarianship 44 (2005), http://www.istl.org/05-fall/refereed.html.

Daniel W. Eller, “Transparency and the Future of Semantic Searching in Academic Libraries,” ed. Bonnie Lawlor, Information Services & Use 42, no. 3–4 (December 16, 2022): 453–61, https://doi.org/10.3233/ISU-220175.

David Earl Noe, “Replicating Top Users’ Searches in Summon and Google Scholar,” in Planning and Implementing Resource Discovery Tools in Academic Libraries (ISR, 2012), 225–49, https://doi.org/10.4018/978-1-4666-1821-3.ch013.

Elizabeth Ketterman and Megan E. Inman, “Discovery Tool vs. PubMed: A Health Sciences Literature Comparison Analysis,” Journal of Electronic Resources in Medical Libraries 11, no. 3 (July 1, 2014): 115–23, https://doi.org/10.1080/15424065.2014.938999.

Elizabeth Namei and Christal A. Young, “Measuring Our Relevancy: Comparing Results in a Web-Scale Discovery Tool, Google & Google Scholar,” 2015, https://alair.ala.org/bitstream/handle/11213/17887/MeasuringOurRelevancy.pdf?sequence=1.

Emily Singley, “Discovery Systems – Testing Known Item Searching,” Usable Libraries (blog), March 18, 2014, https://emilysingley.net/discovery-systems-testing-known-item-searching.

Helen Georgas, “Google vs. the Library (Part III): Assessing the Quality of Sources Found by Undergraduates,” portal: Libraries and the Academy 15, no. 1 (2015): 133–61, https://doi.org/10.1353/pla.2015.0012.

Jan Brophy and David Bawden, “Is Google Enough? Comparison of an Internet Search Engine with Academic Library Resources,” Aslib Proceedings 57, no. 6 (January 1, 2005): 498–512, https://doi.org/10.1108/00012530510634235.

Jared L. Howland et al., “How Scholarly Is Google Scholar? A Comparison to Library Databases,” College & Research Libraries 70, no. 3 (2009), https://doi.org/10.5860/0700227.

Jay Holloway, “WorldCat Discovery Product Insights” (May 15, 2024).

Jayne Dickson, “WorldCat Discovery Services Beta Is Launched,” California Digital Library, April 30, 2014, https://cdlib.org/cdlinfo/2014/04/30/worldcat-discovery-services-beta-is-launched/.

Jenny S. Bossaller and Heather Moulaison Sandy, “Documenting the Conversation: A Systematic Review of Library Discovery Layers,” College & Research Libraries, July 14, 2017, https://doi.org/10.5860/crl.78.5.602.

Joran Beel and Bela Gipp, “Google Scholar’s Ranking Algorithm: The Impact of Citation Counts (An Empirical Study),” in 2009 Third International Conference on Research Challenges in Information Science (2009 Third International Conference on Research Challenges in Information Science (RCIS), Fez, Morocco: IEEE, 2009), 439–46, https://doi.org/10.1109/RCIS.2009.5089308.

Karen Ciccone and John Vickery, “Summon, EBSCO Discovery Service, and Google Scholar: A Comparison of Search Performance Using User Queries,” Evidence Based Library and Information Practice 10, no. 1 (March 15, 2015): 34–49, https://doi.org/10.18438/B86G6Q.

Kate B. Moore and Courtney Greene, “Choosing Discovery: A Literature Review on the Selection and Evaluation of Discovery Layers,” Journal of Web Librarianship 6, no. 3 (July 2012): 145–63, https://doi.org/10.1080/19322909.2012.689602.

Kelly Achenbach et al., “Defining Discovery: Is Google Scholar a Discovery Platform? An Essay on the Need for a New Approach to Scholarly Discovery,” Open Research Europe 2 (June 7, 2022): 28, https://doi.org/10.12688/openreseurope.14318.2.

Lucy Kiester and Clara Turp, “Artificial Intelligence behind the Scenes: PubMed’s Best Match Algorithm,” Journal of the Medical Library Association : JMLA 110, no. 1 (2022): 15–22, https://doi.org/10.5195/jmla.2022.1236.

Mary Shultz, “Comparing Test Searches in PubMed and Google Scholar,” JMLA: Journal of the Medical Library Association 95, no. 4 (October 2007): 442–45, https://doi.org/10.3163/1536-5050.95.4.442.

Michael Levine-Clark and Joseph Kraus, “Finding Chemistry Information Using Google Scholar: A Comparison with Chemical Abstracts Service,” Science & Technology Libraries 27, no. 4 (August 20, 2007): 3–17, https://doi.org/10.1300/J122v27n04_02.

N. S. Redkina, “Open Scientific Content Search Tools for Research and Education,” Professional Education in the Modern World 13, no. 4 (2023): 648–60, https://doi.org/10.20913/2618-7515-2023-4-6.

Nate Day, “Choosing Your Search Relevance Evaluation Metric,” OpenSource Connections (blog), February 28, 2020, https://opensourceconnections.com/blog/2020/02/28/choosing-your-search-relevance-metric/.

Paul T. von Hippel and Stuart Buck, “Improve Academic Search Engines to Reduce Scholars’ Biases,” Nature Human Behaviour 7, no. 2 (February 2023): 157–58, https://doi.org/10.1038/s41562-022-01518-0.

Rachel Cooke and Rebecca Donlan, “Thinking inside the Box: Comparing Federated Search Results from Google Scholar, Live Search Academic, and Central Search,” Journal of Library Administration 46, no. 3–4 (March 6, 2008): 31–42, https://doi.org/10.1300/J111v46n03_03.

Richard Wisneski, “I Can’t Get No Satis-Searching: Reassessing Discovery Layers in Academic Libraries,” Journal of Web Librarianship 18, no. 1 (2024): 1–14, https://doi.org/10.1080/19322909.2024.2326687.

Rosie Hanneke and Kelly K. O’Brien, “Comparison of Three Web-Scale Discovery Services for Health Sciences Research,” Journal of the Medical Library Association : JMLA 104, no. 2 (April 2016): 109–17, https://doi.org/10.3163/1536-5050.104.2.004.

S. Khalid et al., “On the Current State of Scholarly Retrieval Systems,” Engineering, Technology & Applied Science Research 9, no. 1 (February 16, 2019): 3863–70, https://doi.org/10.48084/etasr.2448.

Sarah P. C. Dahlen and Kathlene Hanson, “Preference vs. Authority: A Comparison of Student Searching in a Subject-Specific Indexing and Abstracting Database and a Customized Discovery Layer,” College & Research Libraries 78, no. 7 (October 30, 2017): 878–97, https://doi.org/10.5860/crl.78.7.878.

Seemon Thomas, Basic Statistics (Alpha Science International Ltd., 2014), https://public.ebookcentral.proquest.com/choice/publicfullrecord.aspx?p=5190782.

Simona Ştirbu et al., “The Utility of Google Scholar When Searching Geographical Literature: Comparison With Three Commercial Bibliographic Databases,” The Journal of Academic Librarianship 41, no. 3 (May 1, 2015): 322–29, https://doi.org/10.1016/j.acalib.2015.02.013.

Ştirbu et al., “The Utility of Google Scholar When Searching Geographical Literature.”

Susan Gardner and Susanna Eng, “Gaga over Google? Scholar in the Social Sciences,” Library Hi Tech News 22, no. 8 (January 1, 2005): 42–45, https://doi.org/10.1108/07419050510633952.

Suzanna Conrad and Christy Stevens, “‘Am I on the Library Website?’: A LibGuides Usability Study,” Information Technology and Libraries 38, no. 3 (September 15, 2019): 50, https://doi.org/10.6017/ital.v38i3.10977.

Tao Zhang, “User-Centered Evaluation of a Discovery Layer System with Google Scholar,” in Design, User Experience, and Usability. Web, Mobile, and Product Design, ed. Aaron Marcus, vol. 8015, Lecture Notes in Computer Science (Springer Berlin Heidelberg, 2013), 313–22, https://doi.org/10.1007/978-3-642-39253-5_34

Taryn Marks and Avery Le, “Increasing Article Findability Online: The Four Cs of Search Engine Optimization,” Law Library Journal 109, no. 1 (Winter 2017): 83–99, https://doi.org/10.2139/ssrn.3204550.

William H. Walters, “Comparative Recall and Precision of Simple and Expert Searches in Google Scholar and Eight Other Databases,” Portal: Libraries and the Academy 11, no. 4 (2011): 971–1006, https://doi.org/10.1353/pla.2011.0042.

William H. Walters, “Google Scholar Search Performance: Comparative Recall and Precision,” portal: Libraries and the Academy 9 (January 1, 2009): 5–24, https://doi.org/10.1353/pla.0.0034.

Xiaotian Chen, “EBSCO Collections’ Discoverability Rate by Ex Libris’ Central Discovery Index (CDI),” Collection Management 48, no. 2 (April 3, 2023): 84–96, https://doi.org/10.1080/01462679.2022.2081277.

Irrelevant Discovery Layers?

An Evidence-Based Evaluation of Three Common Library Search Tools

Authors

DOI:

Abstract

References

Downloads

Published

How to Cite

Issue

Section

License

Developed By

Information