AI-Infused Discovery Environments

Information Retrieval Boon or Overpromised Hype?

Abstract

Although still in its infancy, artificial intelligence (AI) is rapidly making inroads into most facets of the library and education spheres. This paper outlines steps taken to examine Primo Research Assistant, an AI-infused discovery environment, for potential deployment at a large US public research university. The researchers aimed to evaluate the quality and relevance of the AI results in comparison to sources retrieved from the conventional search functionality, as well as the AI system’s multi-paragraph overview reply to the search query. As a starting point, the authors collected 103 search strings from a Primo Zero Result Searches report to approximate a corpus of natural language search queries. For the same research area, it was discovered that there was only limited overlap between the titles returned by the AI tool versus the current discovery layer. The researchers did not find appreciable differences in the numbers of topic-relevant sources between the AI and non-AI search products (Yes = 46.3% vs. Yes = 45.6%, respectively). The overview summary is largely helpful in terms of learning more details about the recommended sources, but it also sometimes misrepresents connections between the sources and the research topic. Given the overall conclusion that the AI system did not constitute a clear advancement or decline in effective information retrieval, the authors will turn to usability testing to aid them in further implementation decisions.