Jared L. Howland

Federated Searching: Do Undergraduates Prefer It and Does It Add Value?

C. Jeffrey Belliston, Jared L. Howland and Brian C. Roberts

Presented: March 2007 at the 13th National ACRL Conference in Baltimore, MD.

Citation Information: Belliston, Jeffrey C., Howland, Jared L. and Roberts, Brian C. (2007). “Federated Searching: Do Undergraduates Prefer it and Does it Add Value?” Sailing into the future: Charting our destiny: Proceedings of the Thirteenth National Conference of the Association of College and Research Libraries, March 29–April 1, 2007, Baltimore, Maryland. Pages 103–113. (Full Text)

Abstract

Randomly selected undergraduates at Brigham Young University, Brigham Young University Idaho and Brigham Young University Hawaii, all private universities sponsored by The Church of Jesus Christ of Latter-day Saints, participated in a study of federated searching. This paper reports the study results including differences in time spent between searching databases in federated and non-federated fashion, satisfaction with citations gathered using each method, preference between methods, and quality of citations retrieved by each method judged by two different rubrics. Undergraduates rated their satisfaction with the citations gathered by federated searching 6.5% higher than their satisfaction using non-federated search methods. Additionally, 70% of undergraduates at the participating schools prefer federated searching and saved time using a federated search compared to a non-federated search. Which search method yields higher citation quality was ultimately indeterminable. The study sheds light on assumptions about federated searching and may interest librarians in different types of academic institutions given the diversity of the three institutions studied.

Introduction

In 2004, the Directors Council of the Consortium of Church Libraries and Archives (CCLA) licensed WebFeat’s federated search product for three years for all member institutions that wished to implement federated searching. About sixteen months prior to the expiration of the contract, the CCLA Directors Council requested data to assist in their decision concerning license renewal. We undertook this study to provide that data.

CCLA’s eight member libraries include four academic libraries serving undergraduates. These four libraries, at Brigham Young University (BYU), Brigham Young University Idaho (BYUI), Brigham Young University Hawaii (BYUH) and LDS Business College (LDSBC), have been the primary users of the licensed federated search technology. The study was to have gathered data at all four institutions but, due to a poor response rate, LDSBC was dropped from the study. Although all participating universities have similar names and serve undergraduates, the environments are quite diverse (Table 1).

Table 1: Institutional Information
Library Institution Abbreviation Degrees Granted Student Population (FTE)
Harold B. Lee Library Brigham Young University BYU Bachelors, Masters, Doctorate 31,225
Joseph F. Smith Library Brigham Young University-Hawaii BYUH Bachelors 2,467
David O. McKay Library Brigham Young University-Idaho BYUI Associates, Bachelors 12,209

The study was designed to answer the following questions for undergraduates:

  1. Does federated searching save time?
  2. Does federated searching satisfy students’ information needs?
  3. Do students prefer federated searching to the alternative of searching databases individually?
  4. Does federated searching yield quality results?

Because all of the CCLA institutions have implemented federated searching differently, we designed the study to be implementation-neutral thereby providing data on federated searching itself, not on the WebFeat software.1

Literature Review

End-user federated searching (sometimes known as broadcast searching, distributed searching, metasearching, or parallel searching) of multiple databases stored by different companies in multiple locations is a relatively recent development. The concept of a single search of multiple databases goes back to at least 1966 when the Dialog service made possible the simultaneous searching of multiple discrete, proprietary databases. However, in contrast to the databases searched by current federated search products, the Dialog databases were (1) stored by a single company in a single location and (2) usually searched for an end-user by a librarian due to both the fee structure and the proprietary command-driven nature of the search interface. Roger K. Summit’s 1971 article on Dialog’s user interface and Stanley Elman’s various articles on the cost-benefit of Dialog examined this major forerunner to federated search.2

The majority of articles about today’s federated search technology tend to fall into four categories. They (1) discuss the desirability and/or difficulty of creating a robust federated search tool,3 (2) report on one or more specific federated search implementations,4 (3) compare federated search products currently on the market to each other and/or to Google Scholar,5 or (4) look at how to implement a subject-specific federated searching tool.6 Because these articles are theoretical, report of experience, or compare feature sets, they contain little data based on objective research.

The literature includes many explicit, and reasonable, assumptions about federated searching. The Serials Review column, “The One-Box Challenge: Providing a Federated Search That Benefits the Research Process,” edited by Allan Scherlen with contributions from five academic librarians, provides a recent example of this. The editorial introduction to the column states, “Federated searching will certainly make some aspects of research easier, but will it make it better?”7 Contributor statements include the following. For Marian Hampton, “[t]he benefit of metasearching is obvious – one simple interface for several sources…”8 Penny Pugh quotes the “minimal instruction” on West Virginia University’s federated search: “‘E-ZSearch provides a quick and easy way to search multiple databases at once.’”9 Frank Cervone writes, “the point of federated searching is to make searching as simple as possible…”10 This study tests the assumptions that federated search is better than the alternative.

Methodology

Research participants and data gathering. Emails were sent to a random sample of currently enrolled undergraduate students at BYU, BYUI and BYUH inviting them to participate in a research project. Those who responded positively became participants. To ensure a consistent delivery of expectations for the study, participants received written, rather than oral, directions (Appendix A). Each student was randomly assigned to one of two biology-related topics for a hypothetical research assignment. The written directions indicated which topic and the first search method (federated or non-federated) they were to use to locate citations to journal articles that they felt appropriately addressed the topic. Then, using the same user interface and the same set of seven databases, every student compiled a set of citations. The students copied and pasted the citations into a scratch pad area available to the right of their Internet browser on the screen.

The time a participant began researching the first topic was noted. When the participant indicated he or she had completed the research for the assigned topic, the ending time was recorded, the collected citations were captured and the scratch pad was cleared. The process was then repeated for the other research topic searched using the other search method so each student created two citation sets for analysis (Appendix B). Finally, participants completed a questionnaire that asked about their satisfaction with the citations gathered using each method, which method they preferred and why (Appendix C). A total of ninety-five undergraduates from the three schools participated (Table 2).

Table 2: Summary of Participants (n = 95)
Question 1 First Question 2 First
Federated First 26 24
Non-Federated First 24 21

Analysis of citations. To gather different perspectives on quality, each citation set was judged using two rubrics: one created by librarians consisting of objective measures and a more subjective one approved by a faculty member in BYU’s Physiology and Developmental Biology Department (Appendices D and E). The objective criteria in the librarian-created rubric included the impact factor for cited journals as reported by ISI’s Journal Citation Reports, the proportion of citations from peer-reviewed journals (as determined by consulting Ulrichsweb) to total citations, and the timeliness of the articles. While timeliness is not critical for all subject areas, it was deemed to be important to writing an adequate research paper on the two biology-related topics used in the study. Each of the three criteria was weighted equally by normalizing the data for each criterion to a maximum value of ten. Each citation set received a final score by summing the points assigned to each criterion to reach a composite quality score.

The subjective faculty-approved rubric was designed to more closely follow the practices used by faculty members in a real educational setting. The three criteria used in this rubric included relevance to the topic, quality of the individual citations, and quantity of citations. Using the rubric, one undergraduate, a senior majoring in Biology, assigned points to each of the 190 citation sets for each of the three criterion and summed them to create a composite quality score.

Statistical analysis. After gathering the data, we analyzed it using analysis of variance (ANOVA) and multivariate analysis of variance tests (MANOVA). To be consistent, the factors under study included school (BYU, BYUH, BYUI), method (federated versus non-federated), order (the order in which a given student was asked to use federated and non-federated searching), and question (to ascertain if the topic itself – though both were biological in nature – made a difference in the responses). After controlling for those factors, the analyzed data included time to complete the hypothetical research assignment, participant satisfaction rating of citations found, preference for search method, and the two composite quality scores.

Results

Time savings. Statistically significant differences exist between BYU and the other two schools in the time required to complete the hypothetical assignments using the two search methods. While all schools recorded time savings in research by using federated searching, the results were widely dispersed. BYUI students saved an average of only 11.4 seconds and BYUH students saved an average of 26.4 seconds. BYU students, on the other hand, saved an average of 4 minutes, 11.4 seconds. Only the BYU results showed a statistically significant difference between time required for research and the search method used (Table 3).

Table 3: Comparison of Results Between Schools
School Number of Participants % Preferred Federated Average Time to Complete Research
(in minutes) 5
Satisfaction of Results – Average Rating
(Scale of 1–7; 7 being highest) 5
Librarian-created Rubric – Average Quality Scores
(Scale of 0–30; 30 being highest) 5
Faculty-created Rubric – Average Quality Scores
(Scale of 0–9; 9 being highest) 5
Federated Non-federated Federated Non-federated Federated Non-federated Federated Non-federated
BYUH 27 81% 21.17 22.14 5.57 4.131,2 17.71 19.354 5.591 5.743
BYUI 21 52% 23.10 23.54 5.411 5.48 17.67 18.08 6.38 5.79
BYU 47 72% 16.761 21.142 5.77 4.782 18.10 19.204 6.15 6.31
ALL 95 70% 20.34 22.72 5.59 4.802 17.83 18.882 6.04 5.59

1 Statistically significant difference between schools (α = .05)
2 Statistically significant difference between methods (α = .05)
3 Marginally significant difference between schools (α = .10)
4 Marginally significant difference between methods (α = .10)
5 These are adjusted means not pure means. A least squares mean was utilized to create more robust results due to differing sample sizes between the schools.

Satisfaction level of meeting information needs. Participants rated their satisfaction levels with the citations they found using the two search methods. Only BYU and BYUH showed a statistically significant difference in the satisfaction with citations found using the different search methods. Even including BYUI, where no statistically significant difference existed, participants were, on average, 16.5% more satisfied with the results found through federated searching.

Preferences. All three schools showed a preference for federated searching over non-federated searching though BYUI showed only a marginal preference (52%). Overall, 70% of study participants preferred federated searching to non-federated searching. There was a statistically significant (α=.05), but insignificant in practice, negative correlation (−0.18) between time to complete research and preference for search method. Although this is the expected correlation, it is interesting that the correlation was not stronger. One would expect that the less time it takes a student to find citations, the more likely the student would be to prefer the method which took less time, but the correlation is actually very small.

Reasons given by study participants who preferred federated search routinely included that it is faster, easier, simpler, and more efficient.11 One participant’s reason for preferring federated search begins with “Save time.” For this participant, it must have only seemed faster because the time spent using each search method was actually the same.

Quality of citations. Analysis of citation set quality using the librarian-created rubric revealed that, on average, citation sets gathered by using federated search scored a statistically significant 5.6% lower than those gathered by searching databases individually. Analysis using the faculty-approved rubric revealed no significant difference, statistically or in practice, in the quality of citation sets generated by the two methods.

Discussion of Results/Conclusions

Overall, undergraduates appear (1) to strongly prefer federated searching, (2) to be more satisfied with the results found via federated searching, and (3) to save time by using federated searching. In the final analysis, the quality of the citations found using different search methods is ambiguous. The librarian-created rubric showed that searching databases individually yields higher quality citations than does federated searching. However, that finding depends entirely on the definition of quality used in the rubric. Although the criteria themselves were entirely objective, the selection of the criteria was not. In the end, quality is in the eye of the beholder.12 Because real-world educators are more likely to make a subjective judgment of quality – like the one used in the faculty-created rubric – than they are to actually check to see if the journals cited by students have high impact factors, it seems reasonable to give greater credence to the finding that both search methods produce citation sets of similar quality.

Future Studies

The statistical models employed in the analysis of data reported here cannot be extrapolated to the undergraduate population as a whole. They can simply be extrapolated to the participating schools. However, in the future, we plan to apply a statistical model to the data in order to extrapolate the results to all institutions with undergraduates and we speculate that our results will hold.

This study controlled for, but did not address, the effect of implementation of a federated search engine on time savings, satisfaction, preferences, or citation quality. It is plausible that specific implementations could affect the results and either help or hinder a student’s experience. A study examining the effect of various possible implementations of federated searching is needed to determine an optimal implementation.

Finally, this study addressed undergraduate students only. More research is needed to determine the value of federated searching to graduate students and faculty. It is also probable that the results would vary depending on the discipline chosen for the hypothetical research topics as some disciplines may lend themselves more readily to federated search capabilities than other disciplines.

Appendix A: Participant Directions

Directions – Forms 1-F and 2-F

The criminology of genocide: The death and rape of Darfur
Hagan, J.; Rymond-Richmond, W.
Criminology, vol. 43, no. 3, pp. 525–561, 2005
This study examines Sudanese government involvement in the racially motivated murders of nearly 400,000 Africans from the Darfur region of Sudan. Data were obtained from a victimization survey of Darfurian survivors living in refugee camps in Chad…

Appendix A: Participant Directions

Appendix B: Hypothetical Assignments

1 – F INTERNAL USE ONLY Start Time 1: _________ Start Time 2: _________ Net ID: _____________ End Time 1 : _________ End Time 2 : _________

Hypothetical Research Assignment #1</h3>

You’ve been given an assignment to write a 10 page research paper on the topic outlined below:

Ignoring any ethical issues involved, what is the current status of stem cell research for the treatment of diabetes?

Using the resources available to you, find enough citations to complete this assignment (copy citations to the “Scratch Pad” on the right-hand side of your screen).

stopAfter you have completed this hypothetical assignment, stop your work and notify the administrator.

Hypothetical Research Assignment #2

You’ve been given an assignment to write a 10 page research paper on the topic outlined below:

According to recent research, what are the health risks associated with being overweight?

Using the resources available to you, find enough citations to complete this assignment (copy citations to the “Scratch Pad” on the right-hand side of your screen).

stopAfter you have completed this hypothetical assignment, stop your work and notify the administrator.

Appendix C: Participant Questionnaire

  1. How satisfied were you with the citations you were able to discover using the first research method (Hypothetical Assignment #1)? (Circle One: 1=Unsatisfied to 7=Very satisfied)
    1 2 3 4 5 6 7
  2. How satisfied were you with the citations you were able to discover using the second research method (Hypothetical Assignment #2)? (Circle One: 1=Unsatisfied to 7=Very satisfied)
    1 2 3 4 5 6 7
  3. Which method did you prefer? (First)__ (Second)__
    Why?


  4. What other comments do you have about your searching experiences?

Appendix D: Librarian-Created Quality Rubric

Average Impact Factor Proportion of Peer Reviewed Average Timeliness TOTAL
Student #1 Federated
Student #1 Non-federated
Student #2 Federated
Etc.
  1. Average Impact Factor: The impact factor of the journal from each citation was gathered from the Institute for Scientific Information’s Journal Citation Reports database.
    The impact factors for the set of citations the student submitted were averaged. Any citation without an impact factor was assigned a value of zero and included in the average.
    The data was then normalized to a maximum value of 10.
  2. Proportion of Peer Reviewed: Whether the journal from each citation is peer reviewed was determined by checking Ulrich’s Periodicals Directory.
    The proportion of peer-reviewed articles cited by the student differs qualitatively from the impact factor because not all journals with impact factors are peer reviewed.
    The data was then normalized to a maximum value of 10.
  3. Average Timeliness: The average timeliness of the articles in the citations submitted by each student was recorded.
  1. 0–1 years old = 10 points
  2. 2 years old = 9 points
  3. 3 years old = 8 points
  4. 4 years old = 7 points
  5. 5 years old = 6 points
  6. 6 years old = 5 points
  7. 7 years old = 4 points
  8. 8 years old = 3 points
  9. 9 years old = 2 points
  10. ≥ 10 years old = 1 point

Appendix E: Faculty-Approved Quality Rubric

SCORE
Relevance All citations are related to the topic 3
Over half of the citations are related to the topic 2
1 or more citations are related to the topic 1
No citations are related to the topic 0
Quality* All citations are of good quality 3
Over half of the citations are of good quality 2
1 or more citations are of good quality 1
No citations are of good quality 0
Quantity There are enough citations to write a 10 page research paper 3
There are enough citations to write a 5–9 page research paper 2
There are enough citations to write a 1–4 page research paper 1
There are not enough citations to write a research paper 0
TOTAL:

</sup> *Good Quality: Citations reporting primary research results would be considered of higher quality than review articles or other types of articles. Citations from “scholarly” or peer-reviewed sources would be considered of higher quality than citations from “popular” or non-peer reviewed sources.

Notes

  1. We gratefully acknowledge the assistance of WebFeat in setting up the implementation-neutral interface used in the data gathering.
  2. Roger K. Summit, “Dialog and the User: An Evaluation of the User Interface with a Major Online Retrieval System” in Interactive Bibliographic Search: The User/Computer Interface, ed. Donald E. Walker, 83–94 (Montvale, New Jersey: AFIPS Press, 1971); Stanley A. Elman, “Cost-Benefit Experience with Dialog Full-Text Retrieval” in Proceedings of the American Society for Information Science, Volume 10. 36th Annual Meeting, Los Angeles, California, October 21–25, 1973, eds. Helen J. Waldron and F. Raymond Long, 54–55 (Westport, Connecticut: Greenwood Press, 1973); Stanley A. Elman, “Cost Comparison of Manual and On-Line Computerized Literature Searching,” Special Libraries 66, no. 1 (1975): 12–18.
  3. For example: Donna Fryer, “Federated Search Engines,” Online 28, no. 2 (2004): 16–19; Roy Tennant, “Cross-Database Search: One-Stop Shopping,” Library Journal 126, no. 17 (2001): 29–30; Rachel L. Wadham, “Federated Searching,” Library Mosaics 15, no. 1 (2004): 20.
  4. For example: Frank Cervone, “What We’ve Learned from Doing Usability Testing on OpenURL Resolvers and Federated Search Engines,” Computers in Libraries 25, no. 9 (2005): 10–14; Doris Small Helfer and Jina Choi Wakimoto, “Metasearching: The Good, the Bad, and the Ugly of Making it Work in Your Library,” Searcher 13, no. 2 (2005): 40–41; Anne L. Highsmith and Bennett Claire Ponsford, “Notes on Metalib Implementation at Texas A&M University,” Serials Review 32, no. 3 (2006): 190–194.
  5. For example: Xiaotian Chen, “MetaLib, WebFeat, and Google: The Strengths and Weaknesses of Federated Search Engines Compared with Google,” Online Information Review 30, no. 4 (2006): 413–427.
  6. For example: Debbie Campbell, “Federating Access to Digital Objects: PictureAustralia,” Program: Electronic Library and Information Systems 36, no. 3 (2002): 182–187; Geoff Daily, “A Case of Clustered Clarity,” EContent 28, no. 10 (2005): 44–45.
  7. John Boyd and others, “The One-Box Challenge: Providing a Federated Search That Benefits the Research Process,” Serials Review 32, no. 4 (2006): 247 (emphasis ours).
  8. Ibid., 249 (emphasis ours).
  9. Ibid., 251 (emphasis ours).
  10. Ibid., 253 (emphasis ours).
  11. Ibid., 252. Our participants’ terms closely parallel those of West Virginia University students as reported by Penny Pugh in her contribution to the cited column.
  12. A report on the Quality Metrics project at Emory University states, “There was a categorical rejection of the value – and, of the very possibility – of substantive quality indicators presented in the ratings system, in particular as these applied to books and journals. One philosophical objection was to the notion of the quantification of quality in such a reductive manner.” The “quantification” referred to the creation of a rating of search results “hypothetically conceptualized as computed through numerically weighing various factors such as academic peer comments, non-academic comments, number of times cited, and the like.” Rohit Chopra and Aaron Krowne, “Disciplining Search/Searching Disciplines: Perspectives from Academic Communities on Metasearch Quality Indicators,” First Monday 11, no. 8 (2006): under “A. Thematic explication of key findings”, “4. Quality as User Empowerment to Make Judgments about Quality.”