I’d like to start this morning by sharing a story I heard a couple years ago on the NPR program “This American Life”
You will be hearing from Alex Blumberg, the producer of the program:
This story illustrates the problems that are inherent in making assumptions
Assumptions are the biggest challenge facing libraries today
Alex’s embarrassing experience was based on a set of faulty assumptions
I’d like to spend the next couple of minutes summarizing a keynote address I heard in 2008 at the North American Serials Interest Group (NASIG) Annual Conference
That keynote, given by Mike Kuniavsky, was, for me, the genesis of the project I will be discussing today
This summary will be about three-fourths summary and one-fourth my interpretation of his presentation
So, Mike Kuniavsky spoke at the NASIG 2008 annual conference about ubiquitous computing
However, he started by analogizing the world of electric motors to the world of computing
In 1910, “electric motors were expensive, so you bought one for the house and then you bought attachments for it. The motor was a general purpose tool that was adapted as needed.”
As motors got cheaper, you started having multiple motors in the home
Instead of an attachment for a fan, you had a fan
Instead of an attachment to wash clothes, you had a washing machine
Eventually, we ended up with hyper-specialized electric motors (for example, there are now, on average, 20-30 motors per car)
This led to many unintended consequences (vacuums → wall-to-wall carpeting)
We see the same thing happening with computers today as what happened with electric motors at the turn of the century
We have entered the 3rd stage of computing (mainframe → desktop → ubiquitous computing)
The phrase ubiquitous computing was phrased in 1988 by Mark Weiser
He describes it as follows:
The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.
Just like hyper-specialization of the electric motor led to some interesting changes, so will ubiquitous computing
Mike Kuniavsky continued by saying the following:
Ubiquitous computing gives us tools to track, trade and share objects much more efficiently than any previous technology.
What does this mean in real terms?
Here are just a few real world examples:
You are all familiar with Netflix
Ubiquitous computing has led to a tracking system that allows for cheap and easy movie rentals and, now, even movie streaming
This model is challenging the way we think about ownership of movies
Why would you need to own a movie when you can just stream a movie straight to your TV anytime you want to watch it (you don’t even need to get out of our chair to put in the DVD)
Here’s another example:
Another example is ZipCar – rent a car anytime you need one
This is a very different idea of ownership than what we are used to
Mike described it as living in a world where everything has dotted lines around it
Instead of owning a car, you own the right to use a car
The idea of subscribing to something instead of owning it is, of course, nothing new to librarians
We have dealt with serials and aggregators for many years now
However, with the advent of ubiquitous computing, the possibilities for what we can have access to without having ownership are almost endless
It was not financially feasible to do this even just a few years ago, however, today we can get access to entire libraries
In the past, the words access and ownership were synonymous in the library world – those two terms were inextricably intertwined
There was no way for librarians to provide access without owning the material
Interlibrary loan changed that to a certain extent
However, it has only been since the advent of ubiquitous computing and the network that access and ownership have been truly decoupled
Going back to Alex Blumberg and his assumptions about Nielsen families, the biggest assumption librarians are now questioning is that we must own content in order to provide research-quality resources
Just how far will this idea go?
How many resources are really so critical to a research library’s mission that we must own them?
And how many of our resources would it be okay for us to just have access to them?
In other words, the big question is ownership vs…
There is no one-size-fits-all answer
However, access, rather than ownership, may be the best answer in many cases in the future
Finding the optimal mix (the right answer) all depends on where we are willing to see dotted lines in our world
But most of all, it depends on how willing we are to go back to the assumptions we made about Nielsen families as a kid and really start to question them?
I’ve dubbed this phenomenon:
Kuniavsky’s law: Ubiquitous computing has permanently decoupled access and ownership?
The second principle I’d like to talk about today is access and discoverability
The music industry serves as a great example of this
We already discussed how electric motors gained popularity at the turn of the last century
One of the hyper-specialized uses of the electric motor was the turntable
The turntable allowed for playback of recorded music, speeches and other orally communicated content
However, if you wanted a specific song on the record album, you had to know where to place the needle attached to the end of the record player’s arm
In other words, the content was accessible, but discovery was sub-optimal
Other technologies came and went, but discovery was still not very good
Then, the compact disc came along
The discs were broken into tracks that allowed for quick access and quick discovery
Discovery has increased significantly with the advent of services such as iTunes where you can search through hundreds of thousands of albums and find the exact song or other content that you were looking for
iTunes and other similar services, such as Pandora (US only), last.fm or Jango, even provide recommendations based on your past music consumption which opens up whole new doors to discovering new-to-you music
Libraries have followed a similar path as the music industry
Before the advent of the network, and specifically the online library catalog, discovery was tedious
To find anything, you had to scour the library card catalog
At the risk of sounding like a very bad librarian, I will tell you that I was never very good at using the card catalog system
It was hard!
Fortunately, I came into librarianship several years after most libraries migrated to online catalogs
Then the network came along which opened up many opportunities for improving discovery of library collections
WorldCat was created and collocated many libraries’ collections and made discovering the nearest location with access a breeze
But WorldCat was the equivalent of the compact disc
It revolutionized discovery, but it was still largely an analog tool
Discovery was digital but access was still analog
Google Scholar is to libraries, what iTunes was to the music industry
It has streamlined both discovery and access to library collections
It integrates with WorldCat so it knows what your print collections are and, if you use your link resolver to give Google your information, it will also know what your electronic collections are
I call this Google Scholar’s Law
Kuniavsky’s law states that ubiquitous computing decoupled access and ownership
Google Scholar’s law states that Google Scholar, a direct outgrowth of ubiquitous computing, has decoupled access and discovery?
So if computers are doing all of this, where does this leave libraries?
My contention today, is that libraries are here to provide access to our patrons to the content that they need
If discovery is best left to Google Scholar and the network at large, libraries should concentrate on building their own collections, which is our strength
Leave the discovery to others that seem to have the advantage?
I want to talk today about some research that my colleagues and I have been working on
The question we wanted to answer was, is Google Scholar a good discovery tool?
Specifically, we wanted to know how the results returned in Google Scholar compared to results in library databases
In other words, which is the better discovery tool?
Library databases or Google Scholar
As you might imagine, there was a significant amount of skepticism in the library world when Google Scholar was introduced
Many did not want to acknowledge that Google Scholar might have an important role to play in every librarian’s toolbox
It might be helpful to give a little bit of history:
Before Google, there were a whole host of search engines available to the average internet user
You may recognize some or all of these
How many of you still use one or more of these search engines?
How many of you use Google? (general public search engine use - 65% only use Google)
Most of these search engines are now gone - they were eaten by the monster that Google became
Here is what Google looked like when it was first released for public consumption in 1998
There were over 25 million pages indexed! :)
The last time they announced how big the index is was in 2005 - it had well over 25 billion pages indexed at that point
So by the time that Google Scholar was introduced, at the end of 2004, Google had lots of search and, more importantly, discovery experience
Despite all this skepticism, many librarians began placing Google Scholar on the homepages of their sites
Many articles were published about Google Scholar
Some of the better and more comprehensive studies were conducted by Peter Jasco, from the University of Hawaii
He described Google Scholar’s search results with such glowing terms as “shallowness” and, my personal favorite, “artificial unintelligence”
Other studies tried to match citations “hit to hit”
These studies were conducted by Neuhaus, Brophy and Bawden
This means that they would gather a large data set from respected database vendors, such as ISI’s Web of Science, and try to find all of the citations in that database to what was available in Google Scholar
They found the obvious - Google Scholar’s coverage was spotty and had lots of holes in it
This is because Google Scholar indexes from reputable sources, such as publisher’s websites (when they’re given permission) but also gathers information from professor’s personal websites, other pages in the .edu domain and from institutional repositories
The nature of those sources is incomplete
The nature of those sources also allows Google to more readily identify and index the seminal research in most disciplines
Library databases typically have a much harder time identifying those types of papers
This also allows Google to identify the prominent authors in each discipline
After the initial onslaught of negative reviews, authors began looking at some of the value-added services that Google Scholar adds for researchers
These value-added features include tracking open access content and other content found in institutional repositories
This line of research was done by Kousha, Thelwall among others
More recently Peter Jasco has re-evaluated Google Scholar and found the results to be much better than they were intially but still found many problems with the way it counts citations and other relatively minor quibbles about the service
The research that we undertook was a very different approach from these other studies
The studies by Neuhaus and others looked at all the content that was going into Google Scholar and found it lacking
We wanted to look at the outcomes of Google Scholar searches
Specifically, we wanted to know if the Google Scholar results were more or less scholarly than the results found in library databases
We did this by directly comparing search results from Google Scholar to search results from library databases
7 Subject specialists from 3 disciplines:
3 from sciences
2 from humanities
2 from social sciences
Blind to purpose of study
Asked them to give us 3 things…
A question they would typically receive from a student (humanities)
A structured query they would use to search a database
The database they would use to search for that question
This is what things looked like after we got all the information back from the librarians
Then we took that information and used it in 2 ways.
The first was to actually run the search query in the suggested database.
We put the first 30 citations into a bibliographic citation manager and saved all of the actual full text
We chose 30 because usability studies (Jakob Neilsen) tell us that less than 1% of all users ever go beyond the 3rd page of results and very few people ever change the defaults (ie, once they run a search they stick with it, success or failure).
Most of our DBs present 10 results per page so 30 results should represent a large enough sample to represent the actual set of results the majority of our users is ever going to see after performing a search.
We ran the same query in Google Scholar and saved the results again in a bibliographic Manager.
We used Zotero to quickly export all of the results.
We also saved the full text of each citation for later use in our study.
So, the first searches we ran using the native DBs and GS was for the query given to us by the librarian
The second set of searches we ran was to see if the citations we found in the DB were available in GS and vice versa
Here is the same screenshot we saw just a minute ago.
We took the bibliographic information for each citation and searched for the citation within Google Scholar.
We then did the same thing in reverse.
We took the 30 results from GS and searched for each citation within the database
This allowed us to later calculate something we called “exclusivity”
We put the citations into 1 of 3 possible “exclusivity” categories
Shows proportion of citations within our study that overlap. As you can see, within our study we found that, on average, GS had a larger result set overall as well as more exclusively than the databases.
So now that we have the citations from the database and the citations from Google Scholar. We used the bibliographic manager to generate a list of references that we input into an Excel spreadsheet. Then, using a random number table, we completely randomized the order of the citations for each subject specialist.
Finally, to deliver the content to the librarians in a way in which it would be easiest for them to evaluate, we saved the full-text of each citation according to its randomly assigned citation number. Then we used Excel to create hyperlinks to the full-text of each citation and delivered this list along with the full-text on a CD to the subject librarians. We asked them to evaluate each citation using a rubric which we provided in hard copy form. As you can see, the subject librarians were only able to see the citation number and the bibliographic information. By clicking on the hyperlinked citation number, the full-text of that citation would appear and the subject librarians could easily rate the citation on the rubric.?
Have full text appear on this page after click to simulate linking from provided document.
This screen shows the rubric that we used. It is based on a rubric that has popularly been used to evaluate print resources (Alexander, 1999)
Alexander, J. E. (1999). Web wisdom: How to evaluate and create information quality on the Web.
We asked each subject librarian to assign a score of between 1 and 3 within 6 different categories to each of the citations (1 was below average, 2 was average and 3 was above average).
These six categories were:
Accuracy – which looks at
Authority – specifically the
Objectivity – looking for
Currency – is the information up to date?
How deep is the Coverage
And finally Relevancy – how well does the citation relate to the research question
This resulted in a total possible score of 18 for each citation - we called this a scholarliness score
We used this statistical model to evaluate the data. Essentially this formula says 2 important things about the way we used the data:
We controlled for the differences between the way librarians grade
We controlled for the differences in how exclusively the citation was available
This allowed us to pinpoint and measure any differences there may have been between disciplines in our data as well as any differences that can be attributed to the source of the citations
Citations found only in GS had, on average, a 17.6% higher scholarliness score than citations found only in the DB
Citations found in both GS and the DB were even higher than citations found only in GS
We found no statistically significant difference in the scholarliness scores between disciplines (ie, humanities citations in GS are just as scholarly as science citations found in GS)
This study can only be extrapolated statistically to the specific topics and subject specialists used in this study
A more robust statistical methodology would need to be employed to make these results generally applicable
We are encouraged by the results we received and feel that they would probably hold up but cannot say so until another study is done
If we had to do it over again, we would have increased the Likert scale on our rubric from 1-3 to 1-7 or 1-10
This would have allowed for a more nuanced statistical analysis and made it easier to spot significant differences, if any, between GS and databases
Our scholarliness calculation, ultimately, was based on the subjective opinions of librarians with subject expertise.
There are lots of ways to create a scholarliness score (citation counts, impact factors, etc). Which is best is still debatable
Our study compared GS to individual library databases. A more appropriate comparison may be GS to federated search tools.