An Evaluation and Comparison of Academic Libraries

Weinstein

An Evaluation and Comparison of Academic Libraries

With Varying User Populations and Goals

Kaitlin Weinstein

17:610:553

Term Project: Final Report

Abstract

Using evaluation criteria proposed by previous literature, this article attempts to identify key features, assess, analyze, and compare three academic digital libraries of varying user population sizes and user goals. The three digital libraries evaluated: Queensborough Community College Digital Library, a community college DL within the larger, incredibly diverse CUNY system; Rutgers University Digital Library, a DL of a major research university; and The College of New Jersey Digital Library, a DL of a smaller public college, focused on the liberal arts, that serves a smaller user population. This analysis ultimately prompts criticism, as well as recommendations, for these libraries, as well as for other current and future academic libraries.

Introduction

According to a study done by Xie (2007), “The concept of a digital library means different things to different people. Even the key players in the development and use of digital libraries have different understanding of digital libraries” (pg. 1346). In creating a digital library, it would seem that the most important “key player” is the end user—for whom the digital library was created. It is for the sake of the user that we begin to evaluate, critique, and ultimately provide recommendations to each institution.

Academic digital libraries can be rather confusing and difficult for students (and even some faculty and staff) to navigate. As a former instructional assistant for the Rutgers University Libraries, I have seen hundreds of students struggle to understand how to find appropriate articles, search databases, and traverse the digital library’s large and rather grandiose collection of resources. Though students may find themselves knowledgeable in the principles of Google searching, digital libraries can often be a very different and less familiar experience.

Furthermore, in “Evaluating Digital Libraries: A User-Friendly Guide,” Reeves et al. (2005) write that many users face certain problems, such as an “unfamiliarity with the interface and functionality of the library being used” and a “lack of information-age skills such as effective search strategies.” Though this was written over a decade ago, these issues are still true for many users—even those who have experience with technology.

This involvement and research has led me to question the usability of all academic digital libraries: how hard is it for users, who may have no prior experience related to digital libraries, to navigate an academic digital library? Are some academic libraries easier to navigate than others? And does the intention of the institution matter to the usability and usefulness of the digital library?

In evaluating three academic digital libraries, which vary in their user population and institutional goals, the notion of usability in postsecondary digital libraries. Rutgers University Digital Librarywas chosen because of its status as a research institution and large user population. The College of New Jersey Digital Library was chosen because it is a smaller college with far fewer students and a heavier focus on liberal arts, rather than research. Lastly, Queensborough Community College Digital Library was chosen because it is a larger scale community college with a user population size somewhere between the digital libraries of Rutgers University and The College of New Jersey. The school is also a member of the City University of New York system of schools, which means that its resources directly reflect that of a larger institutional system.

Using these three academic digital libraries as a starting point, we can begin to evaluate user experience, and make comparisons between the digital libraries of institutions of varying user population sizes and differing institutional aims. In this evaluation, I will make observations and effectively evaluate using those observations and comparisons.

Literature Review

Relevant to this particular evaluation and comparison is the study done by Hong Iris Xie (2006) which sought to understand what users found important to evaluation, in order to better determine what digital library evaluation criteria should be considered. Xie (2006) says that after the users were “informed of the common elements of digital library definitions identified by the Association of Research Libraries (1995),” they were ultimately able to “identify the essential criteria that were important for the development and use of digital libraries” (pg. 437-438). The criteria she determined users want are: interface usability, collection quality, service quality, system performance efficiency, and user feedback solicitation (Xie, 2006, pg. 449).

Xie (2007) also performed a study using a questionnaire in order to how users evaluate digital libraries. She writes, “The commonly agreed DL evaluation criteria and variables are related to the usability of interface, the value of collection, and the system performance. These criteria and variables are the results of research in evaluating human-computer interaction, information retrieval system performance, and traditional libraries” (Xie, 2007, pg. 1351). She notes, however, that most studies focus on a certain aspect of digital libraries, such as usability, and thus limit the users in their study to a particular subject (Xie, 2007, pg. 1351). Xie’s (2007) notion critiques, and possibly furthers, the ideas previously put forth by Jeng (2005a) and Jeng (2005b), who focuses solely on usability and defining usability evaluation criteria.

Tsakonas and Papatheodorou (2008) mention Xie’s (2007) ideas about user evaluation, noting that “user-centered evaluation of a DL should be multi-constructed, capable to capture a panoramic view of users’ opinions, able to take into account their characteristics and grounded on their perceptions and goals” (pg. 1237). However, the authors still focus on usefulness and usability (along with performance) in their own study, signifying the importance of this particular quality in evaluation.

Furthermore, usability still remains a focus in the study conducted by Joo and Lee (2010). They write, “The ISO 9241-11 standard (1997) postulates three major attributes of usability – efficiency, effectiveness, and satisfaction” and they base their study off of these three characteristics of usability, and a fourth, learnability (Joo and Lee, 2010, pg. 527).

As the literature asserts, usability remains a major factor in the way that digital libraries should be evaluated. However, it may not be the only factor to consider; among criteria to consider: the collection or content, the performance of the system, and the design of the interface.

Evaluation Criteria

For this particular evaluation and comparison, I have chosen a combination of the criteria presented in the literature. The following chart depicts the criteria, as well as how to evaluate the criteria:

Evaluation Criteria / Attributes
Usability / efficiency, effectiveness, satisfaction
Collection / quality, quantity
Performance / response time, processing time

In terms of usability, efficiency, effectiveness, and satisfaction are good ways to analyze this criterion, and are purported by the ISO 9241-11 standard, as noted and defined in the work of Joo and Lee (2011, pg. 527).

Using the attributes listed in the above chart, we can create questions that help us analyze the digital library. Looking at the attributes of usability, as well as the work of Joo and Lee (2011), here are questions that will aid a user (and the author) in making observations about a DL:

Usability

Efficiency:

• Can I find what I need quickly and complete the task?
• Does the library’s design and interface allow for an easy and relatively rapid understanding of the system for the user?

Effectiveness:

• How organized is the system?
• Can I complete a search task using this DL?
• How successful am I in using this system?

Satisfaction:

• Am I happy with the results of my search?
• Am I comfortable using this DL?
• Is the interface visually appealing?

Collection

Quality:

• Because this is an academic DL, is the content peer-reviewed and scholarly?
• Do many of the resources contain references?
• Is the collection (and are the featured databases) regularly updated and maintained?

Quantity

• Are there a plethora of resources to choose from?

Performance

Response Time:

• How long does it take the staff to answer virtual reference questions?

Processing Time:

• How long does it take the site to load?

Using these questions, I will begin to make observations about the Rutgers University DL, The College of New Jersey DL, and the Queensborough Community College DL.

Rutgers University

The Rutgers University Digital Library, found at serves more than 67,000 students (48,000 undergraduate students and 19,000 graduate students) and 8,000 faculty members (“Facts & Figures,” 2016). The university is known for its research, as well as its extensive faculty and rigorous coursework. Its digital library serves to aid students and faculty in that research, and provides access to over 400 databases, as well as research guides for a variety of subjects.

Above: the Rutgers Libraries homepage.

The main layout of the site, as shown above, features a tabbed search bar and a live chat box, which allows the user to ask reference questions, or general questions about the libraries, in real time. There is also a spotlight on News, Featured Events, the Institutional Repository (RUcore) and “Information for…,” which provides information for different groups within the community. Above the Rutgers University Libraries logo is a toolbar, which allows users to search not only the site, but also find more information about the libraries, receive help, and find research guides.

Usability

Efficiency

The user can clearly find what they need to, and rather quickly at that. The website is extraordinarily user-friendly; if the user has experience at all with the Internet, they should be able to navigate this site. The structure of the site is not only visually stimulating, but the site is laid out very well. The important features, which are arguably the live chat and the tabbed search box, are at the center of the page. The top navigation bar, which helps lead the user to the part of the site he or she is searching for, is bright red and in large print, making it easy and fast for the user to use.

The tabbed search box is truly the highlight of the homepage, due to its easy-to-use, superior functionality. Once the user chooses which tab is relevant to what he or she is searching for, he or she can begin his or her search. If the user is looking for an article, the “Articles” tab, featuring an Articles+ search bar, is incredibly helpful. Articles+ searches over 200 databases, which means that the user can search more efficiently than if going to one individual database. If necessary, the user can use the link “View All Databases” to find a list of databases, organized by subject, as well as alphabetically. If the user needs to search for books, DVDs, or CDs available in the physical library, they can choose the “Books & Media” tab. There, they can find information regarding where to locate the book, how many copies are available, and if it is out on loan. Under the “Journals” tab, users can locate journal titles; however, they cannot use this tab to find articles. The “Course Reserves” tab is for the user to locate books, media, and articles a professor put aside for a specific class (”Rutgers Libraries,” 2016).

There is also an “Ask a Librarian” tab located on the side of each page (with the exception of the homepage) which links users to three different ways to contact a real librarian: phone, email, or live chat. This allows users to ask questions at any point in the search process, including while they are searching in databases.

Furthermore, making things all the more efficient, users can connect from anywhere, even off campus, with previously established login information provided from the university.

Effectiveness

The system is extremely organized, to say the least. The databases are organized alphabetically and by subject, as stated previously, and the LibGuides (research guides, found under the “Services & Tools” tab; then, “Research Assistance”) are organized by subject, by owner, and alphabetically. Furthermore, the system guarantees that its users are successful by providing a page entitled “How Do I…?” This page has links to answers to common questions, such as how do I “…find an article on my topic (“How Do I…?,” 2016).

Inexperienced users should be rather successful using this system; though databases may be hard to manage without experience, the DL has guides that lead the user to the principles of searching. Each page previously mentioned, “Research Assistance” and “How Do I…?” both serve that purpose, and are quite effective in helping the user in as many ways as they can. To add to that, a reference librarian is around during normal business hours to answer any questions, via email or live-chat, which makes completing a search much easier than ever before.

Satisfaction

Considering the number of databases available, as well as the amount of assistance given to all users, I cannot imagine how anyone would be dissatisfied with the Rutgers University DL. The system is complex, perhaps, harboring a vast number of resources; however, it is simple to use with the amount of aid provided.

The interface is attractive, and certainly more so to the average Rutgers user, given the fact that is designed with the university colors in mind. The red and white provide a stark contrast that allows the user to clearly see the different aspects of the site, and acts as a bright and stimulating environment to begin to search. The “Ask A Librarian” tab is also incredibly helpful, and satisfies the user at every step in their search.

Collection

Quality

The resources provided by the Rutgers University DL include well-known and well-respected databases that contain peer-reviewed, research-based articles that encompass a variety of topics that may be of interest to different majors within the university. Popular databases, such as Academic Search Premier, ProQuest, and EBSCOhost are all available to Rutgers Students, each of which contains hundreds of thousands of articles. When clicking on a database, the Rutgers University DL provides a brief summary of the database, as well as dates covered, the type of articles and content the database contains, and more. It is extremely helpful to students and faculty who are attempting to choose a database that’s right for their research.

Any journal articles that the Rutgers University DL does not have access to through one of its many databases can be requested via interlibrary loan, and made available online within two to three days.

RefWorks is another great tool that Rutgers University’s DL provides, free of charge to any faculty member or student. This particular site allows users to either import citations from databases directly, or lets users fill in the appropriate information into a simple form, in order to create a references list or bibliography. Flow and Endnote, similar tools, is also available for free to students and faculty members.

Quantity

There are a plethora of resources available through the Rutgers University DL. Although it is hard to measure quantity, there are over 400 available databases, and that’s not all. There are 696 research guides, in 82 different subjects; there are three different programs with which users can record citations; a number of open access journals; research data services and copyright workshops—the quantity cannot possibly be measured because there are so many rich resources of which students and faculty can take advantage.

Performance

Response Time

The librarians provide answers in a timely fashion. Live chat, within appropriate library hours, allows users access to information, from an actual reference librarian, immediately. Email usually takes 24 hours, with the exception of weekends and holidays. Phone calls will be answered and users will be helped during regular library hours.

Processing Time

The site loads quickly, and while some databases may take, at most, 30 seconds to load, these are neither managed by the Rutgers University DL, nor to blame due to the sheer amount of content being retrieved.

Results

Evaluation Criteria / Score
Usability / 5
Collection / 5
Performance / 5

The College of New Jersey

The College of New Jersey Digital Library serves a small number of students and faculty, especially compared to the tens of thousands Rutgers University’s Digital Library serves. The College of New Jersey (TCNJ) has approximately 6,400 undergraduate students (“About,” 2016), though I could not find totals on the number of graduate students, nor the number of faculty members (of course, it serves far less than Rutgers University does). TCNJ prides itself on having a more liberal arts focused education system, ensuring that all students are well rounded and have dabbled in a number of different subjects, along with their major coursework.

Above: the TCNJ Library homepage.

The College of New Jersey DL homepage features a plainer layout than the Rutgers University DL. It features more white space and the font sizes are a bit smaller. However, like the Rutgers DL, the main highlights of the page are the tabbed search box and the “Ask a Librarian” live chat box, both of which are at the center of the page. Because they are the most noticeable, the user can easily gauge their importance and choose which feature may best suit their research needs.

Near the top right of the page, the user can log into his or her library account and access information about the books they have checked out, late fees, etc. Underneath that, there are helpful links about the physical library on campus. Just below that is a gray navigation bar, featuring a drop downs with important links that help the user with research, interlibrary loans, and even course reserves. Also on the page is news about the library, links to frequently asked questions about how to search, library guides, and a link to RefWorks—The College of New Jersey, like Rutgers, provides access to this site for its students and faculty.