Transcript ils.unc.edu

NCknows Evaluation Overview
Jeffrey Pomerantz, Lili Luo
Charles McClure
School of Information & Library Science School of Information Studies
UNC Chapel Hill
Florida State University
<pomerantz, luo>@unc.edu
[email protected]
Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931
Evaluation question
Is collaborative virtual reference an effective way to
meet the information needs of North Carolinians?
Secondary evaluation questions
What is required of a library that wishes to offer virtual reference?
What is the value added if different types of institutions work together?
What is the impact on libraries that provide virtual reference service?
Is virtual reference expandable to the whole state?
How will this project increase our knowledge of effective organizational
models?
How can the quality of the reference service provided be measured?
Can staff from different types of libraries provide quality reference service to
users from other types of libraries?
How will the project further greater use of existing resources such as NC
LIVE?
What partnering or leveraging opportunities exist?
Stakeholders
NCknows users
The individual participating libraries
The entire collaborative effort
The State Library of North Carolina
Evaluation from all of these perspectives
Data collection methods
1. Service: statistical analysis
2. Chat sessions: peer review of transcripts
3. Patrons: exit surveys & follow-up phone interviews
4. Librarians: phone interviews
Statistical analysis
What patterns are emerging in the volume of questions
received by NCknows?
Usage over time
NCknows librarians vs. 24/7 staff
Net asker / Net answerer
Ju
l-0
4
Au
g04
Se
p04
O
ct
-0
4
Ju
n04
Ap
r- 0
4
M
ay
-0
4
Fe
b04
M
ar
-0
4
Sessions per month
1400
1200
1000
800
600
400
200
0
on
da
y
y
da
y
Sa
tu
rd
ay
Fr
id
ay
Th
ur
sd
ay
W
ed
ne
s
Tu
es
da
y
M
Su
nd
a
Sessions per day
20
18
16
14
12
10
8
6
4
2
0
1 1A
-2 M
2 AM
-3
3 AM
-4
4 AM
-5
5 AM
-6
6 AM
-7
7 AM
-8
8 AM
9 9A
-1 M
1
11 0 - 0 A
A M 11 M
No - A M
on No
- on
1 1P
-2 M
2 PM
-3
3 PM
-4
4 PM
-5
5 PM
-6
6 PM
-7
7 PM
-8
8 PM
9 9P
11 1 10 M
PM0 - P
1 M
-M 1P
idn M
igh
t
M
id
nig
ht
Sessions per hour: weekdays
450
400
Total
350
NCknows
300
250
200
150
100
50
0
1 1A
-2 M
2 AM
-3
3 AM
-4
4 AM
-5
5 AM
-6
6 AM
-7
7 AM
-8
8 AM
9 9A
-1 M
1
11 0 - 0 A
A M 11 M
No - A M
on No
- on
1 1P
-2 M
2 PM
-3
3 PM
-4
4 PM
-5
5 PM
-6
6 PM
-7
7 PM
-8
8 PM
9 9P
11 1 10 M
PM0 - P
1 M
-M 1P
idn M
igh
t
M
id
nig
ht
Sessions per hour: weekends
90
80
Total
70
NCknows
60
50
40
30
20
10
0
Net asker / Net answerer
40
35
Net asker
30
25
20
15
Net answerer
10
5
0
-5
-10
Transcript peer review
What is the quality of reference service being provided
to users by NCknows?
Overall quality of sessions
NCknows librarians vs. 24/7 staff
Academic vs. public librarians
NCknows vs. 24/7:
Evidence of user satisfaction
% of responses
NCknows librarian 24/7 staff
Yes
32.0
17.6
Indirect evidence
32.3
35.9
3.0
2.3
32.7
44.3
No
Indeterminate
Academic vs. Public librarians:
Complete and correct answer
% of responses
Via academic library
Academic
librarian
Via public library
Public
librarian
Academic
librarian
Public
librarian
Complete &
correct
40.0
62.0
43.1
69.6
Incomplete
but correct
36.4
22.5
22.4
23.2
Incomplete
and incorrect
3.6
1.4
3.4
2.9
Incorrect
0.0
0.0
5.2
0.0
20.0
14.1
25.9
4.3
No answer
provided
Patron exit surveys &
Follow-up interviews
What are the demographics of the user population?
How do users find out about NCknows?
What is the users’ level of satisfaction with the
reference service being provided by NCknows?
What motivates users to use NCknows?
How do users use the information provided to them?
Role
Higher Ed
Faculty, 2.5
Adult Educator,
2.5
Medical
Professional,
2.5
Teacher: K-12,
3.3
Teacher: Preschool, 0.4
Other, 34.3
Parent, 7.9
Librarian, 6.3
Student: K-12,
7.5
Student:
Graduate, 11.7
Student:
Undergraduate,
18.0
Answer completeness
Dissatisfied,
4.9
Very
dissatisfied, 2.4
Satisfied, 21.9
Very satisfied,
70.9
Motivation for the question
For a work-related task (49%): business-related and
school-related (50/50 split)
For Personal Reasons (28%)
Known-item search (23%)
Use of the information provided
Use: patron had used the information provided and
found it useful (61%)
Partial use: patron had used the information provided
and had found it partially useful, or had partially used
the information provided (24%)
No use: patron had not used the information provided at
all (15%).
Discovery of NCknows
Teacher/professor (10%)
Search engine (20%)
Library materials (70%)
Phone interviews with librarians
How has involvement in the NCknows service impacted
the participating libraries and librarians?
Training
Most useful: Patron-librarian “role playing”
Most in need of review: Co-browsing
Policies & Procedures
Scheduling
Handling email follow-ups
Quality control
Tech support
Support:
•NCknows: good
•24/7: ok
Infrastructure:
• Adequate or better in university & large public libraries
• Sometimes less than adequate in community college & small
public libraries
Thoughts on chat reference
Inferior to desk reference:
•Lacks “non-verbal” cues
•Conveys less information in more time
•More difficult to conduct an interview
Good for:
•Quick answers to well-defined questions
Will continue to develop & evolve as an aspect of
traditional reference.
Additional & future work
Cost/benefit analysis
Sustainability
Scalability: to the entire state?
Situational and contextual factors unique to specific
libraries that affect quality of chat reference
MISs & databases to relate reference statistics to other
library statistics
Longitudinal data
Key evaluation issues
Understanding the importance of evaluation
Ongoing funding/support for evaluation: a “culture of
assessment”
How will evaluation data be used?
Quality of data: both the data reported here, and in
other data collection activities
The big picture
The need to have ongoing evaluation data
Importance of a statewide initiative in digital reference
Understanding impacts and applications
The context of digital reference efforts elsewhere
Ultimately the question is:
Do the benefits and outcomes outweigh the costs?
Congratulations!
Reports online
ils.unc.edu/~jpom/ncknows/
THANKS!
Questions or Comments?
Jeffrey Pomerantz, Lili Luo
School of Information and Library Science
UNC Chapel Hill
<pomerantz, luo>@unc.edu
Charles McClure
School of Information Studies
Florida State University
[email protected]