Transcript Slide 1

HOW DO WE KNOW WE ARE ACHIEVING OUR
MISSIONS?
ASSESSMENT OF MISSION AT CHRISTIAN
COLLEGES AND UNIVERSITIES
GAY H. HOLCOMB, PH.D.
Director of Institutional Research and Assessment
ASBURY COLLEGE
QUALITY ENHANCEMENT PLAN (QEP)

Recent requirement of SACS
 Focus
on a learning outcome central to the mission
of the institution
 Become intentional about a critical element of its
mission.
 Broad-based involvement & support
 Commit significant resources, able to sustain
 Topic that institution becomes known for
ASBURY’S QEP ON SPIRITUAL VITALITY

Academic Excellence & Spiritual Vitality
Cornerstone Project
 Holiness
 Scripture
 Stewardship
 Mission
STUDENT LEARNING OUTCOMES
1.
2.
3.
4.
5.
6.
7.
Understand Cornerstones
Core Curriculum
Personally Value Cornerstones
Creation Care
Service Learning
Christian Mission & Stewardship
Increase Student Initiative
MEASURES
1.
2.
3.
4.
5.
6.
7.
8.
Capstone Course Assignments
Furnishing the Soul Inventory by Alidade
Case Studies of Exemplars
Chapel Survey
Alumni Surveys
Co-Curricular Involvement Survey & Participation
Logs
Waste Audits
Mix of qualitative/quantitative & direct/indirect
AT THE HEART OF WHO ASBURY IS

Enthusiastic buy-in
 Each
Cornerstone tied to Mission
 Each Stu Learning Outcome, Measure,
Achievement Target tied explicitly to Strategic Vision
 Impacts everything we do

Cornerstone Project approved by SACS in Dec
 Initial

phase – no data yet
Fully re-accredited by SACS in December 2009
JOEL FREDERICKSON, PH.D.
Professor and Chair, Psychology Department
BETHEL UNIVERSITY
MINING EXISTING DATA
Institutions generally participate in national
surveys (indirect assessment) and
standardized tests (direct assessment)
 Use this existing data to assess your
institution’s mission and core values

EXAMPLES OF SURVEYS
National Survey of Student Engagement (NSSE)
 Cooperating Institutions Research Project
(CIRP) First-year survey (from HERI @ UCLA)
 College Senior Survey (CSS) Follow-up to CIRP
 College Student Beliefs & Values Inventory
(CSBV) from HERI
 Student Satisfaction Inventory (SSI)

 Additional
CCCU items are added to this survey.
EXAMPLES OF DIRECT ASSESSMENTS OF
LEARNING
Collegiate Learning Assessment (CLA)
 Collegiate Assessment of Academic Proficiency
(CAAP), from ACT
 Major Field Tests, from ETS
 Defining Issues Test (DIT): Measure of Moral
Thinking
 California Critical Thinking Skills Test (CCTST)

LINKING ASSESSMENTS TO MISSION
Examine your institution’s mission, vision, and,
if applicable, core values.
 Do an audit of the various assessments your
institution is engaged in.
 Link the two together.

NSSE items: Bethel’s Core Value of Reconciliation





1e. Included diverse perspectives (different races, religions,
genders, political beliefs, etc.) in class discussions or writing
assignments
1u. Had serious conversations with students of a different
race or ethnicity than your own
1v. Had serious conversations with students who are very
different from you in terms of their religious beliefs, political
opinions, or personal values
10c. Institutional Emphasis: Encouraging contact among
students from different economic, social, and racial or ethnic
backgrounds
11l. Educational & Personal Growth: Understanding people
of other racial and ethnic backgrounds
HAD SERIOUS CONVERSATIONS WITH STUDENTS OF A
DIFFERENT RACE OR ETHNICITY THAN YOUR OWN
2.8
2.7
2.6
2.5
Bethel
CCCU
2.4
Carnegie
2.3
2.2
2.1
2005
2006
2007
2008
2009
ENCOURAGING CONTACT AMONG STUDENTS FROM DIFFERENT
ECONOMIC, SOCIAL, AND RACIAL OR ETHNIC BACKGROUNDS
3
2.9
2.8
2.7
2.6
Bethel
CCCU
2.5
Carnegie
2.4
2.3
2.2
2.1
2005
2006
2007
2008
2009
STUDENT SATISFACTION INVENTORY: BETHEL’S
CORE VALUE OF CHRIST-FOLLOWERS
 74.
Being on this campus is contributing to my
spiritual growth.
 75.My understanding of God is being strengthened
by classroom and/or campus experiences.
 76.Faculty, administrators, and/or staff are helpful
to me in processing issues related to my faith.
 78.This campus provides adequate opportunities
for involvement in ministry.
 79.Given where I am spiritually right now, this
campus is a good "fit" for me.
BEING ON THIS CAMPUS IS CONTRIBUTING TO
MY SPIRITUAL GROWTH
6.6
6.4
6.2
6
Importance
Satisfaction
5.8
5.6
5.4
5.2
Bethel
CCCU
CSS: ADDRESSING THE LEADERSHIP
COMPONENT OF OUR MISSION






Students “very satisfied” or “satisfied” with their college in: Leadership
opportunities
Since entering college you have: Participated in leadership training
Students reporting “much stronger” abilities and skills compared to when
they first entered college: Leadership abilities
Compared with the average person of his/her age, student rated self “above
average” or “highest 10%” in Leadership ability
Considerations noted as “essential” or “very important” in thinking about
student’s career path: Leadership potential
Student objectives noted as “essential” or “very important”: Becoming a
Community Leader
CSS ANALYSES
Comparisons with other 4-year religious
colleges & all 4-year private colleges
 Longitudinal comparisons for:

 Becoming
a Community Leader
 Leadership ability
BECOMING A COMMUNITY LEADER
LONGITUDINAL CHANGES
First-year
Senior
Change
Bethel
35.5
44.1
8.6
Religious Colleges
31.8
41.1
9.3
All Private Colleges
34.0
40.8
6.8
CORE VALUES: LEARNERS & TRUTH-SEEKERS

Combining data from direct and indirect
assessments.
 CCTST
indicates good growth in Critical Thinking
 CLA results indicate poor growth in integrative
writing
 NSSE & CSS data indicate that students perceive
that they are growing in critical thinking
SUMMARY
Don’t get caught up in precise definitions (the
“what really is critical thinking?” conversations)
 Focus on using a variety of exisiting measures
to assess mission.
 Look for a convergence of evidence

TIMOTHY DETWILER, Ph.D.
Associate Provost for Traditional Programming
CORNERSTONE UNIVERSITY
CORNERSTONE UNIVERSITY

2650 Students
 1200

Traditional
250 Seminary
- 1000 Adults
- 200 Seminary (Asia)
Began in 1941
 A faith-based institution
 Grand Rapids, St. Joseph, Kalamazoo, Detroit,
Holland
 Worldview, information literacy, civitas and
spiritual formation mark the programs

Mission Statement
University Assessment Plan
Campus-wide Learning Objectives
Core Curriculum Design
A Specific Core Course
CORNERSTONE UNIVERSITY
Mission Statement
To empower men and women to excel as
influencers in our world for Christ by offering a
student-focused learning community where
Jesus Christ is central.
MISSION STATEMENTS

One of the main purposes of a mission
statement is to guide decisions about
organizational priorities in a systematic
manner. Once an organization has identified
what it wants to accomplish, it has to
determine how to accomplish those things and
how to measure the success of those activities.
(Nancy Kidd – NCA Director)

Effective organizations are the momentary
results of constant reassessment and redesign.
When the effectiveness of organizations is not
tested, they rapidly tend to become
unresponsive and inefficient.
Rodney Stark
ACADEMICS/SPIRITUAL FORMATION
A public frame for the assessment efforts
 Resonate deeply with the cultural memory
 A stimulus to initiate change

Referring to a mason’s level
 A leveling instrument or a plumb line
 Placing our “stones” in order and keeping them
in line

MISHQELET PROJECT
“CIVITAS”
DIRECT ASSESSMENT METHODS







Essays (pre/post experiences)
Exams
Experiential Aspects (self reporting/observational)
Electronic Discussion Board (rubric)
Journal (rubric)
Capstone Project
Portfolio
CIVITAS
INDIRECT METHODS OF ASSESSMENT
Student Satisfaction Survey (EFS, SSI, CSS)
 Experiential Reflective Journals
 Alumni Surveys
 Course Evaluation Forms
 Professional Evaluation of the Program

ENG 212 WRITING IN CULTURE
Link to mission
 Link to core curriculum learning objectives
 Link to the Humanities Division
 ENG 212

THE KEY TO IT ALL
Institutional leaders have the major role in determining
whether assessment results are important in
making improvements. Unless top-level
administrators believe that planning and
assessment data are linked and important in
strategic and logistic decisions, the assessment
program in general will be brought into a deadend. (IACBE)
MISSION DRIVEN ASSESSMENT
The end of virtually all assessment activities
(except for some accountability initiatives) is
the improvement of student learning, campus
programming and institutional enhancement –
in other words “organizational development”.
(Nichols & Nichols)
ORGANIZATIONAL DEVELOPMENT

Assessment is a strategy for organizational






Accountability
Distinctiveness
Accreditation
Effectiveness
Reflection
Development
ORGANIZATIONAL DEVELOPMENT

MORE IMPORTANTLY, assessment is a strategy
for understanding, confirming and improving
student learning and educational quality based
upon an institution’s mission.
Your Context
Your Mission, History,
Culture, Vision, Goals
Assessment is a matter of
institutional responsibility,
integrity,
and mission.
DAVID W. KALE, Ph.D.
Director of Assessment and Professor of Communication
MOUNT VERNON NAZARENE UNIVERSITY
ASSESSING STUDENT LEADERSHIP
MVNU Vision Statement: An academic
community of faith preparing Christ-like leaders
for life long service and global impact.
 The purpose of this assessment project was to
help students assess their own leadership
development.
 In the process we wanted to see the degree to
which institutional vision was reflected in
student thinking about leadership.

ASSESSING STUDENT LEADERSHIP
To achieve this, I worked with student leaders
to develop a rubric with which they could
assess their own leadership ability.
 I started by having all leaders list aspects of
leadership they would like to use to assess
themselves.
 I then compiled a master list with all the
leadership aspects included in their lists.

ASSESSING STUDENT LEADERSHIP
I then presented the total list to them and
asked them to vote for five on the list they
wanted to use to assess themselves.
 Based on their voting, I then constructed a
rubric using their top five dimensions.
 You have a copy of the rubric I constructed in
the materials which were distributed to you.

ASSESSING STUDENT LEADERSHIP
As I have found in using this method of rubric
construction in other settings, the method itself
is an important assessment procedure.
 What aspects of leadership do our students
consider so important that they want to use
them in assessing their own leadership
abilities?

ASSESSING STUDENT LEADERSHIP
1.
2.
3.
4.
5.
Compassion
Empowering
Communication – listening
Honesty
Empathy
ASSESSING STUDENT LEADERSHIP
We were particularly interested and pleased to
see compassion as the #1 leadership ability
chosen by our students.
 This certainly ties in to the vision statement’s
emphasis on developing Christ-like leaders.
 One of our staff members who had done his
doctoral studies in leadership indicated that he
had never seen a list of leadership qualities
that included compassion.

ASSESSING STUDENT LEADERSHIP
While we were very pleased with the orientation of
the list toward empowering and meeting the needs
of others, we also had some concerns with the list.
 Leadership characteristics such as problem
solving, ability to motivate others and creativity
came in between 15th and 20th on the list. This
gave our student development staff some
information about the kinds of training student
leaders needed.

ASSESSING STUDENT LEADERSHIP
Another concern we had with the rubric is that
when the student leaders used it to assess
themselves, they gave themselves very high
marks right from the start.
 We therefore had a ceiling effect which negated
our plan to have students assess themselves at
the beginning and end of the academic year to
chart their development during that time.

ASSESSING STUDENT LEADERSHIP

This was a surprise to me because in using this
approach to rubric construction for student self
assessment in their moral and ethical
reasoning ability, almost none of them gave
themselves high marks.
ASSESSING STUDENT LEADERSHIP



In summary of this strategy, we found the rubric
construction process to be very informative in terms of
giving us feedback on the degree to which institutional
vision was reflected in the leadership qualities students
selected to assess their leadership abilities.
It was also helpful in providing information to student
development as to areas of leadership development
they might focus on in training student leaders.
We did not find this strategy helpful in developing a
method of student self assessment that would permit
them to identify areas in which they needed to improve
or to track their development over an academic year.
GENERAL EDUCATION OBJECTIVE

We have a general education objective
associated with our mission statement which
addresses developing students’ abilities in
moral and ethical reasoning.
SENIOR REQUIRED GENERAL EDUCATION
COURSE IN APPLIED CHRISTIAN ETHICS
Section sizes kept to 24 students
 Debate national health care, drilling for oil in
ANWR, affirmative action, waterboarding, etc.
 Course text: Stassen and Gushee, Kingdom
Ethics, Intervarsity Press

DEFINING ISSUES TEST
Based on the concepts of Lawrence Kohlberg
 Essentially a critical thinking test in the area of
ethics
 Students get scores in Personal Interest,
Maintaining Norms, Post conventional or
Principled Reasoning , etc.
 We administer the test to first year students,
juniors and seniors.

DEFINING ISSUES TEST – 2
PERSONAL INTEREST SCORE
MVNU SENIORS 2006
Mean
S. D.
N
MVNU
Seniors
23.06
10.62
76
MVNU
Juniors
26.94
11.43
60
T= -2.04
P= <.05
D = .35
Nat’l
24.8
Sample
All Seniors
12.53
2441
ns
Nat’l
Sample
All
Politically
conserv.
Seniors
12.01
881
ns
23.48
Sig.
MODEL FOR ANALYZING
MORAL AND ETHICAL ARGUMENTS
Powers and
Authorities
Social change
Threats
Openness
Loyalties
Interests
Passions
Models
Rules and Practices
Principles and
Virtues
BASIC CONVICTIONS
God and Human Nature
Forgiveness
Christlikeness and Justice
Mission of the Church
USE OF THE MODEL IN THE TEXT
Students use the model in class in groups to
analyze a case of euthanasia from 20/20 on
television.
 Students use the model individually on an
outside of class assignment to analyze a case
provided by the instructor.
 For mid-term exam, students use the model on
a case they have not seen before exam period.

DEFINING ISSUES TEST – 2
PERSONAL INTEREST SCORE
MVNU SENIORS, 2006 &2009
Mean
S. D.
N
MVNU
Seniors
( 2006)
23.06
10.62
76
MVNU
Seniors
(2009)
21.17
11.28
44
National
24.8
Sample –
All Seniors
12.9953
2441
D = .347
T = -2.01
P < .05
MVNU
Juniors
(2006)
11.82
164
D= .49
T= -4.04
P< .05
26.56
Sig.
SUMMARY

Now that there is evidence that I am making
progress in reducing students’ use of personal
interest in making moral and ethical decisions,
I would like to see even more progress on their
parts in using internalized biblical principles as
opposed to relying so much on external
sources.