Transcript www.lcesc.org

Understanding & Using
Value-Added Analysis
Copyright ©2006. Battelle for Kids.
Learning Targets
• Develop confidence in accessing,
navigating and interpreting value-added
information.
• Understand the connections between
value-added information and school
improvement.
Introductions
• Name
• Role in Building and/or District
• Draw a face on a post it to indicate
your feelings about Value Added
The Work of the Day
• Morning will be spent learning about
value-added information:
– Conceptual understanding of value-added
– Overview of the information available in valueadded reports
• Afternoon will be spent:
– Navigating through value-added reports
– Learning about people and tools that can
support for your learning
– Connecting value-added to school
improvement
Getting Started
Value-Added Analysis
Copyright ©2006. Battelle for Kids.
Presentation Targets
• Understand the difference between
progress and achievement.
• Gain an introductory understanding
of value-added analysis.
• Acquire an awareness of information
available in value-added reports.
• Explore how value-added information
can inform school improvement.
Copyright ©2006. Battelle for Kids.
Stair-Step Expectations
Grade Three
Grade Four
Grade 3 Test
Grade 4 Test
Grade Five
Grade 5 Test
In a perfect world scores on year-end
achievement tests would be enough to
document growth. Students would start
school at the same place, and move
together toward proficiency.
Copyright ©2006. Battelle for Kids.
Copyright © 2003. Battelle for Kids
Differentiated Reality
Grade Three
Grade Four
Grade 3 Test
Grade 4 Test
Grade Five
Grade 5 Test
Students begin school at different places
and progress at different rates. This
complicates accountability. We need more
than scores on a single test to tell the story
of a school’s effectiveness.
Copyright ©2006. Battelle for Kids.
Copyright © 2003. Battelle for Kids
Achievement measures alone
provide a partial picture:
School improvement perspective
1st
Birth
Kindergarten
2nd
3rd
4th Grade Test
Achievement measures are snapshots in time.
They provide information on what a child has
learned since birth.
Copyright ©2006. Battelle for Kids.
If we want to improve schools we need a metric
that provides reliable feedback on the work that
schools produce.
A View of Student Progress:
Accountability perspective
100
90
80
70
Student A
Student B
Standard
Proficient
60
50
40
30
3
Copyright ©2006. Battelle for Kids.
4
5
6
7
8
We must expect progress for all
students.
The question for educators today
Regardless of where students begin,
how do we maximize the progress each
makes in any given year?
84% of the American public believe student progress is
the best measure of a school’s effectiveness!
Copyright ©2006. Battelle for Kids.
Copyright © 2003. Battelle for Kids
What is the
difference between
progress and
achievement?
Copyright ©2006. Battelle for Kids.
Value-Added Analysis
in Ohio
Project SOAR
Ohio’s
Value-Added
System
Copyright ©2006. Battelle for Kids.
Ohio’s System
• Operated by the Ohio Department
of Education.
• Began as a grade 4 pilot in 2006 in
all districts and community schools.
• Provides analysis in math & reading,
grades 4-8.
• Uses only state achievement test data.
• Uses the mean gain model.
• Expected growth is set by the State
Board of Education.
Copyright ©2006. Battelle for Kids.
Project SOAR
(Schools’ Online Assessment Reports)
• Operated by Battelle for Kids.
• Began in 2002 with 42 school districts.
• Now includes 100+ districts.
• Provides analysis in all core subjects,
grades 3-10.
• Uses state and non-state test data.
• Uses the mean predicted model.
• Expected growth is normative
(Average Growth).
Copyright ©2006. Battelle for Kids.
Why EVAAS™?
• Applies the most sophisticated statistical
methodology available to ensure reliability.
• Allows for the use of all student test data.
• Provides valuable diagnostic information.
• Accommodates different types of test data.
• Used statewide in Tennessee for more than
10 years.
Copyright ©2006. Battelle for Kids.
Ohio’s Scale Up Plan
R M Sc SS
Copyright ©2006. Battelle for Kids.
Value-Added in the State
Accountability System
• If a district or school demonstrates
more than expected growth over
two consecutive years it moves up
one level in the system.
• If a district or school demonstrates
less than expected growth over
three consecutive years it moves
down one level in the system.
• Value-added measures are
significant at one standard error.
Copyright ©2006. Battelle for Kids.
Value-Added Analysis*
Conceptual Overview
How is student assessment data
used to produce a value-added
growth metric?
Copyright ©2006. Battelle for Kids.
What is a Growth Measure?
Growth = Observed Score – Baseline Score
Copyright ©2006. Battelle for Kids.
Value-Added Example
4th
5th
6th
7th
8th
A value-added metric measures the average growth
that is produced in one subject in a given year.
This growth is then compared to the amount of
growth that would be expected for that group of
students. Growth above (or below) expectation is
the “value-added.”
Copyright ©2006. Battelle for Kids.
Test Properties
Tests must:
• Be highly correlated to curricular objectives.
• Have sufficient stretch.
• Have appropriate levels of reliability.
Copyright ©2006. Battelle for Kids.
Mean Gain Approach
Used when all tests across grade
levels have consistent scales.
School Effect ≈ Mean Gain – Expected Growth
Copyright ©2006. Battelle for Kids.
Mean Gain Example
Math scaled scores for the same students.
Year 1 • Grade 3
Student 1 377 = 26 (NCE)
Student 2 380 = 27
Student 3 360 = 19
Student 4 377 = 26
Student 5 381 = 28
Year 2 •
Student 1
Student 2
Student 3
Student 4
Student 5
Grade 4
400 = 37 (NCE)
385 = 30
395 = 35
405 = 41
390 = 33
Mean Baseline = 25.2
Mean Observed = 35.2
Growth = Mean Observed – Mean Baseline
Growth = 35.2 - 25.2 = 10.0
A crude measure of the growth for this group is 10 NCEs.
Copyright ©2006. Battelle for Kids.
Mean Gain Approach
(Converted to NCEs)
Sample School
Value-Added
Report Math
High Progress
Positive School
Effects
High
Achievement
High Mean NCE or
Mean Scaled Score
School
Effect
Baseline
Observed
Copyright ©2006. Battelle for Kids.
Mean Gain Approach
Copyright ©2006. Battelle for Kids.
(Converted to NCEs)
Describe, in
conceptual terms,
the “mean gain”
value-added model.
Copyright ©2006. Battelle for Kids.
What New
Diagnostic
Information Do
Value-Added
Reports
Provide?
Value-Added Summary Report
Where are the areas of strength and weakness?
Copyright ©2006. Battelle for Kids.
School Diagnostic Report
Are the curriculum and instruction working for all students?
Copyright ©2006. Battelle for Kids.
Performance Diagnostic Report
What pattern would represent optimum growth?
Copyright ©2006. Battelle for Kids.
Student Report
How does a student’s growth rate compare
to other groups of students?
Copyright ©2006. Battelle for Kids.
Student Projection
With average growth, where will this
student likely score in the future?
Copyright ©2006. Battelle for Kids.
Student Search
How can Predictions help identify at risk students?
Copyright ©2006. Battelle for Kids.
Student Search Results
How could this information change the
way educators approach intervention?
Copyright ©2006. Battelle for Kids.
In Summary, Value-Added
Information Shows…
• The students’ progress made in each subject
area and grade level.
• The progress students made who enter the year
at different achievement levels.
• How students’ progress in one curricular area or
program compares to their progress in another.
• Whether individual students are making
adequate progress toward state standards.
Without data, all we have are opinions!
Copyright ©2006. Battelle for Kids.
What new
information do
value-added reports
provide?
Copyright ©2006. Battelle for Kids.
Why is it important to
focus school
improvement
strategies at the
individual grade level
and/or subject area?
Copyright ©2006. Battelle for Kids.