Transcript Post School Outcomes: What Can We Learn from Trend Data? Pattie Johnson, WOU Charlotte Y.

Post School Outcomes: What
Can We Learn from Trend Data?
Pattie Johnson, WOU
Charlotte Y. Alverson, UO
Building Capacity Institute, 2013
Session Description
• Oregon has three years of PSO data with
consistent definitions for educational and
employment outcomes.
• What can we learn from examining the trends?
• Where are the key areas that districts can use
to evaluate progress toward increased
engagement for their students?
• Where will program changes have impact on
improving performance?
Consistency
• Since FFY 2008, states have had the
same measure and definitions for
Indicator 14, post-school outcomes.
• With FFY 2011 data collection, we
have 3 to 4 years of PSO data.
• We can now start to examine trends
in outcomes across years.
Data Collected Yearly by States
States are measured on their implementation of IDEA
through 20 Part B Indicators.
#14: Percent of youth who are no longer in
secondary school, had IEPs in effect at the time they
left school, and were:
1. Enrolled in “higher education”
2. In “competitive employment”
3. Enrolled in “other postsecondary education or
training”
4. In “some other employment”
5
Outcomes for Student with Disabilities as
Measured by Indicator 14
Median Percentage for Each Measure
Percent of Respondents
100.0
National FFY 2009
National FFY 2010
80.0
72.5
56.3
60.0
72.5
57.2
40.0
26.8
29.0
20.0
0.0
Higher Education
Higher Education + Competitve
Employment
United States Department of Education, Office of Special Education Programs (2011, 2012).
Part B State Performance Plan/ Annual Performance Reports 2011 & 2012 Indicator Analyses.
Higher Education + Competitve
Employment + Other
Postsecondary Education/Training
+ Some Other Employment
Indicator 14 for Federal Reporting
1=
2 =
3=
4=
# Higher Ed
# Competitive
Employed
#Postsecondary
Education or
Training
# Other
Employed
A = 1/
Total
respondents
B = 1+2/
Total respondents
# Other or Not
Engaged
(States are not
required to
report this #
but it is in the
denominator)
1 HE
2 CE
3 OEd
4 OW
5 NE
Cat
= 1+2+3+4/
We will look
the
five
outcome
categories
Total respondents
because these Total
areRespondents
more meaningful for
understanding our data and being able to use our
data for program improvements
Questions Guiding the Analysis
• How representative are these data?
• What direction are our outcomes going?
• Are there differences in outcomes by subgroups?
▫ Gender: Male, Female
▫ Disability: ID, ED, SLD, all other
▫ Method of Exit: Regular diploma, Completed, Dropout
▫ Ethnicity: Minority, Caucasian
• What is contributing to our outcomes?
• How can we use the information?
Looking at Data
• How representative are these data?
▫ Aggregate of response representativeness
• What direction are our outcomes going?
• Graphs of:
▫ Overall A, B, & C Measure x 3 years
▫ Overall 1, 2, 3, 4, & 5 x 3 years
• Are there differences in outcomes by subgroups?
• Gender Disability Method of Exit and Ethnicity categories x 3 years
• What is contributing to our outcomes?
• What supplemental survey questions will help answer this question?
PSO in Oregon
9
• 1-year prior to conducting the survey, districts can collect
accurate contact information on exiting students
• All Local Education Agencies (LEAs) collect follow up data,
larger LEAs are provided with prioritized list of leavers
selected to achieve a representative sample of leavers
based on race, disability, gender, and method of exit
• LEA personnel conduct phone interviews
• Responses are recorded in online secure website
How representative are
these data?
Representativeness: Basic Numbers
from Three Years
School year
Interview year
Total Leavers
Selected for interview
Completed interviews
Response rate
2008-09
2010
4295
2770
1911
68.9%
2009-10
2011
4425
2779
1989
71.6%
2010-11
2012
4244
2714
1748
64.4%
Oregon uses a stratified sample:
•All districts conduct interviews with students each year
•Small districts (15 leavers or less) interview all leavers
•Larger districts are provided with a sample of required students
to interview.
NPSO Calculator Representativeness:
Combining three years of data
Overall
Target Leaver Totals
Response Totals
Target Leaver
Representation
Respondent
Representation
Difference
12974
5648
LD
ED
MR
AO
Female
Minority
Dropout
6630 1246 1053 4045
4410
3198
3015
2840
461 1850
1907
1304
961
51.1% 9.6% 8.1% 31.2%
34.0%
24.7%
23.2%
50.3% 8.8% 8.2% 32.8%
33.8%
23.1%
17.0%
-0.8% -0.8% 0.1% 1.6%
-0.2%
-1. 6%
-6.2%
497
Dropouts are under underrepresented - a finding consistent with the each
years’ separate response analysis. Importance: to ensure sampled group
represents state population, the difference should be 3% or less. Caution
should be used in interpreting any results using the dropout category.
What direction are our
outcomes going?
▫ Overall A, B, & C Measures
▫ Overall 1, 2, 3, 4, & 5 categories
14
National and OR State PSO Data
100
National FFY 2009
State FFY 2010
State FFY 2009
State FFY 2011
Percent of Youth
80
72
56
60
66 68
72
55
51 54
40
27 24 25 25
20
0
Measure A
Measure B
Indicator 14 Measure
Measure C
Data Source: National aggregate of FFY 2009 SPP Submitted February 1, 2011; State data reported in the SPP FFY 2009 & APR FFY 2010, 2011
What direction are our outcomes
going: Outcomes by Three Years
2009
2010
2011
Percent of Respondents
100
80
60
40
20
0
Hi Ed
Comp Empl Other School Other Emp Not Engaged
Outcome Category
What do we see in the trends?
• Higher Education initial increase, then static
• Competitive Employment Increasing
▫ More Oregon leavers employed than in education
• Other School and Other Work relatively unchanged
▫ Dip in middle year- 2010
▫ Slightly more leavers in Other Work than Other
Education
• Not Engaged rate decreasing – right direction
• Why look further?
Are there differences in
outcomes by subgroups?
 Gender x 3 years
 Disability categories x 3 years
 Ethnicity categories x 3 years
 Method of Exit x 3 years
Differences in Outcomes by Gender
Engagement - Females
2010 F
2011 F
2012 F
100
80
Percent Reported
60
40
20
0
Hi Ed
CompEmp
OthSch
OthWork
NE
Engagement Males
2010 M
2011 M
2012 M
100
80
60
40
20
0
Hi Ed
CompEmp
OthSch
OthWork
NE
Observations for Outcomes by Gender
• More Females than Males in Hi Ed and both groups have
fairly static trend
• More Males than Females in Competitive Employment
with increase in trend for Males
• Other School engagement about the same rate for
Females and Males
• Other Employment similar rates, but Females have
increasing trend over time
• Not Engage decreasing trend for both groups
Differences in Outcomes by Disability Categories:
SLD and ED
Specific Learning Disability
1 HE
100
2 CE
3 Oed
4 OW
5 NE
80
60
40
20
0
2009
100
2010
Emotional Disturbance
2011
80
60
40
20
0
2009
2010
2011
Differences in Outcomes by Disability Categories:
All Other (Low Incidence) and ID
Low Incidence
1 HE
2 CE
3 Oed
4 OW
5 NE
100
80
60
40
20
0
2009
2010
2011
Intellectual Disability
100
80
60
40
20
0
2009
2010
2011
Observations for Outcomes by Disability
• SLD: slight positive trend for HE and CE and slight
negative trend for NE – trends going in desired direction
• ED: negative trend in HE, but positive trend in CE;
negative trend in NE- need to explore HE
• AO/Low Incidence: Slight increase in CE, other
engagement categories unchanged
• ID: negative trend in HE, positive trend in all other
engagement categories; highest group NE, but decreasing
• Regardless of disability, about 1/3 of respondents are NE,
HOWEVER, the trend is headed in the desired direction –
seeing a negative trend in all disability categories - need
to explore NE
Differences in Outcomes by Race/Ethnicity Categories
White
1 HE
100.0
2 CE
3 Oed
4 OW
5 NE
80.0
60.0
40.0
20.0
0.0
2009
100.0
2010
2011
Minority
80.0
60.0
40.0
20.0
0.0
2009
2010
2011
Combined Years for Sufficient Size of
Race/Ethnicity Subgroups for Comparison
Combined Three Years
1 HE
2 CE
3 Oed
4 OW
5 NE
Percent of Respondents
100
80
60
40
20
0
Asian
n=128
Black
n=185
Hispanic
n=725
Nat Amer
n=210
Race/Ethnicity Categories
White
n =4348
Observations for Outcomes by Race/Ethnicity
• There are very minor outcome differences when all
minorities are in one subgroup
• There are insufficient interviews to look at trends over
the 3 years by each minority group
• Combining the data from the three years, as a
representative sample was interviewed, allows further
exploration
• Differences are seen in the students combined into the
minority subgroup that need to be explored further by
the state and districts
Outcomes by Method of Exit category Regular Diploma
Regular Diploma
1 HE
2 CE
3 Oed
4 OW
5 NE
100
80
60
40
20
0
2009
2010
2011
Outcomes by Method of Exit category Dropout
Dropout- Not representative
1 HE
100
2 CE
3 Oed
4 OW
5 NE
80
60
40
20
0
2009
2010
2011
Observations for Outcomes by Method of
Exit Groups: Regular Diploma and Dropout
• Regular Diplomas: slight increases in HE and
CE, and decrease in NE; trends are going in
the desired directions
• Dropouts – not representative of the state
▫ Decrease in HE, slight increase in CE with a dip
in 2010
▫ Slight increase in Other Education; static in
Other Work
▫ Decrease in NE
Summary Observations
• Not Engaged – negative trend
▫ Rate is slow
▫ High number of youth in some subgroups
 ID
 Dropout
• Higher Ed and Competitive Employment
shifting to Other categories
▫ ED – negative trend on HE with increase in
Other Education
What is contributing to
our outcomes?
Supplemental survey questions can
help answer this question.
• Additional questions included on the follow- up
interview in Oregon:
▫ Do you have a drivers license?
▫ What is your living situation?
▫ What one thing would you tell your school?
▫ Which independent activities can you do?
▫ Do you receive benefits like co-workers?
▫ What do you do for recreation?
▫ If you haven’t worked, why not?
▫ Have you received support from adult Agencies?
What Agency Services have you
accessed since leaving school?
The list of agencies on the follow-up interview
includes:
▫ Social Security Disability Insurance or
Supplemental Security Income
▫ Developmental Disability services
▫ Office of Vocational Rehabilitation
▫ Temporary Assistance for Needy Families
▫ Supplemental Nutrition Assistance Program
▫ College Disability Services
▫ Loans, Financial Aid
Change in Outcome Classification
• For the next series of charts, the outcome groups
were modified to allow a closer look at students
who tried school or work, but were not successful.
▫ 1 HE 2 CE 3/4 Other ed/work 5-Attempted 5-None
▫ 5-Attempted: students answered Yes to either school/
training or employment, but did not continue long
enough to qualify as ‘engaged’
▫ 5-None: students reported NO school/training or
employment experience
Differences in Outcomes by Agency Services Received
Vocational Rehabilitation Services
100
1 HE
2 CE
3/4
5 -Attempted
5 None
80
60
40
25 26
20
17 17
25 30
11
21
25
31
11
17
0
2009 n = 216
2010 n = 198
100
2011 n = 192
No VR Services
80
60
40
20
21 26
19
27
12
20
11
17
26 22
9
0
2009 n=1695
2010 n=1792
2011 n=1556
17
Differences in Outcomes by Agency Services Received
100
80
Developmental Disability Services
1 HE
2 CE
3 or 4
5 Attempted
5 None
60
40
20
0
2009 n=205
2010 n=180
No DD Services
2011 n=186
100
80
60
40
20
0
2009
2010
2011
How Can We Use These Data:
Next Steps
• Finalize the analysis
▫ Other supplemental questions
• Share and discuss trend data
▫ ODE Transition Specialist
▫ Transition Advisory Council Stakeholders
▫ Agency Partners: ODDS and VR
• Determine what data to share and how to share
▫ District and School Stakeholders
Looking at Data: Process summary
• How representative are these data?
▫ We explored the response size and how the subgroups
matched the population
• What direction are our outcomes going?
▫ Looked at graphs showing performance, trends, and
comparisons
• Are there differences in outcomes by subgroups?
• Worked from general overview to more specific components
• What is contributing to our outcomes?
• Looked at a combination of components, modified the question if
necessary, and summarized what we learned at each step
• For more information:
▫ Pattie Johnson
 Teaching Research Institute, Western Oregon University
 [email protected] 503-838-8779
▫ Charlotte Y. Alverson
 National Post School Outcomes Center, University of Oregon
 [email protected] 541-346-1390