Transcript The State of the Science in Rehabilitation Measurement: Emerging Common Data Elements David S.

The State of the Science in
Rehabilitation Measurement:
Emerging Common Data
Elements
David S. Tulsky
New York University Langone Medical Center
&
Kessler Foundation
1
Presentation Objectives
1.
2.
3.
4.
Discuss the NINDS Common Data Element
Program and how it might impact research and
practice.
Explain how patient centric approaches have
helped identify issues relevant to rehabilitation
populations.
Review new measurement scales and item
banks developed for individuals with spinal cord
injury
Introduce a new traumatic brain injury patient
reported outcome measurement system.
2
NINDS Common Data Elements
Neuromuscular
(ALS, FA,
Dystrophies)
CDEs available for use
are found at:
http://www.commondataelements.
ninds.nih.gov/
TBI
Epilepsy
HD
MS
“General”
Common Data
Elements
SCI
Stroke
Headache
Other
PD
CDE Development Process
• Three descriptive levels of CDEs:
– Core: Minimal set of measures to
characterize a broad spectrum of subjects on
the domain.
– Supplemental: Intended for greater
depth/breadth of exploration and/or more
specialized subpopulations.
– Emerging: Measures requiring further
validation, but fill gaps in currently validated
measures and/or substitute for recommended
measures when validation is complete.
4
TBI CDE publication – Archives
of Phys Med & Rehab, 2010
The work groups that produced these articles identified
emerging measures that may become important and
useful in the future. These range from nearly empty
placeholders … to recently developed measures that are
theoretically attractive in meeting a measurement gap but
have very little empirical support or psychometric
development to date, to measures that have substantial
psychometric support but are not quite at the stage of
replacing older measures or have been infrequently used
in the proposed context.
--- Whyte, Vasterling, & Manley (2010)
5
Emerging Measures
•
•
•
•
•
Patient Reported Outcomes Measurement
Information System (PROMIS)
Quality of Life in Neurological Disorders - NeuroQOL
Spinal Cord Injury – Functional Index Measures
(SCI-FI)
Spinal Cord Injury Quality of Life Measures (SCIQOL)
Traumatic Brain Injury Quality of Life Measures
(TBI-QOL)
Toolbox for Neurological and Behavioral
6
Functioning
Emerging Measures
•
•
•
•
•
Patient Reported Outcomes Measurement
Information System (PROMIS)
Quality of Life in Neurological Disorders - NeuroQOL
Spinal Cord Injury – Functional Index Measures
(SCI-FI)
Spinal Cord Injury Quality of Life Measures (SCIQOL)
Traumatic Brain Injury Quality of Life Measures
(TBI-QOL)
Toolbox for Neurological and Behavioral
7
Functioning
Collaborating Centers
•
•
•
•
•
•
•
•
•
•
•
•
Northwestern University
Rehabilitation Institute of Chicago
Kessler Foundation
University of Michigan
Craig Hospital
Boston University Medical Center
The Institute For Rehabilitation Research
University of Washington
Mt. Sinai Medical Center
Bronx VA
Tampa VA
Santa Clara Valley Medical Center
Funding Sources
• PROMIS II: NIH Grant # 1U01AR057929
• SCI-FI: NIDRR Grant # H133N060022, H133N060024,
H133N060005, H133N060014, H133N060027, H133N060032
• SCI-QOL: NIH Grant # 5R01HD054659
• SCI-QOL Module: NIDRR Grant #s H133N110002, H133N110006,
H133N110020, H133N110014, H133N110019, H133N110003,
H133N110005, H133N110007
• SCI-FI 2: NIDRR Grant #s H133N110002, H133N110007,
H133N110006, H133N110020, H133N110019, H133N110014,
H133N110005, H133N110003
• TBI-QOL: NIDRR Grant # H122G070138
• TBI Model System: NIDRR Grant # H133A070037, H133A080045,
H133A070038, H133A070043
• Deployment-related TBI-QOL: VA RR&D Grant #B6237R
Objectives of PROMIS/NeuroQOL/SCI-QOL/TBI-QOL
• Develop and test large item banks measuring
patient-reported outcomes (PROs)
• Focus on a wide range of domains of functioning
• Create a computerized adaptive testing (CAT)
system for efficient, psychometrically robust
assessment
• Create a publicly available item repository and
CAT platform that can be accessed by the public 10
PROMIS Measurement
Systems
• Part of NIH Roadmap (or Common Fund)
• Designed to re-engineer the clinical
research enterprise (throughout NIH).
• Allows comparison across medical
conditions.
• Large initiative – $100 million
• Spanish and English
• Adult and Pediatric
11
PROMIS II
12
PROMIS Domain Framework
Symptoms
Physical Health
Function
Affect
Self-Reported
Health
Mental Health
Behavior
Cognition
Social
Health
Relationships
Function
PROMIS Current Physical Health Banks
Adult
Pediatric
Pain Behavior
Physical
Health
Pain Interference
Pain Interference
Fatigue
Fatigue
Physical Function
Upper Extremity
Function
Sleep Disturbance
Mobility
Sleep-related
Impairment
Asthma Impact
Sexual Function
PROMIS Current Mental Health Banks
Adult
Mental
Health
Pediatric
Anxiety
Anxiety
Depression
Depression
Anger
Anger
Illness Impact
Negative
Illness Impact Positive
Applied Cognition Concerns
Applied Cognition Abilities
PROMIS Current Social Health
Banks
Pediatric
Adult
Ability to Participate
in Roles & Activities
Social
Health
Satisfaction with Roles
& Activities
Companionship
Emotional Support
Informational
Support
Instrumental Support
Social Isolation
Peer Relationships
The PROMIS Metric
T Score
Mean = 50
SD = 10
Referenced to the US General
Population
An item bank is a large
collection of items measuring a
single domain.
Any and all items can be used to
provide a score for that domain.
0
50
100
Physical Functioning Item Bank
Item
1
Item
2
Item
3
Item
4
Item
5
Item
6
Item
7
Item
8
Are you able to get in and out of bed?
Are you able to stand without losing your balance for 1 minute?
Are you able to walk from one room to another?
Are you able to walk a block on flat ground?
Are you able to run or jog for two miles?
Are you able to run five miles?
Item
9
Item
n
Computer Adaptive Tests
3. Question
Questionnaire
with a high
precision AND a
wide range
2
2
1
1
-3
-2
-1
0
0
2
high
depression
1
3
2. Question
low
depression
20
Neuro-QOL
21
Broad Objectives of Neuro-QOL
RFP-NIH-NINDS-03-04
 Develop a core set of questions that cut across
chronic neurological disorders
 Develop supplemental questions that address
additional concerns of specific diseases, subgroups
of patients
 Create a publicly available, adaptable and
sustainable system allowing clinical researchers
access to a common item repository and CAT
• Will likely become the instrument of choice (at least
in research)
22
Expert Interviews (N = 44)
Rationale for Disease Choices
• Please list the 5 neurological
conditions for which you think it is
most important to measure patientreported quality of life.
• Please tell me your rationale for the
choice of each condition.
Target Disorders for Field
Testing
• Adult conditions
– Stroke
– Multiple Sclerosis
– Parkinson’s disease
– Epilepsy
– Amyotrophic Lateral Sclerosis
• Pediatric conditions
– Epilepsy
– Muscular dystrophies
Number 6 on the list was spinal cord injury
and number 8 was traumatic brain injury
Neuro-QOL
27
Disability Outcomes Research
• “measures of generic HRQOL have not
been thoroughly tested nor frequently
applied in broadly defined groups of
people with disabilities”
Andresen & Meyers. Arch Phys Med Rehabil Vol 81 Suppl 2,
Dec. 2000
Andresen & Meyers reported how the
SF-36 mobility items are irrelevant, and
perhaps, “offensive” to some
individuals with disabilities.
28
Neuro-QOL in Rehabilitation
• With the investment being made by NINDS,
Neuro-QOL may become the “expected” outcome
variable in clinical trails research in the US with
patients who have neurological disorders.
• However, some aspects of Neuro-QOL may not
be relevant nor specific enough (e.g., lacking
sensitivity and specificity) for individuals who
have experienced a traumatic injury.
• Will not cover core issues in SCI/TBI patients:
wheelchair mobility, headaches, secondary
complications, traumatic nature of
29
injury/disability.
An Unmet Need in SCI
• Despite rapid growth in the number of clinical
trials designed to restore functioning and treat
secondary complications following SCI,
existing outcomes measures are insufficient
– Narrow in focus, single domain of function
– Generic scales lack sensitivity and specificity for
SCI specific issues
– Generic measures can contain irrelevant or even
offensive items
SCI-QOL/SCI-FI Development
Projects
1. SCI - specific measurement scales
•
•
Spinal Cord Injury – Functional Index (SCI-FI)
Spinal Cord Injury – Quality of Life (SCI-QOL)
2. Worked closely with RIC and Northwestern
University Medical Social Sciences
3. Worked closely with the Spinal Cord Injury
Model System.
31
PROMIS/Neuro-QOL/SCI-FI/SCI-QOL
32
Developing the SCI-QOL/SCI-FI
Phase IV: Validation
of item banks
Phase II: Field
testing
Phase I: Developing
item pools
• Using focus groups of
persons with SCI to
identify key and
relevant domains of
QOL. Develop new
item banks.
855 persons with SCI
representing all levels
of injury
CAT = Computer Adaptive Test
Phase III: Calibration
and item bank
creation
• Using Item Response
Theory (IRT). Ensure
dimensionality and
calibrate within a
sample of persons
with SCI. Develop
CATs and Short Forms.
• Test new SCI-QOL
item banks (CATs and
Short Forms). Test
responsiveness and
convergent and
discriminant validity
SCI-FI Collaborative Project
• SCI-FI Combined Project: Combination of the
Kessler Foundation collaborative modular project and
the Boston University site specific project. Involves 6
Model System sites.
• Focus: developing an item bank specific to the
Activity Limitation (physical function dimension of
HRQOL).
• Ensure that measure is compatible with Neuro-QOL
and PROMIS systems – Functional Activity
Domains.
34
SCI-FI Collaborators
KEY:
Currently funded centers
35
SCI-FI:
Focus Group Methodology
• Semi-Structured Focus Group Guide
• Broad discussion of functional activities/activity
limitation:
– How has your SCI affected your day to day activities?
– What day to day activities are most important for
maintaining your independence?
• Specific topics (e.g. Mobility, Personal Care, Upper
Extremity function, ADLs):
– What components of [topic] are important to you?
– How do aspects of your [topic] impact your HRQOL?
Reference: Slavin MD, Kisala PA, Jette AM, Tulsky DS. (2010).
Developing a contemporary functional outcome measure for
spinal cord injury. Spinal Cord. 2010 Mar;48(3):262-7
36
SCI-FI:
Qualitative Analysis Methodology
• 18 focus groups across 6 collaborating
sites
• n=69 adults with traumatic SCI (6 groups with
paraplegia, 6 groups with tetraplegia)
• n=50 clinicians (physiatrists, physical therapists,
etc.; n=6 groups).
37
SCI-FI:
Qualitative Analysis Methodology
38
SCI-FI:
Item development
• Emerging themes not covered by items in existing
measures
• Using themes and comments from the transcripts,
a comprehensive item pool (k=743) was
constructed.
39
Item winnowing/refinement
• Qualitative Item Review (QIR)
– Expert Item Review - 743 items “winnowed”
down to the SCI-CAT “Calibration Version” of 328
items.
– Cognitive Interviews - with individuals with SCI
(n=14) were conducted and the item pool was
updated accordingly.
• Translatability Review
40
SCI-FI Item Development Process
41
SCI-FI Phase 2:
Calibration Field Testing - Goals
• 750 cases (125 per site at 6 sites)
• Balanced Sample will be Stratified by:
– Diagnosis (Paraplegia / Tetraplegia)
– Severity (Complete / Incomplete)
– Time Since Injury ( <1 yr / 1 yr– 3 yrs / >3 yrs)
• Collect injury level / AIS scores for all participants
42
SCI-FI Phase 2:
Calibration Field Testing - Results
Total n = 854
Paraplegia
Tetraplegia
Complete
Incomplete
Incomplete Paraplegia
Incomplete Tetraplegia
Complete Paraplegia
Incomplete Tetraplegia
<1
1-3
3+
389
465
446
408
161
247
228
218
264
237
353 43
Final Item Banks – Unidimensional CFA
Content Domain
Items CFI
TLI
RMSEA
Ambulation
39
0.999
0.999
0.039
Basic Mobility
54
0.969
0.968
0.081
Fine Motor
36
0.998
0.998
0.049
Self Care (F)
85
0.993
0.993
0.049
Self Care (M)
84
0.992
0.992
0.052
Wheelchair
56
0.932
0.929
0.063
Final Item Banks (k=275)
Content Domain
Items
Self-Care Function
90
Wheelchair Mobility
56
Basic Mobility
54
Ambulation
39
Fine Motor
36
45
Sample Items: Basic Mobility (k = 54)
Basic Mobility Items
Are you able to touch/hug a partner?
Are you able to stand supported in a standing frame?
Are you able to move your upper body while lying down in…
Are you able to sit in a car going around a corner, without…
Are you able to sit on a bench without a back, when you are…
When I am in bed, I can roll from my back to my side…
How much difficulty do you currently have picking up a…
When sitting, are you able to reach down to pick up a shoe…
Are you able to get out of bed into a chair?
I can move onto a shower chair….
How much difficulty do you currently have sitting down on a…
Are you able to get in and out of a car?
Are you able to move from lying down to sitting up (legs…
Are you able to sit on a bench without a back, when you can
Are you able to drive from a regular car seat?
Are you able to reach to take a box of cereal from the top…
Are you able to get up off the floor from lying on your back…
Are you able to stand without any support for 1 minute, for…
How much difficulty do you currently have standing up from…
Are you able to stand without any support for 5 minutes, for…
0
20
40
60
80
100
46
Sample Items: Self-Care
Function (k = 90)
Self Care Items
Are you able to chew and swallow your food (if someone…
I can scratch my face…
Are you able to wipe/blow your nose?
Are you able to brush your teeth?
I can use a spoon to eat soup…
I can dry my hair with a towel…
Are you able to brush your hair?
Are you able to floss your teeth?
Are you able to dress your upper body?
Are you able to use a long handled mirror to inspect your…
How much difficulty do you currently have opening…
Are you able to bathe yourself in your accessible shower in…
How much difficulty do you currently have chopping or…
After someone has helped with my clothes and setup, I can…
Are you able to take your shoes off?
Are you able to insert and remove a tampon?
How much difficulty do you currently have cleaning yourself…
Are you able to insert a suppository?
Are you able to dress your lower body?
Are you able to cut your toe nails?
0
20
40
60
80
100
47
SCI-FI Project Goal
• SCI-FI addressed physical functioning
• Traumatic SCI affects all areas of life
• Need for a valid, SCI-specific measure of
HRQOL
What about remaining domains of
HRQOL?
48
PROMIS/Neuro-QOL/SCI-FI/SCI-QOL
49
MEASURING HEALTH-RELATED
QUALITY OF LIFE (HRQOL) IN SCI:
Development of the SCI-QOL
NIDRR SCIMS
Collaborative
Project:
The SCI-FI
NIH R-01:
The SCI-QOL
Physical /
Medical
Health
Emotional
Health
Social
Participation
Physical
Function
50
SCI-QOL Collaborators
KEY:
Currently funded centers
51
Sampling
• Very diverse sample (N=757) across
stratification criteria
– Diagnosis
– Severity
– Time Since Injury
• Heterogeneous SCI sample similar to
demographics of SCIMS database
52
SCI-QOL Item Banks
SCI-QOL
Selected item banks
that have been equated
with PROMIS or NeuroQOL
Emotional Health
Domain
8 item banks
Depression, Anxiety and
Positive Affect and WB
Social Health
Domain
4 item banks
Social role participation
and Satisfaction, Stigma
Physical Health
Domain
7 item banks
Pain interference
Physical-Medical Domain:
Final Item Banks (n=757)
Model
Bladder Mgmt Difficulties
Bladder Complications
# Initial Items # Final Items
15
38
7
Bowel
52
26
Skin
30
18
Pain Interference
58
41
54
Bladder Management Difficulties:
Sample Items
 I worried that I would have a bladder accident.
 Bladder accidents limited my independence.
 I was bothered by urine leakage.
 Bladder accidents have disrupted my daily
activities.
 I felt my bladder management was under control
55
Bladder Complications: Sample
Items
 A UTI (urinary tract infection) limited my daily
activities.
 I had a urinary tract infection (UTI) that would not
go away.
 I had an increase in spasms because of a UTI
(urinary tract infection).
56
Bowel: Sample Items
 I worried about the odor associated with bowel
accidents.
 Bowel accidents limited my independence.
 I worried that I would have gas at an
inappropriate time
 I was embarrassed that I needed a bowel
program
 I spent a lot of time taking care of a bowel
accident
57
Skin: Sample Items
 I had a pressure ulcer.
 A pressure ulcer decreased the quality of my life.
 Pressure ulcers interfered with my ability to
work.
 I was bedridden due to a pressure ulcer.
 A problem with my skin limited my ability to do
things.
58
Emotional Domain:
Final Item Banks (n=716)
Bank
# Initial
Items
# Final
Items
Depression
35
28
Anxiety
38
26
Positive Affect and Well-Being
32
27
Resilience
32
23
Self-Esteem
30
20
Grief/Loss
20
18
Trauma
31
22
59
Grief/Loss: Sample Items
 I missed the activities I used to do
 I longed for the life I had before my injury
 I felt sad thinking about things I used to
enjoy
 I spent a lot of time thinking about what I
have lost since my injury
 I felt lost because of my injury
60
Resilience: Sample Items
 I took action to improve my life
 I used positive ways to cope with my injury
 I was able to recognize and overcome
challenges
 I could adapt easily to new situations
 I was driven to succeed in my life
61
Self-Esteem/Self-Evaluation:
Sample Items
 I felt good about myself
 I felt inferior to my friends or family
 I had high self-esteem
 I felt attractive
62
Trauma: Sample Items
 I was watchful for anything bad that might
happen
 I had upsetting thoughts about the event of
my injury
 I was nervous when something reminded
me of the accident
 I was frightened by sudden noises
 I was afraid in crowds
63
Social Participation Domain:
Item Banks (N=641)
•
•
•
•
Social Role Performance
Social Role Satisfaction
Independence / Autonomy
Stigma
64
SCI Banks
NEW
PROMIS
Neuro-QOL
Bladder Mgmt. Diff
Pain Interference
Positive Affect & WB
Bowel Mgmt. Diff
Depression
Stigma
Skin/Pressure Ulcers
Anxiety
Ability to Participate
Resilience
Self-Evaluation
Grief-Loss
Psych. Trauma
Basic Mobility
Wheelchair Mobility
Ambulation
Fine Motor
Self Care
Sat w Participation
PROMIS vs. SCI-QOL
• The beauty of PROMIS (and Neuro-QOL) is
its use across populations and directly
comparability of outcomes across studies,
domains and populations
• BUT do the SCI condition-specific calibration
of PROMIS item banks take us back to the
same problem that PROMIS tried to solve in
the first place?
Calibrations optimized for SCI
• SCI-QOL banks are calibrated within SCI sample
– Allows CAT to administer the most discriminating
items for persons with SCI
• Irrelevant and misfitting items within SCI sample have
been removed.
• What impact will this have on objectives of
PROMIS/Neuro-QOL?
• Transform the scores so they reflect the PROMIS metric
– If we could place the scores on the same metric then
we could interpret scores across conditions.
– We have developed a method to preserve the
PROMIS metric and thus ensure comparability of
scores
Developing Item Banks to Ensure
Consistency with PROMIS/Neuro-QOL
SCI-QOL
Domain
Item Bank
Linking
Instrument
# Items in Final
Calibration
# of Possible
Anchors
Emotional
Depression
PROMIS
22
15
Emotional
Anxiety
PROMIS
28
18
Emotional
Positive Affect and Well
Being
NeuroQoL
28
22
Phys-Med
Pain Interference
PROMIS
25
17
Social
Ability to Perform Social
Roles and Activities
NeuroQoL
27
27
Social
Social Role Satisfaction
NeuroQoL
35
34
Social
Stigma
NeuroQoL
23
18
New item banks: Bladder and bladder management difficulties, skin/pressure ulcers,
resilience, self-evaluation, grief, psychological trauma, basic mobility, wheelchair
mobility, ambulation, fine motor dexterity, and self-care
The Potential Problem
General Population
(Mean 50, SD 10)
Spinal Cord Injury
(Mean 50, SD 10)
= 77.9 Fahrenheit
(Multiply by 9, then divide by 5,
then add 32)
Problem = Same score, different metric!
Or push the button
A Linear “Transformation” Factor
General Population
(Mean 50, SD 10)
Spinal Cord Injury
(Mean 50, SD 10)
SCI-QOL Bank
Anxiety
Depression
Pos. Affect WB
Slope
Intercpt
0.926747 0.118130
1.030450 0.220082
0.804648 0.352426
Transformation to PROMIS Metric
PLINK (IP Link)
STUIRT
Domain
Method
Slope
Intercept
Slope
Intercept
Stocking-Lord
0.926747
0.118130
0.926611
0.118309
Stocking-Lord
1.030450
0.220082
1.030066
0.220473
Anxiety
Depression
•
•
•
•
All items are then transformed to PROMIS metric.
IRT Equating using Stocking-Lord
Change the location and dispersion of the sample.
Additive constant (intercept) and multiplicative
constant (slope) were used to shift and either shrink or
expand the distribution.
T-scores for common PROMIS /
Neuro-QoL item banks
• Reference group for t score on PROMIS/Neuro-QOL is
in relation to the general population
• T-score for SCI-QOL and TBI-QOL has been
transformed to be on the same metric as PROMIS or
Neuro-QOL
Scoring Before and After Transformation
T-score Before
Transformation
(Calibration Parameters)
Bank
Fatigue
N
590
T-score After Transformation
(Transformed Parameters)
Mean
S.D.
Mean
S.D.
45.52
9.47
50.11
11.26
Landing on the Same Metric
SCI-QOL Bank
N
T-score Before
Transformation
T-score After
Transformation
Mean
S.D.
Mean
S.D.
Anxiety
716
49.69
9.60
50.75
9.12
Depression
716
48.59
9.29
50.66
9.62
Positive Affect and WB
717
51.15
9.61
54.47
7.92
Pain Behavior
757
49.94
9.60
53.50
9.30
Pain Interference
757
48.65
9.26
53.10
9.89
Stigma
611
49.84
9.66
53.18
6.69
Ability to Perform Social
Roles and Activities
641
50.46
9.76
45.42
6.57
Social Role Satisfaction
641
50.72
9.77
45.44
5.59
T-Score: Mean = 50, SD = 10
Conclusions on Transformation
• This is one way to address the problem of a
CAT choosing the wrong items for a
population
• We have made sure that the PROMIS and the
Neuro-QOL CATs are the best they can be for
the SCI population
• We have then applied a linear transformation
to preserve the PROMIS metric
The Validation and
Responsiveness of the SCI-FI
and SCI-QOL Item Banks
77
Phase IV:
Reliability, Validity and Responsiveness
– Reliability: Internal Consistency (coefficient alpha)
and 1-2 week Test-Retest Reliability
– Validity: Concurrent and Discriminant Validity
(Correlations with criteria measures).
– Responsiveness to Change: Natural recovery
78
Study 1. SCI-FI Validation
Study Overview
• Community sample of 269 adults with traumatic
SCI
• Participants completed SCI-FI CATs, SCI-FI Short
Forms, and the Self Report Functional Measure
(SRFM; Hoenig 1999)
– Basic Mobility, Self Care, Fine Motor,
– Wheelchair (if Applicable), Ambulation (if Applicable)
• Test-Retest design:
– Baseline Assessment
– Repeated Assessment @ 7-14 days
Methods & Data Collection
• Traumatic SCI
• Stratified sample
• Individuals who are
community dwelling
Baseline
Two weeks
• Interview format
• May be completed in person or
via telephone
• Complete CATs, short forms, and
self-report FIM
Study 1. SCI-FI Validation
Participant Demographics
n=269
Age
Gender
Ethnic Background
Racial Background
42.9
(SD 15.5)
79.1% male
20.9% female
87.8% non-Hispanic
10.6% Hispanic
71.1% Caucasian
17.2% Black or African American
1.5% Asian
0.9% Amer. Indian/Alaska Native or Native Hawaiian/Pacific Islander
1.5% More than one race
6.7% “Other”
Education
38.4% High school or less
Some
college
Values in red represent the33.5%
specific
SCI-QOL
item bank and legacy matches
28.1% Bachelor’s degree or more
Study 1. SCI-FI Validation
Participant Injury Characteristics
n=269
Diagnosis/Severity
Time Since Injury
Cause of Injury
24.2% Paraplegia Complete
21.6% Paraplegia Incomplete
14.1% Tetraplegia Complete
39.0% Tetraplegia Incomplete
23.0% < 1 year
29.0% 1 – 3 years
48.0% >3 years
28.6% MVA
19.7% Fall
10.8% Diving
9.3% Other Sports
11.9% GSW/Violence
7.4% Medical/Surgical Accident
12.3% Other
Study 1. SCI-FI Validation
Test-Retest Correlations
T1=Baseline, T2=7-14 day follow up
SCI-FI Item Banks
**p<.01
Test-Retest Correlation
Coefficients
Basic Mobility (n=255)
.91**
Self Care (n=254)
.94**
Fine Motor (n=255)
.93**
Wheelchair (n=204)
.91**
Ambulation (n=92)
.91**
Study 1. SCI-FI Validation Test-Retest Correlations
SCI-FI Item Banks
**p<.01
Test-Retest
Correlation
Coefficients
Basic Mobility (n=255)
.91**
Basic Mobility SF (n=247)
.95**
Self Care (n=254)
.94**
Self Care SF (n=252)
.96**
Fine Motor (n=255)
.93**
Fine Motor SF (n=254)
.95**
Wheelchair (n=204)
.91**
Wheelchair Manual SF (n=122)
.89**
Wheelchair Power SF (n=94)
.90**
Ambulation SF (n=87)
.95**
Study 1. SCI-FI Validation
Correlations with Self Report FIM at Baseline Assessment
SCI-FI Item Banks
**p<.01
Self-Report FIM
Basic Mobility (n=263)
.86**
Self Care (n=263)
.86**
Fine Motor (n=263)
.79**
Wheelchair (n=212)
.83**
Ambulation (n=91)
.69**
Study 1. SCI-FI Validation
Short Form Correlations with Self Report FIM
SCI-FI Item Banks
Self-Report FIM
Basic Mobility (n=263)
.86**
Basic Mobility SF (n=260)
.89**
Self Care (n=263)
.86**
Self Care (n=263)
.88**
Fine Motor (n=263)
.79**
Fine Motor (n=262)
.81**
Wheelchair (n=212)
.83**
Wheelchair Manual SF (n=135)
.73**
Wheelchair Power SF (n=107)
.81**
Ambulation (n=91)
.69**
Ambulation SF (n=90)
.69**
Study 2
SCI-QOL/SCI-FI Validation: Community Dwelling
Sample
• Community sample of 250 adults with traumatic
SCI
• Includes SCI-FI CATs, SCI-FI Short Forms, and the
Self Report Functional Measure (SRFM; Hoenig
1999) in addition to SCI-QOL CATs, short forms,
and an extensive set of legacy measures
• Test-Retest design:
–
–
–
–
T1: Baseline Assessment
T2: 7-14 days
T3: 3 months
T4: 6 months
Study 2. SCI-QOL Validation: Community
Dwelling Sample
Participant Demographics
n=250
Age
46.1
(SD 14.7)
Gender
74% male
26% female
Ethnic Background
92% non-Hispanic
8% Hispanic
Racial Background
74% Caucasian
16% Black or African American
2% Asian
2% More than one race
6% “Other” or not provided
Education
28%specific
High school
or less
Values in red represent the
SCI-QOL
item bank and legacy matches
42% Some college
30% Bachelor’s degree or more
Study 2. SCI-QOL Validation: Community
Dwelling Sample
Participant Injury Characteristics
n=250
Diagnosis/Severity
25% Paraplegia Complete
21% Paraplegia Incomplete
21% Tetraplegia Complete
32% Tetraplegia Incomplete
Time Since Injury
Mean 10.3 (SD 15.6)
14% between 4 and 12 months
15% between 1 and 2 years
71% > 2 years
Cause of Injury
39% MVA
24% Fall
8% Diving
4% Other Sports
10% GSW/Violence
7% Medical/Surgical Accident
8% Other
Study 2
SCI-QOL/SCI-FI Validation: Community Sample
Test-Retest Correlations
T1=Baseline, T2=7-14 day follow up
SCI-FI Item Banks
**p<.01
Correlation between
T1 and T2
Basic Mobility (n=227)
.92**
Self Care (n=227)
.95**
Fine Motor (n=227)
.94**
Wheelchair (n=180)
.92**
Study 2
SCI-QOL/SCI-FI Validation: Community Sample
Correlations with Legacy Measure: Self Report FIM
at Baseline Assessment
SCI-FI Item Banks
**p<.01
Self-Report FIM
Basic Mobility (n=77)
.84**
Self Care (n=77)
.88**
Fine Motor (n=77)
.83**
Wheelchair (n=65)
.79**
SCI-FI Reliability and Validity
• Both the SCI-FI CATs and Short forms are reliable
and stable (IRT based estimates and test-retest).
• Lower reliability at ends (especially high end) for
some banks especially those that have gaps in the
item pool/item banks. Similar problem in legacy
measures.
• High correlations coefficients between SCI-FI
CATs and Short Forms and the self report FIM –
Criterion Related Validity
Selected SCI-QOL & Legacy Measures
SCIQOL Item Banks
Legacy Measure(s)
Depression
Patient Health Questionnaire, Depression (PHQ-9)
Anxiety
Generalized Anxiety Disorder Scale (GAD-7)
Positive Affect and Well Being
Meaning of Life Questionnaire (MLQ; Presence
subscale) and Satisfaction with Life Scale (SWLS)
Pain
The Brief Pain Inventory (BPI), short form
Ability to Perform in Social
Roles and Activities
Social Role Satisfaction
Craig Handicap Assessment and Reporting Technique
(CHART) and Participation Assessment With
Recombined Tools–Objective (PART-O)
Stigma
No suitable legacy measure
Reliability of Selected SCI-QOL CATs
Baseline to Two Weeks
All test re-test reliability coefficients exceeded 0.7 suggesting excellent reliability
SCI-QOL Item Bank (N = 248)
Test-Retest Reliability
Coeff.
Depression
0.81**
Anxiety
0.81**
Positive Affect and Well Being
0.77**
Pain
0.84**
Ability to Perform in Social Roles and Activities
0.74**
Social Role Satisfaction
0.78**
Stigma
0.81**
Selected SCI-QOL & Legacy Measures
Correlations with Legacy Measures in Individuals with Chronic (>4 mos) SCI, N = 89
SCI-QOL Item
Banks
Legacy Measures
PHQ-9
GAD-7
MLQ
SWLS
BPI
CHART
PART-O
Total
Total
Depression
0.70**
0.59**
-0.40**
-0.61**
0.47**
-0.26*
-0.30*
Anxiety
0.59**
0.70**
-0.24*
-0.50**
0.45**
-0.16
-0.06
Pos. Affect/WB
-0.54**
-0.42**
0.66**
0.60**
-0.18
0.09
0.23
Pain Interference
0.61**
0.54**
-0.04
-0.28**
0.71**
-0.09
-0.04
Ability to Perf.
Social Roles
-0.54**
-0.46**
0.13
0.59**
-0.38*
0.44**
0.47**
Social Role Satisf.
-0.60**
-0.49**
0.34**
0.63**
-0.33**
0.30*
0.36**
Values in green represent the specific SCI-QOL item bank and legacy matches
*p<0.05; **p<0.001
Gathering Evidence:
Reliability, Validity and Responsiveness
– Reliability: Internal Consistency (coefficient alpha)
and 1-2 week Test-Retest Reliability
– Validity: Concurrent and Discriminant Validity
(correlations with criteria measures).
– Responsiveness to Change: Natural recovery
96
What is Responsiveness?
• Accurate detection of change when it has
occurred (DeBruin et al, 1997)
• Ability to detect change, specifically
important changes in the way patients are
feeling, even if those changes are small (Guyatt
et al., 1997)
Measuring Responsiveness
Effect size (ES) =
𝑴𝒆𝒂𝒏 𝒄𝒉𝒂𝒏𝒈𝒆
𝑺𝑫 𝒂𝒕 𝒃𝒂𝒔𝒆𝒍𝒊𝒏𝒆
• Guidelines for interpreting Effect Size:
– Small: 0.2-0.3
– Medium: ~0.5
– Large: >0.8
SCI-QOL Validation & Responsiveness
Study: Objectives
• Examine the responsiveness of the SCI-FI/SCI-QOL to
clinically meaningful change.
• Use “Natural Recovery” as a proxy for an intervention
• Compare the sensitivity of the SCI-FI/SCI-QOL to that
of “legacy measures.”
99
Longitudinal Study Design
Recent SCI
(<4 mos)
100
Longitudinal Study Design
Recent SCI
(<4 mos)
101
Longitudinal Study Design
Recent SCI
(<4 mos)
102
Longitudinal Study Design
Recent SCI
(<4 mos)
103
Longitudinal Study Design
Recent SCI
(<4 mos)
104
Longitudinal Study Design
Recent SCI
(<4 mos)
105
Predicted Changes in Individuals with SCI
• PHYSICAL FUNCTIONING:
SCI-FI physical function variables and Self-Report Functional
Measure will show large effect sizes (e.g., ≥ 0.8) over the initial
3-6 months.
• EMOTIONAL FUNCTIONING:
SCI-QOL Depression/Anxiety and PHQ-9, GAD-7, and SWLS
variables will not show a consistent pattern of change over
short intervals. Responsiveness will likely be detected between
the longest intervals (e.g., 2 years) and baseline.
• MODERATORS:
The rate/pattern of change of SCI-QOL emotional functioning
variables may be moderated by resilience.
109
Longitudinal Study Design
• Measure individuals with SCI shortly following injury
and reassess them 3, 6, and 12 months later.
• Collect criterion variables (PHQ-9, GAD-7, SWLS, Selfreport Functional Measure)
• Assessments can be performed rapidly using CAT
technology; a wide variety of measures can be
collected.
110
Included Measures
SCI-FI/SCI-QOL
CATs
• Basic Mobility
• Fine Motor
• Self Care
• Wheelchair Mobility
• Anxiety
• Depression
• Positive Affect & Well-Being
• Resilience
Legacy
Measures
• SRFM (Self Report Functional
Measure – Hoenig et al)
• GAD-7
• PHQ-9
• SWLS
111
Longitudinal Study Design
• All data collected in interview format (in person or via
phone)
• Assessments were streamlined using CAT technology –
interviewers entered item responses directly into the
Assessment CenterSM platform
• Goal for each center was to assess all inpatients at
baseline and continue to follow participants over time
112
Demographic Information: Baseline Sample
N=206
Age
41.8 (16.5)
Time Since Injury (Days)
68.4 (26.1)
Gender
Male
Female
83.1%
16.9%
Ethnicity
Hispanic
Non-Hispanic
Not Reported (NR)
6.8%
90.8%
2.5%
Race
Caucasian
Black or African American
Asian
Other or NR
81.6%
7.8%
1.9%
8.7%
Diagnosis
Paraplegia
Tetraplegia
41.7%
57.8%
Completeness
Complete
Incomplete
45.7%
54.3%
113
SCI-FI: PHYSICAL FUNCTIONING
114
Descriptive Statistics: SCI-FI Measures
and Self Report Functional Measure
Baseline
Measures
Mean
SD
Basic
Mobility
48.4
8.98
Fine Motor
3 Months
n
Mean
SD
206 52.8**
48.7
Self Care
6 Months
Mean
SD
n
9.31
130 53.9**
9.69
117
10.88 206 52.8**
9.30
128 51.8**
9.27
117
48.1
9.99
206 52.2**
9.28
129 52.1**
8.90
117
Wheelchair
49.0
8.20
184 51.4**
9.15
102 51.2**
8.60
84
SRFM†
30.6
11.25 201 36.8**
12.29 128 36.1** 12.41
113
**p<.01 when compared to baseline score
n
Effect Size: SCI-FI Basic Mobility vs. SRFM
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
Effect Size: SCI-FI Self Care vs. SRFM
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
Effect Size: SCI-FI Fine Motor
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
Effect Size: SCI-FI Wheelchair
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
SCI-QOL: EMOTIONAL HEALTH
120
Descriptive Statistics: SCI-QOL
Measures and Legacy Measures
Baseline
Measures
3 Months
6 Months
Mean
SD
n
Mean
SD
n
Mean
SD
n
SCI-QOL Anxiety
52.5
9.53
205
52.9
8.31
129
51.8
9.58
117
SCI-QOL
Depression
50.6
8.10
206
51.7*
8.44
128
51.9*
8.58
117
SCI-QOL PAWB
54.6
7.88
206
54.1
7.39
129
54.1
8.14
117
SCI-QOL
Resilience
51.2
9.30
206
51.4
9.11
128
50.9
9.60
117
PHQ-9
5.1
4.87
202
4.9
5.38
128
5.0
5.13
115
GAD-7
3.5
4.56
203
3.3
4.16
128
3.5
4.36
115
SWLS
20.0
7.58
203
19.5
7.98
129
19.7
8.41
115
*p<.05 when compared to baseline score
Effect Size: SCI-QOL Anxiety vs. GAD-7
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
122
Change in ANXIETY:
6 Months - Baseline
# SD Units
% Sample
< -2.5
0.9%
-2.5 to -2.0 0.9%
-2.0 to -1.5 5.4%
-1.5 to -1.0 6.3%
SYMPTOMS
WORSENING
-1.0 to -0.5 13.4%
-0.5 to 0.5
51.8%
0.5 to 1.0
10.7%
1.0 to 1.5
6.3%
1.5 to 2.0
1.8%
2.0 to 2.5
0.9%
> 2.5
1.8%
123
Effect Size: SCI-QOL Depression vs. PHQ-9
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
124
Change in DEPRESSION:
6 Months - Baseline
# SD Units
% Sample
-2.0 to -1.5 2.7%
-1.5 to -1.0 1.8%
-1.0 to -0.5 14.3%
SYMPTOMS
WORSENING
-0.5 to 0.5
56.3%
0.5 to 1.0
16.1%
1.0 to 1.5
4.5%
1.5 to 2.0
1.8%
2.0 to 2.5
1.8%
> 2.5
0.9%
125
Effect Size: SCI-QOL PAWB vs. SWLS
Effect Size: Small: 0.2-0.3, Medium: ~0.5,Large: >0.8
126
Resilience as a Moderator:
Predicting Scores & Explaining Variance
Linear Regression: Resilience at Baseline as Independent Variable/Predictor
Dependent
Variable
(Score at 6
Months)
β
t (116)
R2
F (1, 115)
Anxiety
-0.40
-4.65***
0.16
21.64***
Depression
-0.49
-6.09***
0.24
37.03***
PAWB
0.53
6.74***
0.28
45.40***
*p<.05; **p<.01, ***p<.001
127
Evidence Suggests…
–In a sample of individuals with very
recent injuries (< 4 months post injury):
• Physical functioning improves significantly over the
first 6 months (even the first 3 months) post injury.
• SCI-FI and SRFM show improvement. SCI-FI Fine
Motor shows change in individuals with tetraplegia
(not in paraplegia)
• Very little change in Emotional variables over 3 or
6 months.
• Emotional variables are moderated by level of
resilience
128
What’s Next?
– Monitor change over extended time periods – 1 year
and 2 year follow-up data
• Will psychosocial variables show improvement from baseline
as time since injury increases?
– Examine other SCI-QOL banks (social participation,
pain, bowel and bladder management difficulties,
other emotional item banks).
– Utilize SCI-QOL variables in physical & emotional
intervention studies
– Examine additional mediating or moderating variables
(e.g., injury characteristics, gender, race, time since
injury)
129
SCI-FI2
(PI: Alan Jette)
• 5 Item Banks – Can Administer as CAT or
Short Form
• Ask people to respond based upon their
ability without any assistive device –
Capacity
• We are developing 4 new item banks
asking people to respond based upon their
ability to perform task with what ever
assistive device - Performance
130
PROMIS/Neuro-QOL/TBI-QOL
131
SCI-QOL and TBI-QOL
Domains/Subdomains
Pain Interference
Pain Behavior
Headache Pain
Fatigue
Skin / Pressure
Physical-Medical
Ulcers
Health
Bladder
Complications
Bladder
Management
Bowel
Management
SCI-QOL
Final
TBI-QOL
Sample Item
Final
25
7
---
40
10
13
73
How much did pain interfere with your family
life?
When I was in pain I became irritable.
I had constant pain from headaches.
How often were you too tired to enjoy life?
14
--
I was bedridden due to a pressure ulcer.
5
--
Bladder accidents limited my independence.
15
--
Bladder accidents limited my independence.
26
--
I was worried I would have a bowel accident.
132
SCI-QOL and TBI-QOL
Domains/Subdomains
Physical
Functioning
SCI-QOL
Final
TBI-QOL
Sample Item
Final
Mobility
Ambulation
-39
32
--
Basic Mobility
54
--
Wheelchair Mobilty
Upper Extremity
Self-Care
Fine Motor
54
-90
36
-33
---
Are you able to get on and off the toilet?
Are you able to walk for 5 minutes outside?
When you are in bed, are you able to turn
your lower body?
From the floor, I can get into my manual
wheelchair.
Are you able to water a house plant?
Are you able to take your socks off?
Are you able to pick up a piece of paper?
133
SCI-QOL and TBI-QOL
Domains/Subdomains
Mental Health
(Emotional and
Cognitive)
SCI-QOL
Final
TBI-QOL
Sample Item
Final
Loss / Grief
Resilience
Self-Esteem
Depression
Positive Affect &
Well-Being
Anxiety/Fear
17
21
20
28
17
27
13
28
I longed for the life I had before my injury.
I tried to see the positive side of things.
I was comfortable with myself.
I felt that nothing was interesting.
27
25
32
28
Stigma
Anger
Emotional &
Behav. Dyscontrol
23
--
27
35
--
26
Trauma
Cognition General Concerns
19
--
--
39
Executive Function
Communication
---
37
31
I was living life to the fullest.
I had trouble relaxing.
Because of my injury, strangers tended to
stare at me.
I lost my temper.
Other people got annoyed because I was so
talkative.
I was watchful for anything bad that might
happen.
How much difficulty do you currently have
remembering directions?
How much difficulty you currently have
adapting to a new situation?
How much difficulty do you have speaking?
134
SCI-QOL and TBI-QOL
Domains/Subdomains
Social
Participation
SCI-QOL
Final
TBI-QOL
Sample Item
Final
Ability to Participate in
Social Roles and
Activities (SRA)
27
45
Satisfaction with SRA
Independence
35
8
42
13
I am able to perform my daily routines.
I am satisfied with my ability to do things for
my friends.
I am able to ask for help if I need it.
135
Deployment-Related TBI-QOL
Collaborative Project
• PIs: Groer and Vanderploeg
• Development of TBI-QOL for Veterans of current
conflicts (OEF/OIF)
• Rehabilitation Research and Development (RR & D)
VA grant
• 4 DVBIC/Polytrauma Rehabilitation Centers.
Tampa, Minneapolis, Richmond, and Palo Alto.
• Focus Groups with Vets. Develop additional item
banks – Guilt, Military related Loss, PTSD.
• Field test new items and develop CATs
136
www.assessmentcenter.net
137