Transcript Document

Template methodology for
multidimensional ranking: key
provisions, approbation outcomes and
potential for application
Zavarykina L. (NTF)
IREG Forum on Universities Rankings: Methodologies under Scrutiny
16-17 May 2013
Warsaw
TEMPLATE METHODOLOGY FOR MULTIDIMENSIONAL
RANKING
Template methodology for multidimensional ranking – basis
for a national approach to HEIs’ assessment with due account
of the national higher education system diversity.




Multidimensional ranking of the Russian HEIs – an approach
based on:
IREG audit criteria
experience of global and national rankings construction
quantitative indicators
rejection of applying an aggregate indicator
2
TEMPLATE METHODOLOGY FOR
MULTIDIMENSIONAL RANKING: KEY PRINCIPLES
Methodology should:





provide reliable data on a university’s performance and its position among
other HEIs’
take into account the diversity of the Russian higher education institutions
and their functions
support users of educational services providing friendly and easy-to-use
information on various educational institutions and their services
facilitate quality enhancement and competitiveness of the Russian higher
education institutions
facilitate integration of the Russian higher education institutions into the
global education and research area as their position in rankings is an
important signal of competitiveness
3
TEMPLATE METHODOLOGY FOR MULTIDIMENSIONAL
RANKING
Mission
Provide reliable data on Russian HEIs’ performance on the basis of multidimensional ranking to
satisfy information needs of various users groups with due account of the Russian higher education
system diversity
Goal
Forming a basis for a national approach to HEIs’ assessment that takes into account diversity of the
national higher education system and contributes to:
1)
comprehensive assessment of education quality and increasing competitiveness of the Russian
higher education
2)
integration of the Russian HEIs into the global education area
Objectives:




Developing a tool for transparency and external assessment of HEIs quality in Russia
Developing a database of the Russian higher education system (current state and
development trends) taking into account its diversity with a possibility of creating HEIs
rankings and ratings on specific indicators
Assessment of higher education institutions on several functions
Contributing to the Russian higher education system development through creating an
information and analytical basis for benchmarking (best practices identification) and
facilitating demand for higher education services in the country
4
TEMPLATE METHODOLOGY FOR MULTIDIMENSIONAL RANKING
Target audience:




Prospective students and their
parents
Government (central and local)
Employers and other labour
market actors
Academic community
(researchers, lecturers)
5
TEMPLATE METHODOLOGY FOR MULTIDIMENSIONAL RANKING
Ranking objects:

State and private institutions

Leading universities

Classical universities

Engineering and technical HEIs

Humanitarian and pedagogical HEIs

Economics and law HEIs

Medical HEIs
6
TEMPLATE METHODOLOGY FOR MULTIDIMENSIONAL RANKING
HEIs’ functions to assess:





Research
Teaching and learning
Internationalisation
Knowledge transfer
Engagements with regional stakeholders
7
TEMPLATE METHODOLOGY LIST OF INDICATORS
RESEARCH (15)













Ratio of expenditure on research to the total institution expenditure
Ratio of research income to the total institution’s income
Number of citations per academic staff (Scopus)
Number of publications per academic staff (Scopus)
Number of citations per publication (Scopus)
Number of citations per academic staff (Russian e-library)
Number of publications per academic staff (Russian e-library)
Number of citations per publication (Russian e-library)
Number of grants awarded (Russian Humanitarian Fund, Russian Foundation
for Basic Research)
Total sum of grants awarded (Russian Humanitarian Fund, Russian
Foundation for Basic Research)
Ratio of bachelor full-time students participated in research to the total
number of bachelor students

Number of citations per academic staff (full-time equivalent) (Web of Science)
Number of publications per academic staff (full-time equivalent) (Web of
Science)
Number of citations per publication (Web of Science)

Ratio of academic staff with PhD degrees to the total number of academic staff
8
TEMPLATE METHODOLOGY LIST OF INDICATORS
TEACHING AND LEARNING (18)








Average Unified State Examination score of admitted students
Ratio of students to academic staff
Ratio of students enrolled in master programmes to students enrolled in bachelor programmes
Proportion of graduates entered PhD programmes
Number of PhD students
Number of academic staff who defended their doctoral thesis
Expenditure on facilities and infrastructure for education provision
Funds for provision education services (federal funds allocated for students’ education and
training)






Proportion of internationally accredited education programmes
Proportion of programmes enrolling students with high Unified State Examination scores
Proportion of applicants who won national education Olympics
Proportion of students awarded prestigious scholarships
Ratio of graduates from other universities enrolled in master programmes to the total number of
students enrolled in master programmes
Proportion of academic staff under 35 y.o. who won competitive national awards to the total
number of academic staff under 35 y.o.
Proportion of academic staff who are members or associate members of the Russian Academy of
Sciences
Proportion of academic staff who won prestigious international and national awards


Proportion of graduates who find employment by specialization within 1 year after graduation
Ratio of PhD students defended their thesis within 1 year after completion their education


9
TEMPLATE METHODOLOGY LIST OF INDICATORS
INTERNATIONALISATION (12)












Proportion of international students to the total number of students
Number of international academic staff
Number of international research grants awarded
Total sum of international research grants awarded
Ratio of income from international sources (teaching, research, contracts with
international organisations) to the total institution income
Proportion of students (full-time equivalent) studied abroad
Proportion of PhD students participating in study placements abroad
Proportion of academic staff (full-time equivalent) invited as lecturers by
international universities
Proportion of academic staff (full-time equivalent) with MSc/PhD degree from
international universities
Proportion of education programmes developed in collaboration with international
partners
Proportion of students taking programmes developed in collaboration with
international partners
Ratio of teaching load of international academic staff to the total teaching load of
academic staff (full-time equivalent)
10
TEMPLATE METHODOLOGY LIST OF INDICATORS
KNOWLEDGE TRANSFER(5)





Proportion of funding received from other sources than federal budget
Income from intellectual property products
Number of education programmes implemented by an institution at the request of third
party organisations
Number of specialists from third party organisations who took professional development
courses
Number of intellectual property items put on accounting balance sheets
ENGAGEMENT WITH REGIONAL STAKEHOLDERS (4)



Proportion of income from local/regional sources
Number of research contracts with regional partners
Percentage of students in internships in local enterprises

Percentage of students working in the region
11
SAMPLING STRUCTURE FOR METHODOLOGY
APPROBATION
Federal district
Classical
universitie
s
Technical
HEIs
Pedagogical
HEIs
HEIs with
Economics/Law
programmes
Medical HEIs
1
Far-Eastern
6
2
2
Volga
9
12
3
2
1
3
North-Western
7
8
3
3
1
4
North-Caucasian
5
1
1
5
Siberian
5
7
4
6
Ural
3
3
7
Central
9
17
8
South
3
4
Total number of
invitees
47
54
Total number of
participants
39
39
Agricultural
HEIs
1
9
2
29
22
7
3
2
1
1
8
TOTAL
22
7
5
2
1
42
2
1
19
17
7
4
148
12
9
4
0
103
10
12
DATABASE CONSTRUCTION

data collection
tool approbation
 data verification

 data

mathematical and statistical processing, constructing
tools for HEIs assessment
 data

proceeding
analysis
grouping and comparative analysis
13
DATA VERIFICATION
 Data verification on its integrity and reliability
 Identification of sub-indicators with unavailable data
 Data absence
influences the final distribution of the basic indicators’ scores
for each HEI
 reduces the total HEI’s score in the ranking
 Real and nominal data absence:
 real absence: universities do not carry out relevant activities
and do not have outcomes reflected by a sub-indicator
 nominal absence: universities do not collect data on a subindicator
 in both cases «0» score is assigned to a sub-indicator

14
DATA RESTORATION
 Step 1: missing data is restored from open sources (f.e., Russian elibrary, Scopus, Web of Science)
 Step 2: if data restoration from open sources is impossible, a
special list of indicators with missing data is compiled for each
university
 Step 3: individual consultations with a university on its missing
data
 Step 4: if it is not possible for a university to provide data (nominal
data absence), missing data can be substituted by estimated data
Options for missing data substitution:
 minimal value within HEIs category,
 minimal value + 1 standard deviation,
 average value – 1 standard deviation,
The 4th step is used in some rankings methodologies but it is
criticized for mispresentation of rankings outcomes. Missing data
substitution was not used within the methodology approbation
15
APPROACH TO HEIs GROUPING
Grouping by interval values - statistical method of data grouping. It is used to
identify group of HEIs with high, middle and low performance
i – interval length,
X max and Х min - max and min of a
grouping characteristic,
n – number of groups
The sample was divided into 3 groups for research purposes.
3 groups of HEIs were identified based on the approbation outcomes:
Leaders
Competitors
Catching up
16
APPROBATION OUTCOMES: 6 TYPES OF
RANKINGS
1. Overall ranking (n103);
2. Overall ranking on 13 indicators of global rankings
(n103);
3. Ranking of separate HEIs categories (n8, n28+1,
n31, n35);
4. Ranking of separate HEIs categories on five
functions (n8, n28+1, n31, n35);
5. Ranking of separate HEIs categories on 13
indicators of global rankings (n8, n28+1, n31, n35)
6. Overall rating (n103)
17
APPROBATION OUTCOMES: 3 GROUPS OF HIGHER
EDUCATION INSTITUTIONS
18
APPROBATION OUTCOMES: RANKINGS ON 5 HEIs
FUNCTIONS
19
13 INDICATORS OF GLOBAL RANKINGS
Number of publications
per academic staff (Web of Science)
Number of citations per paper
(Web of Science)
Number of citations
per academic staff
(Web of Science)
THE,
Leiden
Number of citations per academic staff
(Scopus)
QS
THE,
Leiden
Number of publications
per academic staff (Scopus)
QS
THE,
Leiden
Research income
THE
THE
Income from
regional /local
sources
THE
THE
Proportion of funding
received from other
sources than federal
budget
Ratio of PhD students defended
their thesis (within 1 year after
completing education; within 2
years after completing education)
THE, QS
THE, QS
International staff
THE, QS
Staff to student ratio
International students ratio
RESEARCH
TEACHING
INTERNATIONALISATION
KNOWLEDGE TRANSFER
APPROBATION OUTCOMES: RANKING ON 13 INDICATORS OF
GLOBAL RANKINGS
3 HEIs
Challenging indicators:
• Number of citations per
academic staff (Scopus)
• Number of citations per
paper (Scopus)
• Number of citations per
paper (Web of Science)
• Number of international
academic staff
16 HEIs
Satisfying performance:
• Funding from sources other
than federal budget
• Funding from regional/local
sources
• Proportion of students
defended their thesis within 2
years after completion their
education
84 HEIs
Leaders
Competitors
Catching up
21
FACTOR ANALYSIS
Conducted to:
•
identify key factors that explain correlation between
indicators
•
improve the indicators’ set used in the methodology for
multidimensional ranking
Method of key factors identification:
•
stepwise comparison of indicators to identify correlation
between them (Pearson and Spearman correlation matrix)
Outcome:
•
correlation matrix
•
adjusted the set of indicators with a high level of correlation
22
Total sum of scores
Reducing the number of indicators
Линия тренда
Trend line
indicators
23
FACTOR ANALYSIS OUTCOMES: WITHDRAWAL OF
INDICATORS
Indicators with high scores and large distance between them
Percentage of students working in the region (82,4)
Ratio of academic staff with PhD degrees to the total number of academic staff (76,0)
Proportion of graduates who find employment by specialization within 1 year after
graduation (59,6)
Ratio of PhD students defended their thesis within 1 year after completion their
education (42,5)
Indicators with low scores and large distance between them
Ratio of teaching load of international academic staff to the total teaching load of
academic staff (full-time equivalent) (1,3)
Number of intellectual property items put on accounting balance sheets (1,1)
THE MULTIDIMENSIONAL RANKING METHODOLOGY
TRANSFORMATION
5
dimensions
for ranking
95 sub-indicators
54 indicators
Ranking on
special
indicators
Factor
analysis
48 indicators
Express monitoring
3 ranking dimensions:
research potential,
education quality, third
role
Ranking on special
indicators (13
indicators of global
rankings, “excellence
indicators”
25
SPECIAL RANKING ON “EXCELLENCE INDICATORS” FOR LEADING UNIVERSITIES
Excellence indicators – highly correlating indicators that influence HEIs positions in the ranking and define their
leading and competitive features
№
1
Indicators
Research income
Code
B12
2
3
4
5
6
7
8
9
10
11
12
13
14
Number of citations per academic staff (Russian e-library)
Number of citations per academic staff (Scopus)
Number of citations per academic staff (Web of Science)
Number of citations per publication (Russian e-library)
Number of citations per publication (Scopus)
Number of citations per publication (Web of Science)
Number of publications per academic staff (Russian e-library)
Number of publications per academic staff (Scopus)
Number of publications per academic staff (Web of Science)
Number of international research grants awarded
Number of Russian grants awarded
Total sum of Russian research grants awarded
Total sum of international research grants awarded
Ratio of bachelor full-time students participated in research to the total number of
bachelor students
Proportion of students awarded prestigious scholarships
Proportion of graduates entered PhD programmes
B3
B4
B5
B6
B7
B8
B9
B10
B11
B13
D2
B14
D3
15
16
17
B15
C4
C6
26
HEIs ASSESSMENT ON “EXCELLENCE INDICATORS”: SCOPE FOR LOOKING
FORWARD
Leadership based on a limited
number of competitive features.
Risk of loosing leadership
Balanced performance on the five
functions. Leadership can be ensured
by focusing on excellence indicators.
POTENTIAL FOR THE METHODOLOGY
APPLICATION
The methodology for multidimensional ranking should be applied
for :
• a comprehensive analysis and assessment of the HEIs in their
diversity enabling comparison and benchmarking, and
enhancing Russian higher education system competitiveness
through support to planning and strategic development of the
HEIs
• identification of strengths and weaknesses of the HEIs, their
comparative assessment with competitors, HEIs’ strategic
development planning and increasing their competitiveness
• constructing special rankings, for example, using the set of five
“excellence indicators” to assess leading universities or 13
indicators of global rankings to increase universities’
competitiveness in the global higher education area
28
Thank you for your attention!
НФПК
www.ntf.ru
Центр международных сопоставительных
исследований
ИМОМС НИУ ВШЭ
www.iori.hse.ru
29