Transcript IDEAS Delhi, Workshop – DAC April 2005 Independent Evaluation « Quality Assessment of SDC’s External Evaluation Reports » Anne Bichsel Evaluation + Controlling Division.

IDEAS Delhi, Workshop – DAC April 2005
Independent Evaluation
« Quality Assessment of SDC’s
External Evaluation Reports »
Anne Bichsel
Evaluation + Controlling Division
1
IDEAS Delhi, Workshop – DAC April 2005
Content
1. Objectives of the Quality Assessment
2. Procedures and Method
3. Findings
4. Recommendations
2
IDEAS Delhi, Workshop – DAC April 2005
1. Objectives ot the Quality Assessment
•
To assess the quality of SDC‘s
external evaluation reports
•
To contribute to improving future
performance and achieving a higher
evaluation quality of SDC‘s external
evaluations
3
IDEAS Delhi, Workshop – DAC April 2005
2. Procedures and Method (I)
•
Step I: Random Sample of 12 external
evaluation reports
•
Step 2: devise a list of evaluation standards
on the basis of
- „DAC Minimum Sufficient Evaluation
Standards“
- SEVAL Standards
- Key questions in Approach Paper
4
IDEAS Delhi, Workshop – DAC April 2005
2. Procedures and Method (II)
List of evaluation standards with 4 categories
•
Utility (9 standards): readable, accessible, timely
evaluations with a good summary
•
Feasibility (3 standards):
realistic well thought-out evaluations
•
Propriety (4 standards): ethical aspect of evaluations
•
Accuracy (7 standards):
proper scientific methods and procedures
5
IDEAS Delhi, Workshop – DAC April 2005
2. Procedures and Method (III)
•
Step 3: review documents, conduct interviews
with desk managers and evaluators
•
Step 4: work through all the standards for the
sample of evaluations. Result: 12 fact sheets
•
Step 5: analysis and comparison
•
Step 6: conclusions and recommendations
6
IDEAS Delhi, Workshop – DAC April 2005
3. Findings (I)
Strengths of external evaluation reports
•
Identification and participation of stakeholders
•
Timely reporting
•
Cost effectiveness
•
Complete and balanced assessment
7
IDEAS Delhi, Workshop – DAC April 2005
3. Findings (II)
Weaknesses of external evaluations reports
•
Selection procedures of evaluation team
•
Formal written agreement
•
Comprehensive and clear reporting
•
Description of evaluation purpose and objectives
•
Description of evaluation procedures and methods
•
Making findings available
8
IDEAS Delhi, Workshop – DAC April 2005
4. Recommendations (I)
 Draft good, realistic and comprehensive TORs
•
Clear purpose
•
Clear, focused, concise and understandable
objectives with three to four questions
•
Agreement on TORs among SDC divisions
•
Realistic expectations in terms of resources and
timeframe
•
Beware of questions at the impact level
9
IDEAS Delhi, Workshop – DAC April 2005
4. Recommendations (II)
 Ensure more competitive and open selection
procedures
•
Aim at balance in desired knowhow and
independence of evaluators
•
Place more weight on evaluation knowhow (methods)
•
More open and transparent selection procedures
•
More competition
•
Careful team building in matching personalities,
skills, division of labor
10
IDEAS Delhi, Workshop – DAC April 2005
4. Recommendations (III)
 Improve the conditions for the utilisation of
external evaluations
•
Accompany evaluations closely for maximum
learning potential
•
Continuity on the part of desk managers
•
Reports need to be more accessible to a wider
audience
•
Enhance the quality of evaluation reports
•
Make results of external evaluations more widely
available
11
IDEAS Delhi, Workshop – DAC April 2005
4. Recommendations (IV)
 Enhance the conditions for high quality external
evaluations
Evaluations are a challenging business, hence:
Provide support and training for desk managers
• Training in evaluation
• Support in drafting TORs and commissioning
evaluations
• Support in obtaining high quality reports
12
IDEAS Delhi, Workshop – DAC April 2005
In sum: less is more !
13
IDEAS Delhi, Workshop – DAC April 2005
Follow-up by the Core Learning Group: Evaluation &
Controlling Net
Direktor DEZA
Stellv. Direktor
Gerhard Siegfried
Samuel Wälty
Anne Bichsel
Sekretariat (je 50%)
Regula Herlan
Christa Rohner
jeweilige Bereichsleitung
H-Bereich
Christoph
Jakob
O-Bereich
Kuno
Schläfli
E-Bereich
Peter
Meier
Regula
Bäbler
M-Bereich
F-Bereich
A-Bereich
Markus
Glatz
Ivo
Angehrn
Alexandre
Kobel
14
IDEAS Delhi, Workshop – DAC April 2005
Steps of an
Independent Evaluation
COSTRA
Comité Stratégique
0.
•COSTRA Decision
I.
•Approach Paper
•Core Learning Partners (CLP)
II.
•TORs&contracts
•Implementation
•Draft Report
•Debriefing
•Final Report
III.
•Agreement at Completion Point
•Dissemination to Broad Learning
Partners (BLP)
IV.
•Senior Management Standpoint
discussed in COSTRA
15
IDEAS Delhi, Workshop – DAC April 2005
Measures to improve quality
 Communicated evaluation results /
recommendations in all departments
 Clarified and communicated the palette of
instruments
 Developing toolkit for commissioners of
evaluations
 Intraweb: Description of Process / examples
 Principles to ensure successful Use of
Evaluations
 Improving training courses: curriculum and
availability (1 day training Ext. Evals)
 Backstopping and Quality Control by the E&C
Net
16
IDEAS Delhi, Workshop – DAC April 2005
Palette of Instruments
Independent Evaluation
External Evaluation
External Review
Self-Evaluation
Expert Opinion
17
IDEAS Delhi, Workshop – DAC April 2005
Independent
Evaluations
since 2001
External
Evaluations
External
Reviews
2 – 5 annually
5 – 10 annually
according to
operational needs
outside line management
( E&C Division)
- issues of interest
across departments
- selected country
programs
within
- to generate knowledge
within the department
- of interest across
divisions
line management
- ongoing operations
- in the context of
program cycle
management
aim for the same standards
DAC Executive Summary
Registration
in data management
system (DMS)
18
IDEAS Delhi, Workshop – DAC April 2005
19
IDEAS Delhi, Workshop – DAC April 2005
20