Transcript Gabarits

TELOS TESTBED – 2006
Evaluating Learning Design Tools
Karin Lundgren-Cayrol
Isabelle Savard
Marcello Maina
Claire Banville
En collaboration avec:
Theme 6.3 France Henri & Josianne Basque
•
•
•
•
•
Outline
2006 - Testbed Context, Goals and Objectives
General Evaluation Plan & Formative Evaluation Design
Participants
Hands-on Evaluation Scenario
Results
– Background questionnaire
– Testbed Evaluation Scenario
– Final Questionnaire
• Preliminary conclusions
• Connected Student Thesis Work
– Isabelle Savard (Poster78 : Cultural Diversity and
Reuse of Learning Objects)
– Marcelo Maina (Adapting MISA to IMS Learning
Design)
• Questions
Goal
To identify strengths and weaknesses of
the TELOS Instructional Design Interface
potential Tools
Method
Cycle 2: C
guides and best
practices
MOT+editor and
Cycle 1: Revisions
modelling guide
Cycle3: IDLD
METHOD
Action Research
Processes
C
C
Plan the event
C
Carry out and
observe
C
Analyse and list
strengths,
weaknesses and
recommendations
Reflect and
Revise
Objectives 2006
• To test tools that are scheduled to be
incorporated into the TELOS instructional design
Interface. This year:
– MOT+LD editor
– Canadian LD Repository
– Learning Design Methodology
• To elicit information about practitioners ways and
needs in terms of facilitating their instructional
design tasks.
• To send results to concerned teams about their
tools in regards to their integration into TELOS.
– Team 6.3 – ID’s opinions, tasks and perceived competency profile
– Team 2.1 and 6.2 – Revisions of processes, tools and guides
Protocol & Testbed Cycle
demands
Revise
TELOS
Core
Identify needs
Carry out
testbed
demands
demands
Revise
LKMS
Revise
LKMA
Revise
LKMA
LKMS
Create testbed
scenario
Analyse
results
LKMA
LKMP
Create testbed
protocol
Disseminate
Context 2006

TELOS consists of four different interfaces
and toolsets according to actor

Engineer (TELOS Kernel)

Technician (LKMS)

Instructional Designer (LKMA)

Learner and Teacher Interfaces (LKMP)
Call for Participants
Requirements:

Experience in designing online learning /
e-learning

Good computer knowledge (Windows)

Experience with web-based technology
Testbed Agenda & Instruments

Complete Background Information
concerning (15 min)
 Work Context
Technical Profile and Tool Usage
 ID Experience and Practice


Carry out the Evaluation Scenario
 Demo Session (1h30)



Hands-on (5 h)
Complete a Final Questionnaire (30min)
Participate in a Focus Group (1h)
Let’s visit the IDLD site
And carry out this
EVALUATION Scenario
http://www.idld.org
Some Preliminary Results
Completion Rates
Background questionnaire (N=11)

9 Mauritians (Validation of instrument)

11 Canadians
Demo session (N=9)

1 in LORIT; 6 = ClickToMeet; 2 by LORIT Webcast & chat; 1 used

2 abandon
Evaluation Scenario (N = 6 - 1)

2 abandon; 1 finished but did not return it;

3 about half; 1 all but adapting UoL; 1 all the tasks
Final Questionnaire (N = 6)
4
most questions
2
less than half
Focus Group = cancelled
Background Information - Context

Mauritius (N=9)
 F2F interviews allowed validation of BQ
All worked at the University (7 ID; 2 tutors)
 3 designed in English, 2 in French, and 4 in Both languages
 5 had LO repository access


Canada (N=11)
 Geographically distributed (contact by email and phone)
8 University, 2 College and 1 Government
 4 Prof/Teach; 5 ID; 1 Tutor ; 1 Training Manager
 7 designed in English, 1 in French, & 3 Both languages
 2 LO access; 2 under development; 6 NO &1 didn’t know

Variable
Background Information –
Context (1)
value
Age group
20 to 30
31 to 50
50+
Mother Tongue
Profession
English
French
Other
Professor/Teacher
Instructional Designer
Trainer/Tutor
Training Manager
Mauritius
N=9
Canada
N=11
5
4
0
4
3
4
1
1
7 Creole
6
3
2 Chinese
0
7
2
0
4
5
1
1
Context (2)
Variable
Organization
Value
Mauritius
Canada
University
College
High School
Government
9
0
0
0
8
2
0
1
English
French
Both
3
2
4
6
3
3
5/8
0
0
2
8
1
Design Language
LO Repository Availability
Yes
No
Don’t know
Tendency: Instructional designers work in any
discipline, teachers referred to their domain
expertise.
Solve computer
problems
MAURITIUS
(N=9)
Background Information –
Most of
SomeRarely
Never
Canadians
the time Tool
times Usage
(pick one method)


Read manuals
1
10/11
mostly1 used Windows
Online help
1
2
6/11
used
IE,
4
Mozilla
&
1
Netscape
Web Forums
0
Human
resources
Solve computer
problems
For each method
indicate your preferred
use
7
Canada
(N=11)
Most
of the
time
Sometimes
Rarely
Never
T
Read manuals
4
5
0
2
11
Online help
8
1
1
1
11
Web Forums
4
4
2
1
11
Human resources
8
2
1
0
11
Preferred Help Method
Type method to learn
Canada Mauritius
(n-11)
(n=9)
Total
N=20
Read printed manuals
2
2
4 (20%)
Explore on my own
4
2
6 (30%)
Ask an expert
2
2
4 (20%)
Take a training course
2
2
4 (20%
Use a guided tour
1
1
2 (20%
Give up and leave it
0
0
0 (0%)
Conclusion: Telos must provide a plethora
of advisory systems as well as having
human resources available through the use
of sophisticated communication tools.
Tool Usage (1)
Software Name
Canada
(N=11)
Mauritius
(N=9)
Often
Never
Often
Never
11
0
9
0
Spreadsheets
4
1
5
0
PowerPoint
4
1
7
0
Videoconference
7
2
0*(3)
4
Textual forums
7
2
5
1
Community of practice
5
4
2
3
News Group
4
7
2
4
Chat
7
3
6
1
Wiki
6
2
-
-
Blog
5
3
-
-
Word processor
Communication Tools
Tool Usage (2)
Software
Canada
Mauritius
N=11
N=9
Often Never Often
Never
Concept Mapping Software
Internet Search Tools
HTML Autorhing Software
Webbased platform
with LOR
without LOR
5
5
3
3
11
0
9
0
8
0
6
1
3
8
2
1
1
4
4
2
Tool Usage (3) LO and Repository
Software Name
Canada N = 11
IHELP
MOT+L
D
Never heard about
it
5
4
4
7
3
I know about it
3
3
2
0
2
Seen demo
0
1
0
0
0
Used once
Use regularly
1
2
2
4
2
2
1
3
0
4
Type of usage
RELOAD PALOMA
MERLOT
Conclusion:
Most LO, IMS LD related tools are not widely known.
Background Information –
ID Experience and Practice (1)
Instructional Design Method
C
M
%
Formal method
1
0
9%
Adaptation of formal method
4
5
36%
Imposed by institution
2
2
18%
My own method
3
2
27%
10
9
Total
Canada
Mauritius

5/11 had more than 6 years


3/11 had between 3 and 6 years


3/11 less than 2 years

1/9 had more than 6 years
16/20 more than
years
7/9 had 3
between
3 and 6 years
1/9 less than 2 years
Conclusion: TELOS must be able to provide flexible
instructional design environments.
Reusing LO’s
Use existing LO’s in the
following manner
Canada
N=11
Mauritius
N=9
Total
N=20
To create a new course
1 (9%)
6 (55%)
7 (35%)
To adapt an existing course
3 (27%)
0 (0%)
3 (18%)
Both
7 (64%)
3 (30%)
10 (50%)
Canada
N=11
Mauritius
N=9
Total
N=20
As is
1
0
1 (5%)
As an inspiration
4
0
4 (20%)
All possibilities
6
9
15 (75%)
How do you use existing
LOs
Conclusion: LO’s are used in different ways and
are starting to be a well known concepts.
Validation process and results
Validation process in Mauritius (Savard, 2006)
• First version of the background questionnaire
• Semi-structured interviews to find out their
regular ID process and tasks
• Similar to Gagné’s ADDIE Instructional Design
Model.
Results:
• Comment on each question which led to
– changes to Que’s 5, 10 & 21
– addition Que’s 17, 19 & 22 (ADDIE)
ADDIE - Instructional Design
Task
•
•
•
•
•
Analysis
Design
Development/Production
Implementation
Evaluation
ADDIE - Instructional Design Task
• Analysis
Conclusions

In general, the Analysis task is already done when a demand
comes to Mauritians Instructional Designer’s.

They often have to reformulate objectives.

For the Canadians, the Analysis is an important task.
ADDIE - Instructional Design Task
• Design
Conclusions:
• Instructional Structure is an important task for both
ID from Mauritius and Canada.
ADDIE - Instructional Design Task
• Development/Production
Conclusions:
•
•
Material production is an important task in both Mauritius and Canada.
In Mauritius the delivery and maintenance plan seem not to be their
responsibility.
ADDIE - Instructional Design Task
• Implementation
• Conclusions:
• Small field testing doesnt take too much time for
ID from both contries.
ADDIE - Instructional Design Task
• Evaluation
Conclusions
• The course evaluation is not a major concern in Mauritius.
• 70% of Canadians do carry out this task
ID Competencies (ibstpi)
• Four groups of Competency statements for
instructional designers:
–
–
–
–
ABCD-
Professional Foundations
Planning and Analysis
Design and Development
Implementation and Management
• Validate which competencies were the most
important in their job situation.
http://www.ibstpi.org/
ID Competencies (ibstpi)
Results
A. Professional Foundations
Mau Can
1- Communicate effectively in visual, oral and
written form.
9
11 20
2- Apply current research and theory to the
practice of instructional design.
8
7
15
3- Update and improve one’s knowledge, skills and
attitudes pertaining to instructional design and
related fields.
8
4
12
4- Apply fundamental research skills to
instructional design projects.
6
4
10
5- Identify and resolve ethical and legal
implications of design in the work place.
5
3
8
Total
ID Competencies (ibstpi) - Results
B. Planning and Analysis
Mau Can
Total
6- Conduct a needs assessment.
7
6
13
7- Design a curriculum or program.
7
6
13
8- Select and use a variety of techniques for
determining instructional content.
7
4
11
9- Identify and describe target population
characteristics.
8
5
13
10- Analyze the characteristics of the
environment.
5
7
12
11- Analyze the characteristics of existing and
emerging technologies and their use in an
instructional environment.
8
6
14
12- Reflect upon the elements of a situation
before finalizing design solutions and strategies.
8
6
14
ID Competencies (ibstpi)
Results
C. Design and Developement
Mau Can
Total
13- Select, modify, or create a design and
development model appropriate for a given project.
7
9
16
14- Select and use a variety of techniques to
define and sequence the instructional content and
7
7
14
15- Select or modify existing instructional
materials.
9
7
16
16- Develop instructional materials.
9
5
14
17- Design instruction that reflects an
understanding of the diversity of learners and
groups of learners.
6
4
10
18- Evaluate and assess instruction and its impact.
8
5
13
strategies.
ID Competencies (ibstpi)
Results
D. Implementation and Management
Mau Can
Total
4
7
8
7
12
14
21- Apply business skills to managing instructional
design.
2
5
7
22- Design instructional management systems.
3
4
7
23- Provide for the effective implementation of
instructional products and programs.
6
4
10
19- Plan and manage instructional design projects.
20- Promote collaboration, partnerships and
relationships among the participants in a design
project.
Evaluation Scenario
Variable
Yes
No
T
Intro to IMS LD motivating
3
2
5
Methodological Guide sufficient
5
0
5
Preferred type of resource to learn new subjects
0
4
1
Papers
Presentations
Guided Tours
Trouble occurred during
Download of MOT+
Installation of MOT+
5
1
0
2
3
3
Modeling technique sufficient to get started
3
0
3
Troubles occurred accessing PALOMA
1
2
3
3
3
0
0
3
Rubric's Evaluation Grid is valuable for reuse
3
0
3
ID Principles doc is valuable for reuse
2
1
3
Narrative description
Useful for reuse
Helps understand graphical UoL
Comments and Suggestions
• Experience too time consuming because of all
the documents to be studied
• Several separated Demo Sessions are
necessary
• IMS LD very complex
• MOTPlus has significant advantages
• Graphical modeling is interesting
• All tools were very interesting
Final Questionnaire
IDLD PORTAL
Feature (n = 6 to 7)
Positive
Neutral Negative
Answers
Answers
Organisation
Simple
Clear
Attractive
Comprehensive
Motivating
4
7
7
4
6
2
0
0
2
0
Interface Text
Uses clear Vocabulary
Well adapted to target audience
Is free of jargon
Highlights the activity
Is motivating
4
4
4
4
3
3
2
3
2
3
1
Conclusion: Very divided opinions, we need more evaluators!!
MOT+LD and Graphical Modeling
Usability (N=4)
Positive
Answers
Neutral
Negative
Answers
Well designed
2
1
1
Responds to my needs
2
1
0
Is useful
4
0
Is functional
4
0
Is transparent
3
1
Opens new design possibilities
2
I could easily go from one model to
another without being lost or confused
2
The links are helpful and pertinent
3
Sub-models are a very good way of viewing
4
2
0
2
1
0
0
detailed information
Conclusion: At least useful and functional for those who tried
PALOMA Resource Manager
Usability (n=4 or 5)
++
Neutral
--
Well designed
2
Responds to my needs
1
2
1
Is useful
2
2
0
Conclusions:
Is functional
Divided opinions and
4
Needs more evaluators!!
Is easy to use
2
Has easy to understand terminology
3
Asset to my organisation
0
Facilitates reuse
2
Motivates me to share resources
2
Has a very helpful user guide
3
I would suggest this tool to my collegues
1
Suggestions:



2
0
1
1
1
3
1
2
1
1
1
1
Need a more user-friendly version
We need more reliable federated search protocols
Advanced search is a bit confused
2
Some Preliminary Conclusions
• The ADDIE Model and ibstpi competency
– Generally practiced by ID’s (few differences)
– Questions whether Planning and Evaluation is an ID
task?
• TELOS must be able to provide
– Sophisticated communication tools to simulate
human resource contact
– flexible instructional design environments
• Confirms complexity of the IMS LD concepts
– Need to adjust vocabulary
– Provide transparent & flexible tools
– Provide more examples in the Canadian LD
Respository