1. Measure Training Results Some Concepts
Some experiences
Some lessons
Frank Cao
Lucent Technologies
fcao@lucent.com1
2. Training Process Needs Analysis
Design
Delivery
Evaluation2
3. Four Evaluation LevelsReaction
How do participants feel about the program?
Learning
To what extent did the participants increase knowledge, improve skills, and/or change attitude?
Behavior
To what extent did their job behavior change?
Results
What final results occurred? (Quantity, quality, safety, sales, costs, profits, ROI)3
4. Level One: ReactionWhat is evaluated?
Content
Trainer
Methodology
Material
Facilities
Logistics
Registration
4
5. Example QuestionsProbability you will use ideas from this session in your work
Content relevance to my job
Practical examples and experience
Speaker’s knowledge of subject area
Speaker’s presentation/facilitation skills
Speaker’s ability to respond to questions
Group participation5
6. Level One: ReactionWhat is evaluated?
Overall
6
7. Example QuestionsWhat did you like most about the program?
What did you like least about the program?
In what ways could this program be improved?
Would you recommend this course to others who are interested in the subject?
Which of the following features were important in your decision to attend? Rank in order of importance)7
8. Level One: ReactionHow to evaluate?
Questionnaires
Interview
Focus Group
Phone survey
8
9. Level One: ReactionQuestionnaire: scales
Excellent Well Fair Poor
Excellent Very Good Good Fair Poor
Strongly Agree Disagree
7 6 5 4 3 2 1
High Low
10 9 8 7 6 5 4 3 2 1
9
10. Level One: ReactionAdvantages
Easy to execute
Participants are free
Some relationship to on job performance Issues
Purely subjective
Halo effect (smile sheet)
Horn effect
Central tendency
Time pressure
10
11. Level One: ReactionTip 1: highlight purpose, solicit cooperation
To determine the degree that the seminar met your needs, we would like you to give us your honest opinion…
To make our future sessions as meaningful as possible, we would appreciate your candid evaluation of this program.
Your comments will be used to improve future offerings of this seminar.11
12. Level One: ReactionTip 2: Encourage comments
Comments are more accurate than scores
Tip 3: Set up baseline scores
Historical data
Benchmarking data
Pilot group data (Formative evaluation)
Tip 4: Combine questionnaires, interviews, focus groups
Tip 5: instant feedback for multi subject courses12
13. Level Two: LearningHow to evaluate?
Test
Demonstration
Presentation
Discussion
Role Play
13
14. Level Two: LearningAdvantages
Pressure to participants
Pressure to trainer
First step to change behavior Issues
Tension
Reliability
Creditability
Difficulty
Differentiation
Not the best indicator for on job change
14
15. Level Two: LearningTip 1: Fully utilize test
Test is good to evaluate knowledge based training
Functional skills
New Employee Orientation
Supervisor ABC
Be serious about test result of formal training
Instant self scoring and symbolic incentive for informal training
Reliability, Creditability, Difficulty and Differentiation
Set up baseline
Final score vs. gain
15
16. 16
17. Level Two: LearningTip 2: Set up rules in advance for demonstration and presentation
Topic
Time
Criteria to evaluate
Evaluators are key success factor
Tip 3: People don’t speak might be best learner
Tip 4: People good at role play might not be good learner17
18. Example:Management SkillsParticipants selected 3 topics from 11
Topic requirement
Time and assessors
Practice
Right before the stage…...Participants knew the topic
Assessors interacted with and provided feedback to participants18
19. Example: Leadership Style
Describe a situation when you used what you learned about leadership styles to improve the effectiveness of a relationship. Be sure to cover:
What motivated you to examine the style difference
What did you find and how
What did you do to flex your style
What were the results
19
20. Example: Performance Appraisal20
21. Level Three: BehaviorHow to evaluate?
Observation
Performance Appraisal Information
Supervisor feedback (oral or written)
Customer feedback (oral or written)
Employee feedback (oral or written)
21
22. Level Three: BehaviorAdvantages
Direct purpose of training
Recognized by supervisor and management teamIssues
Time consuming
Expertise scarcity
Cooperation from supervisor/customer
Multiple “cause-effect”
22
23. Level Three: BehaviorTip 1: Select program carefully
Behavior based
Company behavior
Professional skills
Managerial skills
Customer service (smile, response)
Trainer’s cooperation
Trainer’s expertise in evaluation
Documents ready before training
23
24. Level Three: BehaviorTip 2: Evaluation time in accordance with skills practice
Tip 3: Supervisor’s cooperation is key success factor
Commitment when register
Mutual benefit
Brief the key points24
25. Level Three: BehaviorTip 4: “Take advantage” of vendor
Key behavior identification
Questionnaires design
Time
Result analysis
25
26. Example: Time Management26
27. TOT: Transfer of TrainingOnly % of classroom learning was transferred to work!
The transfer must be PLANNED!27
28. TOT: Transfer of TrainingA. Before taking…, could you…?
B. Can you currently…?
C. How often are you applying…on your job
TOT occurred if
A is N
B is Y
C is >=1Y N
Y N
0 1 2 3 4 5 28
29. TOT: Transfer of Training
A: “Y” /All -Prior Knowledge Rate
B: “N” /All -Incomprehension Rate
C: “0” /All -Disconnect Rate
TOT Rate
Acceptable Rate
0-66%
0-33%
0-33%
Low Mid High
0-33% 34-66% 67-100%
29
30. Level Four:ResultsHow to evaluate?
Quantity
Quality
Safety
Sales
Costs
Profits
ROI
30
31. Level Four:ResultsAdvantages
Ultimate purposeIssues
Time consuming
Expertise scarcity
Cooperation from Management
Multiple “cause-effect”
31
32. Level Four:ResultsTip 1: Select programs that other causes are stable
Quality training - error rate
Safety training - accident rate
Sales training - revenue growth rate/order lose rate
32
33. Level Four:ResultsTip 2: Use a control group if practical
Tip 3: Allow time for possible results to take place
33
34. Example: Retention Skills34
35. Example: EMS ProjectShop Reengineering
Cost Reduction
Improve Customer Service Process
Order Entry via Configuration System
China CBU AR Improvement
35