Level 3 and 4 Evaluation – Mailbox Management Lesson
We
use Kirkpatrick’s (1998) Four Level Model for evaluation. The Level 3 and 4
evaluations follow his guidelines for these evaluations. Level 3 measures the effectiveness
of the training solution that we implement. Did the knowledge that we attempted
to impart to the student actually transfer. Level 4 measures the results of our
lesson plan, as measured in dollars.
Level 3 - Transfer
The
effectiveness of this lesson will be measured by using the statewide
call-tracking system called “Remedy”. Details of the problems reported to
Office of Information Technology (OIT) by CHFS (Cabinet for Health and Family
Services) employees can be extracted from this database using any listed field
or fields for the report. Total number of reported problems, types and network
names of computers, user names, locations of equipment, length of time
employees experienced difficulties before placing the call to OIT, and lengths
of time to repair the problem are among the items tracked for all problems
reported to OIT. This lesson was designed to reduce the amount of problems
associated with the Microsoft Outlook application coming into the OIT helpdesk.
If problems are, indeed, reduced, we assume that it decreased due to knowledge
gained in the lesson the students took.
Level 4 – Results (Return on
Investment)
In this
lesson we will measure the results in dollars. Costs for service and lost
employee productivity can be estimated using data generated from the Remedy
database. It has been calculated that 100% ROI (Return on Investment) for the
annual training and development costs can be generated in the first year using
the formula:
In addition
to monitoring the number of help desk calls generated by CHFS-OIT employees,
students of the Mailbox Management lesson will be re-interviewed 90 days after
the completion of the lesson. If the objectives of the Mailbox Management
lesson have been met, we will ask the
trainees to retake the Level 1 evaluation they took at the end of the class. If
the objectives of the course have not
been met we will ask the trainees to retake both the Level 1 and Level 2
Evaluations that were taken at the end of the lesson. Also, one year after
employees take Mailbox Management training, we will ask the members of OIT
whether or not they have observed any change in user attitude or expertise, in
reference to Outlook mailboxes.
References
Kirkpatrick,
D. L. (1998). Evaluating Training
Programs: The Four Levels. San
Francisco: Berrett-Koehler Publishers, Inc.