Running head: PRINTER
MAINTENANCE PROGRAM EVALUATION PROPOSAL
Printer Maintenance Program Evaluation Proposal
Anthony P. Niemann
ELFH 606-50
Printer Maintenance Program Evaluation Proposal
Needs Assessment
The Printer Maintenance Program in
The
structure and context of the program were delineated in previous years. Employees
in the Department for Community Based Services (DCBS) include social workers
who have responsibility to provide
If a printer develops a problem, a
CHFS employee will call the OIT helpdesk and request service. Technicians working for OIT helpdesk will
temporarily set up an alternate printer that is not as conveniently located to
the employee’s workspace as the original printer. OIT personnel generate a
web-based request for service from IBM. IBM
subcontracts the maintenance contract with Grumman. Grumman replaced Pomroy as
subcontractors for the IBM Printer Maintenance Program in 2006. There were no
changes in printer issues encountered as a result of the change in subcontractors.
The Printer Maintenance Program is
being evaluated because of end user dissatisfaction, claims of inadequate
repairs, and the possibility of inadvertent over-billing. The purpose of the
evaluation is program improvement. Frustration on the part of some employees at
offices throughout the state as well as OIT staff is high because results of efforts
spent in obtaining service through the program are often unsatisfactory. Requests
for services made by CHFS and OIT workers do not always translate into satisfactory
resolution of problems by authorized IBM subcontractors unless repeated calls
for service are placed. OIT workers must re-enter a “request for service” on
the IBM website, which could result in over billing by IBM. Occasionally the OIT staff will replace a printer
because timely service is not provided by IBM subcontractors. An evaluation of
this program should provide a method to ensure that proper corrective
maintenance activities are performed, the services are not over-billed, and
frustration levels of CHFS and OIT employees are reduced.
Program Evaluation Plan
A participant-oriented approach was chosen to
evaluate the Printer Maintenance program to insure that the needs of those who
participated in this program were the cynosure in this evaluation (Fitzpatrick,
Sanders, & Worthen, 2004; Scriven, 1991). This program has never been
evaluated by an approach that had, as its primary objective, a review of the
concerns and consequences of the participants. The program has been examined
from a viewpoint of whether it met stated objectives, as well as whether it met
the information needs of stakeholder managers. The participant-oriented
approach was selected in order to evaluate the effect of this program on
participants.
According to Fitzpatrick (2004) Stake
pioneered and favored the participant-oriented approach over a concern that
evaluations had taken on “parochial, objectivist, mechanistic, and stagnant
conceptions and methods” (cited in Fitzpatrick et al., 2004, p. 131). Stake saw
this form of evaluation as a method that would allow for higher levels of
responsiveness to the participants of the program being evaluated. Many
researchers today favor using some variation of the participant-oriented
approach for evaluation. Particularly critical in carrying out this evaluation
approach is a reliance on naturalistic
methodology. Proponents of this methodology have correctly pointed out the
impropriety of many assumptions made by those who favor a more formal
methodology that relies on a social science model (Scriven, 1991). The
participant-oriented approach emphasizes qualitative data, rather that merely
relying on quantitative data. It also emphasizes the human element in
evaluation.
While I would not use the participant-oriented
approach for evaluating all programs, it particularly fits the needs for
evaluation of this specific program. As stated earlier, previous evaluations of
the program utilized objectives-oriented and management-oriented approaches.
Additional characteristics of using the participant-oriented evaluation approach have been highlighted as:
dependent on inductive reasoning; data comes from many sources; the evaluation
plan is developed for each program (not standardized); and results of
evaluation often record multiple realities (Fitzpatrick et al., 2004). The
participant-oriented approach is more responsive to printer users, taxpayers,
and grass-root OIT workers who are concerned that the program is not effective.
Stake (1975) suggested that there are two
countenances of evaluation: description and judgment. His framework for the
participant-oriented model was used because, “it relates more directly to
program activities than to program intents” (as cited in Fitzpatrick et al., p.
136). The stakeholders in this program are:
Evaluations that use the participant-oriented
approach often do not follow a standard plan (Fitzpatrick, et al., 2004). This
is a formative evaluation because CHFS intends to insure that any problems
found will be corrected, and maximum benefit to the Cabinet will be realized
after recommendations have been implemented. The program has been in existence
for many years and organizations throughout the country use programs similar to
this one. This evaluation addresses problems that were pointed out in an
earlier needs assessment. A list of survey questions were developed (Appendices
A – E), although new questions may arise in the process of conducting this
evaluation. Cronbach (1982) sees evaluations going through divergent and convergent
phases. The divergent phase of this evaluation was conducted during the needs
assessment. Survey questions were developed for five of the seven stakeholder
groups. Survey questions were not developed for two of the seven groups -
taxpayers and clients. It can be assumed that the first of these two groups,
the taxpayer group, will always want to eliminate wasteful government spending,
and that this is their main concern. It can be further assumed that the second
of these two groups, the client group, is not critically affected as a result
of printer malfunctions. Individual clients may experience a prolonged or
delayed appointment for no more than 15 minutes with their case worker. This
would be an abnormal situation that would be corrected as soon as the worker
connected to another printer located in the vicinity of their office.
Interviewing clients about printing issues would be more disruptive for the
client than the slight delay of services.
Survey questions for this evaluation were developed
for the five following stakeholder groups: CHFS employees; OIT employees; OIT
management; IBM; and Grumman (Appendices A – E). Evaluation questions were
developed to focus on perceived needs of the program that were identified in the needs assessment. There are three
methods that will be used to obtain answers to questions: personal interviews;
surveys; and data from existing IBM databases listed on a website that OIT
technicians use to request printer service. This hard data will help to provide
verification or refutation of responses to questions asked in surveys and
interviews. As the IBM database information is proprietary, it must be excluded
from this document. A matrix based on Cronbach’s work (1982) was developed by Fitzpatrick
et al., (2004, p. 249, Figure 12.2) and was used as criteria for selecting
evaluation questions. Survey questions will guide the evaluator in interviews
with members of various groups. The interviews will allow a verbal give and
take between the evaluator and groups that are personally interviewed. Survey (interview) evaluation questions for
various groups appear as follow: Appendix A – CHFS Employees; Appendix B – OIT
Employees; Appendix C – OIT Management; Appendix D – IBM Management; and
Appendix E – Grumman Employees.
Participants in the CHFS and OIT employee groups will
be chosen for surveys and interviews randomly from each of nine Department for
Community Based Services (DCBS) regions in the state. One percent of all CHFS
workers will receive a survey questionnaire. Seventy-five percent of the OIT
staff who holds the job title of Network Analyst II will receive a survey and twenty-five
percent will be interviewed. Interviews will be conducted with five OIT
management personnel, and will be selected by the program evaluator. One interview will be conducted with senior
IBM management personnel familiar with the Printer Maintenance Program.
Individual interviews will be arranged with all Grumman technicians located in
Evaluators’ time will be billed at the rate of $400
per day, not including expenses. Expenses include hotel bills, travel, and per
diem of $30 per day. It is estimated that interviews will take four days for
one evaluator. Hotel accommodations will be required for two days, at a rate of
$100 per day. Two additional days will be billed for the evaluator’s time to
develop survey questionnaires. Finally, the evaluator will require an
additional five days to analyze data and prepare reports, for a total of ten
days. Total evaluator costs amount to $4,000 plus travel and expenses of
approximately $400. Research assistant’s time will be billed at a rate of $150
per day, not including expenses or travel. A research assistant will take three
days to develop the sampling plan and conduct the non-response bias check
procedure. The research assistant will also oversee work performed by a clerical
worker assigned to the project for such tasks as sending out emails and
tallying results of surveys. The
research assistant will be assigned to these tasks for three days, for a total of
six days and a cost of $900. The clerical worker will be assigned to email
tasks associated with sending and collecting survey data for a total of ten
days. The Clerical worker’s time will be billed at the rate of $75 per day.
CHFS facilities and material will be used to generate all reports in this
evaluation, so no paper, duplication, or postage costs are expected. The total
cost for this evaluation amounts to $6,714 and is itemized in Appendix F.
Data Collection and Reporting Results
Program implementation will be assessed using both
quantifiable and qualitative means.
Quantifiable methods are available through an existing IBM database that
is available to OIT and IBM personnel via a secured website. Printers that
await repair are included in an area of the database called “Active Requests”. Information
describing details about printers that have been repaired are maintained and
searchable for a period of three months. A new Access database will be
established and maintained by OIT personnel for the purpose of listing printers
that have not been repaired within one week of the date the request form was
filled out and submitted. The new database will identify reasons why printers
are not being repaired in a timely manner. It will also allow entry of a
printer serial number, already included on the service request in the IBM
database. All printers that experience two service requests for the same problem
within a thirty-day period will be listed in this database to determine the
extent of this critical issue. In addition, surveys will accumulate
quantifiable data to determine the extent of the difference in attitudes about
the effectiveness of the Printer Maintenance Program.
Analysis of qualitative data is critical in the
implementation phase of this evaluation in order to “’build levels of
confidence’ in the evaluations ultimate conclusion” (Fitzpatrick et al., 2004,
p. 361). The main method for accomplishing this is through interviews and
questionnaires. Results from questionnaires sent to CHFS and OIT workers will
be tallied, which will allow future measurement of CHFS and OIT worker
attitudes. Interviews conducted with four stakeholder groups will allow
qualitative data to be gathered by allowing open-ended responses to survey
questions. This, in turn, should shed more light on questions such as whether
information provided by the IBM database is sufficient, or whether Grumman
receives all the information that the OIT worker enters into IBM’s database.
Problems associated with this area of the evaluation surfaced during the needs
assessment and consequently, the following steps will be taken.
The OIT investigative team will enter information
from the IBM database into the newly established Access database. The OIT
investigative team will examine the work-order entered in the IBM database by
the OIT worker and insure that it has been entered correctly. If it has not
been entered correctly, the investigative team should note this fact in the
database and notify OIT management.
If the data has been entered in the IBM database
correctly, the OIT investigator shall contact IBM and inquire as to whether the
call was dispatched to Grumman. If not, the investigator should enter this
information into the Access database. If IBM has sent the work-order to
Grumman, the investigator should call Grumman management to determine if they received
the work-order.
If Grumman has not received the work-order, the
investigator should note this in the Access database and notify IBM management
about the problem. If Grumman has received the work-order, they should be asked
why they have not contacted the person listed as a contact. If the reason for
not calling the contact person is because the phone number did not work, or the
person was unavailable, the investigator should note this and call OIT
management. If the Grumman technician did get in touch with the contact person
and the printer is fixed, IBM should be contacted to determine why the call has
not been cleared from their list of printers requiring service. If the Grumman
technician did get in touch with the contact person and the printer is still
broken, it shall be noted, along with the reason. Once quantifiable reasons why
printers are not being repaired are listed in the Access database, solutions
can be provided.
Interim reports will be generated and circulated through
the use of the Management Response and Tracking Form in Appendix G. Employees
will be provided with copies of this form to allow review, discussion, and
comments throughout the evaluation. Final results of the evaluation shall be
presented to the Director of CHFS in written and verbal reports by the
evaluators at the end of the evaluation. The written report shall consist of an
executive summary, an introduction to the report, a description of the Printer
Maintenance Program with questions to be answered, a discussion of the methods
used to perform the evaluation, and a discussion of the conclusions and
recommendations. The verbal report shall consist of a presentation of the
evaluation with emphasis on actions that the evaluator feels need to be
addressed by CHFS, OIT, IBM, or Grumman. This will include a list of measures
that can be taken to bring about program improvement. It is expected that some
quantifiable improvements will be recognizable immediately, and will be
reported via email memos, web-site postings, and briefings as they occur. Other
improvements will only be measurable if additional questionnaires, such as the
ones that appear in the Appendices, are completed. All issues will be discussed
in written and verbal formats. In following the suggestions of Rothwell,
Sullivan, and McLean (1995), care shall be taken to avoid data overload in the
verbal presentation of evaluation results, but details will be provided in the
written report.
The source of influence for this evaluation is via
results-oriented use. According to Fitzpatrick et al., (2004), this is a
traditional focus for evaluators. The results of information gained through
conducting this evaluation will be used to ensure that printers are repaired in
a timely manner and that all repairs are effective. Frustration levels of CHFS
workers will be reduced as a result of printers that are returned to service in
a speedy manner. Frustration levels of OIT workers will be reduced because they
will have an Access database tool which will let them accurately track problems
associated with any printer. These methods will ensure that proper corrective
maintenance is performed in a timely manner, the service is not over-billed,
and frustration levels of CHFS and OIT employees are reduced.
It follows that if printers operate
satisfactorily, attitudes of CHFS and OIT workers about the Printer Service
Program will improve. Questionnaires designed to measure attitude of workers
can be redistributed after the evaluation is over. Results of data gathered on
future questionnaires and interviews (Appendixes A-E) may again be used to
assess frustration levels of CHFS and OIT workers. Quantifiable data gathered
from the existing IBM databases coupled with the newly established OIT database
will record immediate and future improvements in the Printer Maintenance
Program. The number of printers that remain on the Service Request list longer than
one week will identify whether improvements have been made. Printers that
require repetitive service calls within 30 days will also be tracked, thereby
insuring that over-billing, if any is found, will be resolved.
References
Cronbach,
L. J. (1982). Designing evaluations of
educational and social programs. San
Francisco: Jossey-Bass.
Fitzpatrick,
J. L., Sanders, J. R., & Worthen, B. R., (2004). Program
evaluation:
Alternative approaches
and practical guidelines (3rd
ed.).
Bacon.
Howe,
K. R. & Ashcraft, C. (2005). Deliberative democratic evaluation: Successes and
limitations of an evaluation of school choice. Teachers College Record, 107(10),
2275-2298.
Rothwelll,
W. J., Sullivan, R., & McLean, R. W. (1995). Practicing organizational
development:
A guide for consultants.
Scriven,
M. (1991). Evaluation thesaurus (4th ed.).
Publications, Inc.
Stake,
R. E. (1975). Program evaluation,
particularly responsive evaluation. (Occasional
Paper No. 5).
|
Question |
Strongly Agree |
Agree |
Neutral |
Disagree |
Strongly Disagree |
|
Printer Availability |
|||||
|
My printer seldom has problems. |
|
|
|
|
|
|
My printer always has problems. |
|
|
|
|
|
|
My printer is too far away from my workplace. |
|
|
|
|
|
|
When my printer breaks, I have to use a printer that is much farther
away. |
|
|
|
|
|
|
Printer Capability |
|||||
|
I need a printer that can print envelopes. |
|
|
|
|
|
|
I have problems printing to copiers that are used for printing also. |
|
|
|
|
|
|
Printer Repair |
|||||
|
When my printer has problems, it is fixed quickly. |
|
|
|
|
|
|
It sometimes takes more than a week for someone to fix my printer. |
|
|
|
|
|
|
The state has a contract with IBM to repair printers when OIT places a service request. |
|
|
|
|
|
|
CHFS technicians can’t fix many printing problems. |
|
|
|
|
|
|
Other |
|||||
|
My supervisors are
responsive to my printing problems. |
|
|
|
|
|
|
CHFS technicians are responsive to my printing problems. |
|
|
|
|
|
|
Overall, I am satisfied with the printer I use. |
|
|
|
|
|
|
Question |
Strongly Agree |
Agree |
Neutral |
Disagree |
Strongly Disagree |
|
IBM Service Request Website |
|||||
|
There are significant problems with the IBM service request website. |
|
|
|
|
|
|
We can’t enter the model on the website. |
|
|
|
|
|
|
There is not sufficient room to enter problem description/comments for
the printers. |
|
|
|
|
|
|
All requests for service received by IBM are dispatched to Grumman. |
|
|
|
|
|
|
There are always several printers on the active requests list for more than one month. |
|
|
|
|
|
|
IBM does not provide Grumman
with all the information that OIT enters |
|
|
|
|
|
|
There is no way that to tell if
“active requests have been dispatched to Grumman |
|
|
|
|
|
|
Service Related |
|||||
|
It appears that many calls for service go unanswered. |
|
|
|
|
|
|
Grumman is unaware that some printers require service. |
|
|
|
|
|
| Grumman does not have access to IBM’s “active requests” website. |
|
|
|
|
|
|
Grumman does not leave the customer with a copy of the service ticket. |
|
|
|
|
|
|
OIT has no effective way to track printer service calls. |
|
|
|
|
|
|
IBM is unaware whether Grumman has received a request for service or not. |
|
|
|
|
|
|
Billing Questions |
|||||
|
The only way OIT can get service
on printers that appear on the “active requests” list for more than two weeks
is to place another call. |
|
|
|
|
|
|
There is no way to tell IBM that we have a repeat call. |
|
|
|
|
|
|
There is no place for the OIT tech to inquire about billing issues. |
|
|
|
|
|
|
Question |
Strongly Agree |
Agree |
Neutral |
Disagree |
Strongly Disagree |
|
IBM Service Request Website |
|||||
|
There are significant problems with the IBM service request website. |
|
|
|
|
|
|
We can’t enter the model on the website. |
|
|
|
|
|
|
There is not sufficient room to enter problem description/comments for
the printers. |
|
|
|
|
|
|
All requests for service received by IBM are dispatched to Grumman. |
|
|
|
|
|
|
There are always several printers on the active requests list for more than one month. |
|
|
|
|
|
|
IBM does not provide Grumman
with all the information that OIT enters |
|
|
|
|
|
|
There is no way that to tell if
“active requests have been dispatched to Grumman |
|
|
|
|
|
|
Service Related |
|||||
|
It appears that many calls for service go unanswered. |
|
|
|
|
|
|
Grumman is unaware that some printers require service. |
|
|
|
|
|
| Grumman does not have access to IBM’s “active requests” website. |
|
|
|
|
|
|
Grumman does not leave the customer with a copy of the service ticket. |
|
|
|
|
|
|
OIT has no effective way to track printer service calls. |
|
|
|
|
|
|
IBM is unaware whether Grumman has received a request for service or not. |
|
|
|
|
|
|
Billing Questions |
|||||
|
The only way OIT can get service
on printers that appear on the “active requests” list for more than two weeks
is to place another call. |
|
|
|
|
|
|
There is no way to tell IBM that we have a repeat call. |
|
|
|
|
|
|
There is no place for the OIT tech to inquire about billing issues. |
|
|
|
|
|
|
Question |
Strongly Agree |
Agree |
Neutral |
Disagree |
Strongly Disagree |
|
Service Request Website |
|||||
|
The service request website works great for entering service requests. |
|
|
|
|
|
|
If no one calls about an open service request after two months we delete them. |
|
|
|
|
|
|
This website is not meant to be a database used for CHFS purposes other
than to request service. |
|
|
|
|
|
|
Without a valid contact name on the service request, all other
information is useless. |
|
|
|
|
|
|
Sending Grumman a Work-Order |
|||||
|
We give Grumman all the information that the technician gives to us. |
|
|
|
|
|
|
Once we send Grumman a work-order we assume that it is taken care of unless we hear from CHFS. |
|
|
|
|
|
|
It is expected that the contact will give Grumman information that they need to work on a printer. |
|
|
|
|
|
| The problem description is part of the information the contact should provide. |
|
|
|
|
|
|
The model number is part of the information the contact should provide. |
|
|
|
|
|
|
Billing for Work-Orders |
|||||
|
We do not bill for “repeat” calls. |
|
|
|
|
|
|
We must be informed if a call is a repeat problem to ensure we do not send a bill. |
|
|
|
|
|
|
We do not bill for a work-order
unless the printer has been worked on. |
|
|
|
|
|
|
If there are billing issues to address, CHFS must inform us what the problem is. |
|
|
|
|
|
|
Our service reps do not leave copies of completed work-orders with CHFS. |
|
|
|
|
|
|
We will address billing problems but CHFS has internal issues that prevent them. knowing whether we sent a bill. |
|
|
|
|
|
|
Question |
Strongly Agree |
Agree |
Neutral |
Disagree |
Strongly Disagree |
|
Service Request Website |
|||||
|
We do not get all information that is on the IBM website when we receive a work-order from IBM. |
|
|
|
|
|
|
We do not have access to the IBM website. |
|
|
|
|
|
|
|
|
|
|
|
|
|
CHFS Technicians & Contacts |
|||||
|
CHFS is not able to give us the information about the printer problem that we need. |
|
|
|
|
|
|
CHFS contacts can’t provide local contact names for people who use the printers. |
|
|
|
|
|
| Local people who use the printers know practically nothing about them. |
|
|
|
|
|
|
Ample time was given to complete scenario. |
|
|
|
|
|
|
|
|
|
|
|
|
|
IBM Work-order |
|||||
|
The contact info IBM sends is not valid or they can’t provide information we need. |
|
|
|
|
|
|
|
|
|
|
|
|
|
Billing |
|||||
|
We do not leave copies
of completed work-orders with the customer. |
|
|
|
|
|
Appendix F – Total Cost for Evaluation
Evaluator ($50 per hour or $400 per day)
Interviews
(4 days) 1600
Survey
development (2 days) 800
Data
Analysis & Report Preparation (5 days) 2000
Per
Diem ($30 per day) 120
Travel
(about 800 miles) 344
Hotel
Accommodations ($100 per day) 200
Research
Assistance ($75 per 4 hours)
Survey associated and
clerical worker
supervision (48
hours) 900
Clerical
Worker ($75 per 8 hours)
Email & survey
associated (80 hours) 750
Total $6714
Evaluation
Title: __________
Evaluation Completion
Date:____________
|
Key issues and Recommendations |
Management Response* |
Tracking** |
||||
|
Response |
Key Actions |
Timeframe |
Responsible unit(s) |
Status*** |
Comments |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
* Unit(s) assigned to be responsible for the preparation of a
management response will fill the columns under the management response
section.
** Unit(s) assigned to be responsible for
the preparation of a management response will be updating the implementation
status. Assigned with an oversight
function monitors and verifies the implementation status.
*** Status of Implementation:
Completed, Partially Completed, Pending