Effects of Web-Based Training on End User Attitude and End User Compliance with Computer Usage Policy
by
Anthony P. Niemann
Submitted in partial fulfillment of
the requirements of ELFH 600
Fall Semester, 2005
Abstract
Email systems are
used extensively by companies, organizations, and government agencies. Records
management policies present these entities with significant problems that are
handled in a wide variety of ways, dependent on the culture of the organization.
This longitudinal study examined email records management policies of the
Effects of Web-Based Training on End User Attitude and End User Compliance with Computer Usage Policy
Email systems pose
significant problems for Information Technology (IT) departments, end users,
and organizations. IT department policy varies as to what users are required to
do. The German automotive and tool giant, Bosch, requires its 250,000 computer
users to delete email that is older than 60 days. If they do not, the computer system
will automatically delete the email without the users’ consent. It is public
knowledge that many companies have struggled with email retention policy in the
wake of debacles such Enron, Coca-Cola, Union Bank of
The size of the
mailbox determines how much mail can be stored before users are prohibited from
using the computer to send out new email. The more email the user is allowed to
save, the higher the costs in manpower, resources, and network bandwidth. Users
receive warning messages once the size of their mailbox approaches the
prescribed size allowed by the email administrator. Once the mailbox size limit
is exceeded, the computer runs slowly, or locks up completely. The user is not
only prevented from sending email, but often can’t use other applications. Users
at the Cabinet for Health and Family Services (CHFS) who have full mailboxes call
a technician working for the Office of Information Technology (OIT) to help
them. The reported problem may be about an application other than Outlook,
because the user only reports symptoms that s/he sees at that time. Once the
technician discovers that the problem is a mailbox that has reached the maximum
allowable size, the user is told that they must clean it out. If mailbox limits
are raised, the following additional network resources are often required:
storage space on the Exchange servers; time requirements to run tape backup
procedures before the start of the next business day; and costly Wide Area
Network (WAN) bandwidth between
Literature Review
Some organizations
provide training for employees through in-house, instructor-based training (IBT)
while others out-source training experiences. Some companies have turned to the
internet as a medium for delivery of training. In an article published by the
American Society for Training and Development (ASTD), Wulf (1996) reported that
internet-based training included one advantage that was particularly
significant: time and place independence. Employees at CHFS are spread
throughout the state of
We must find new
ways to address technology training needs in the workplace. Desai and Richards
(1999) recognized this when they derived a training model based on results of
“studies in cognitive psychology (
Conclusions about motivation and attitude were based on qualitative comments of participants and the large proportion of those in IBT verses the small number of participants in CBT. CBT trainees enjoyed freedom of schedule, although their frustration levels were higher when confronted with difficult material. Desai and Richards concluded that CBT training was more effective than IBT training when pre-performance was not considered. Employees choosing CBT were more independent and motivated to learn on their own, as the participants were allowed to choose the training method, so it was not surprising that those receiving CBT training out-performed employees receiving IBT training. If pre-performance was considered, the performance levels of trainees receiving IBT and CBT were not found to be significantly different. Desai and Richards (1999) found that training schedules and training methods were significant for long-term retention, and suggested that organizations must work closely with training providers to insure the quality of CBT training programs. Conclusions about the long-term effects of IBT and CBT were also attributed to whether the trainee utilized the product s/he was trained on after the training was over. The authors also identified a problem with being able to “sell” the CBT method of training to various organizations as a formal training tool. They suggested additional research in this area.
Concerns the authors have about their research can be addressed and empirically tested by future research if studies include a few critical elements. Participants could be randomly assigned to the CBT or IBT method of training. The CBT training material should be thoroughly tested to insure that it is free of problems, and is of the highest quality. End users should have a need to learn the content presented in CBT in order to perform their job if they are to receive maximum benefit from training. There should be an empirical method to measure the effectiveness and utilization of learning acquired. Internal and external participant motivations are also crucial elements in web-based learning strategies.
Techniques used to enhance motivation and studies that measure attitude during and after the training process may help to insure CBT training is utilized effectively. Orr, Allen, and Poindexter (2001) studied the relationship that demographic, educational, personality, and learning style variables had on computer users’ attitude and experience. In this study, 214 traditionally-aged college students were enrolled in six different sections of a university computer literacy course. Computer experience varied. Students were provided with a multimedia CDROM that included interactive features, and they gained software competency through CBT. Participant self-reported computer attitudes were compared at the beginning and end of the course using four instruments to measure potential contributing factors: Computer Attitude Scale (Loyd, 1984); Computer Experience (Orr, 2001); Personality Type (Keirsey, 1998); and Learning Style (Kolb, 1985) (as reported in Orr, 2001).
There were several hypotheses examined:
1. There is no change in computer attitude between the beginning and end of the course.
2. There is no significant relationship between computer experience and initial computer attitude, final computer attitude, or change in attitude.
3. There is no significant relationship between demographic/educational differences and initial computer attitude, final computer attitude, or change in attitude.
4. There is no significant relationship between personality type and initial computer attitude, final computer attitude, or change in attitude.
5. There is no significant relationship between learning style and initial computer attitude, final computer attitude, or change in attitude.
Hypotheses four and five were accepted, while two and three were rejected. Hypothesis one had partial support. The researchers failed to recommend applying this study to users in the workplace. They suggested that their findings may be applicable to training needs in the workplace, but this has not been effectively proven. The 214 participants were students, the oldest group listed as “27+”. End users in the workplace would likely have significant demographic/educational differences when compared as a group. Studies that include a practical assessment model is needed that can measure effects of demographic/educational differences among diverse groups of adult trainees in the workplace.
Alexander Astin’s Input-Environment-Output Assessment Model (I-E-O) allowed researchers to simultaneously analyze and compare the effects of student characteristics and instructional activities on achievement (as cited in House, 2002). House wanted to build upon the results of previous research that studied either student characteristics or specific instructional experiences. Students in this study included 721 traditionally aged students who started college five years prior to being surveyed. Data from this survey were merged with data about student participant’s initial characteristics when they began college five years earlier. Input variables included high school grade point average, academic self-concept, and achievement expectancies. Environment variables included studying / homework, classes / labs, and talking with faculty outside of class. The output measurement made in this study was the student’s self-reported average undergraduate grade. House found that four student characteristics and eight instructional experiences significantly correlated with achievement. House acknowledged that his analysis was conducted at a single institution, and that his students were traditionally-aged. One drawback in House’s study was that achievement was self-reported and no effort was made to verify grades the students reported. This could reduce the validity of his results. Perhaps the biggest contribution of his study was in providing a demonstration of the I-E-O model. It should be possible to apply this model to the adult worker and evaluate the effects of student characteristics and web-based instructional activities on achievement and attitude. Achievement could be self reported, as in House’s study, but it would be helpful to measure changes in attitudes of adult participants in the workplace to strengthen the external validity of the study.
Bhattacherjee and Premkumar (2004) reported that student attitudes and beliefs were key perceptions driving computer technology usage, and that these attitudes changed over time, dependent upon one’s experiences. They found that how learners view usefulness of technology learning had a strong positive effect on user intention to acquire and use this knowledge. This was the first known study of this type to be done, as far as the authors knew. Questions (theories) that were studied by the researchers included:
1. Do users’ beliefs and attitude regarding IT usage change over time?
2. What emergent constructs drive this change?
3. To what extent are these effects generalizable across technological and usage contexts?
Students from a large university took part in two longitudinal studies involving web based instruction. At the beginning of the semester, undergraduate IT students enrolled in multiple sections of a data communications course were introduced to web-enabled CBT by providing them with a description of the benefits to be derived and its relevance to their curriculum and professional training. They were guided through one CBT module to get started, and then given a pre-usage questionnaire about usefulness perceptions and attitude at the first time period (t1). Students were then asked to complete a short assignment that required them to use one of the CBT modules. Student participation from this point on was voluntary. Two additional questionnaires were voluntarily filled out at the second time period (t2) and third time period (t3). Study A had 189 end-user responses at the first time period, 175 responses at the second time period (two to three weeks later), and 172 responses at the third time period (nine to ten weeks after the second time period). They used web-enabled CBT in a self-paced structured learning environment for their data communications class. It was believed, from studies cited by the authors, that learning retention increased while learning time decreased by using web-based training. Students in the study had no prior experience with CBT. Study B involved 77 graduate students enrolled in an electronic commerce course using Rapid Application Tool Usage software (RAD). Perceived usefulness and attitude were compared at three time periods in Study A and two time periods in Study B. Responses by each student were matched to create a single record so that changes in belief and attitude could be compared.
Constructs of disconfirmation and satisfaction influenced the perceived usefulness and attitude that was measured by evaluating student’s responses on the questionnaires. The authors of this study designed questionnaires that allowed them to measure disconfirmation and satisfaction. The theory behind the use of these constructs were based on a study by Oliver (1980) called Expectation Disconfirmation Theory (EDT). EDT is an extension of cognitive dissonance theory (CDT) and had been used by researchers to understand consumer satisfaction, complaining behaviors, and, most recently, IT usage (Bhattacherjee, 2001). Disconfirmation in this study referred to the extent that pre-usage expectations of using technology changed after actually using the technology.
Five constructs were of interest in this study: perceived usefulness; attitude; intention; disconfirmation; and satisfaction. The first two constructs were measured at t1, t2, and t3 while the latter three constructs were measured at t2 and t3. Usefulness, attitude, and intentions were measured using pre-validated scales adapted from prior IT usage literature. Usefulness was measured via a four-item Likert scale using questions that examined perceptions of performance, productivity, effectiveness, and overall usefulness. Attitude was measured using a four-item semantic differential scale that employed adjective pairs. Usage intention was measured using two Likert-scaled items that asked participants if they intended to continue using web-based learning tools.
The research design consisted of two longitudinal studies: the first study examined CBT usage and the second studied RAD software usage. Quantitative data gathered at three time intervals were analyzed and compared to discover whether the participants’ attitude and perceptions about the usefulness of computers changed over time. The final stage of the first empirical analysis was a qualitative analysis of open-ended questions on a questionnaire. In the second part of this study, the authors collected information on RAD software usage across two time periods. This section of the study examined the same theoretical questions and used the same constructs as the first section of the study.
By analyzing data from longitudinal study questionnaires, the authors were able to make comparisons of changes in satisfaction and attitude. The authors found that users’ beliefs and attitudes do change over time, and that these effects were generalizable across technological and usage contexts. The authors acknowledged that using student subjects was a limiting factor in confirming the generalizability of their findings in the workplace. Longitudinal studies are difficult to accomplish in the workplace, but further research needs to be done in this area to discover how training would affect users’ attitudes and satisfaction when applied in actual job settings. The authors acknowledged that longitudinal studies have inherent threats to internal validity. They also stated that performance was excluded from this study because of the possibility of confounding the results. Only one usage-related belief was examined: perceived usefulness of IT usage. Although the findings of this study were significant, so were the limitations. Two limitations that seemed to be especially significant included the failure to measure performance and the use of students as participants. It is difficult to understand how the authors could claim that users’ beliefs and attitude were generalizable across technological and usage contexts as a result of this study. There is a dearth of studies that use adults in the workplace as participants and measure performance change after application of web-based training. The authors used self-reported results from students who filled out questionnaires at various time periods in these two studies to measure attitude change. Student performance is usually measured through grades, but in the workplace performance must often be assessed by other methods.
Performance was the focus of a study by Jawahar and Elango (2001). This study included 467 undergraduate students in three different sections of a Management Information Systems course. Students in these classes were taught by the same instructor at a large state university. There were 207 male and 224 female students, and the majority of the students were between 20 and 23 years old. The authors studied the effects of attitudes, goal setting, and self efficacy on end user performance. The theories behind the study were: attitudes about working with computers will be positively related to end user performance; specific and challenging goals will be positively linked to end user performance; and self-efficacy will be positively related to end user performance. The authors asserted that computer anxiety and negative attitudes toward computers are two distinct concepts. Students reported their opinions over the time frame of one semester by answering various assessment instruments that measured attitudes, goal-setting, and self-efficacy. The dependent variable of performance included objectively scored lab assignments and exams on conceptual material, and the authors found support for all three theoretical hypotheses. Implications for practitioners included the following:
1. Improvements in attitudes, goal-setting, and self-efficacy can positively influence achievement. This issue signifies the inclusion of motivation, both internal and external, as a valid construct in future research.
2. Individuals do poorly when they feel they have no control.
3. Goal setting is very important to the transfer of learning.
4. Specific attitudes are more easily influenced than general attitudes.
Jawahar and Elango (2001) concluded that “attitudes toward working with computers,
goal-setting, and self-efficacy are entwined with motivation of end users”. They suggested additional studies to discover if results of this study will generalize to end users from organizations in the workplace. They also suggested examining additional factors having a potential to influence end user performance.
It has been demonstrated that attitudes of end users are positively correlated to motivation of end users (Desai, 1999). Studies such as the one conducted by Jawahar and Elango (2001) have also shown that self-efficacy and goal setting are correlated to end user performance. The five articles presented in this literature review represent studies designed to measure effectiveness of web-based training, especially as it correlates to attitude, student characteristics, and performance. Web-based training can be effectively and economically delivered in geographically diverse organizations using a flexible training schedule (Wulf, 1996). Web-based training can target unique requirements that an organization places on end users not available with off-the-shelf training products (Munger, 1996).
Hypotheses
This study addresses the effects of web-based training on attitude and end user compliance with organizational policy requirements. The end users in the government organization that was examined were considered to be novice users. They had been trained to use mainframe applications, but had little expertise in using Microsoft Office applications. It was theorized that if career end users are given training that allows them to do there job more easily and efficiently, attitude and compliance with policy should be positively affected. There have been few studies of this type using adults in the workforce as participants. How will end user attitude be affected by web-based training? The dependent variables in this study are end user attitude and end user compliance with computer usage policy. The independent variable is web-based training. The following questions were examined:
Research
Question One
Does web-based training
significantly affect end user compliance with computer usage policy?
Null Hypothesis: There will be no significant relationship between web-based training and end user compliance with CHFS computer usage policy.
Research hypothesis: Compliance with CHFS computer usage policy will be significantly improved after CHFS employees receive web-based training.
Research Question Two
Does web-based training significantly and positively affect end user attitude toward computer usage policy?
Null hypothesis: There will be no significant relationship between web-based training and end user attitude.
Research hypothesis: Attitude will be significantly improved as a result of web-based training.
Research
Question Three
Is there a positive correlation between attitude and compliance with CHFS computer usage policy?
Null hypothesis: There is no significant relationship between attitude and compliance
with CHFS computer usage policy.
Research hypothesis: There will be a significant and positive relationship between attitude and compliance with computer usage policy for those who receive web-based training.
Methods
Sampling
A
criterion-based method was used to select participants for this study. Participants
were 750 state employees from a total population of 35,000 employees whose
mailbox sizes were the greatest amount. This was determined by daily automated
reports generated by Microsoft Exchange servers located in
Instruments
Threats to internal
and external validity as a result of instruments used for observing and
measuring mailbox size were minimized by using automated reports generated by
OIT personnel who administered Microsoft Exchange Servers. Reports were imported
into an SPSS spreadsheet on a daily basis for the entire twelve week period of
the study. End users were unaware that storage limits were being tracked.
Demographic data on 750 participants were made available by the Personnel Department
for the
A four item
questionnaire was developed from pre-validated scales for measuring attitude by
using prior IT usage literature (Bhattacherjee, 2004) and given to group A at
week five, before groups A and B received web-based training. The same
questionnaire was given to groups A, B, and C at week ten, after groups A and B
had received web-based training. Demographic data of all users were examined
for positive or negative correlation with attitude, as reported on the
questionnaires that had been filled out. In addition, demographic data of users
in group A were examined for correlation to changes in attitude as reported on
the before and after questionnaires that they filled out at weeks five and ten.
Attitude was measured by four Likert-scale items. The actual questionnaire is
included in Appendix A.
1. By using web-based training I can learn how to manage my mailbox.
2. Using web-based training to learn how to manage my mailbox is a good idea.
3. The problem is not the mail I keep, but the policy that sets limits on it.
4. I know how to manage my mailbox size.
The responses to the four items were measured by using a five-item Likert scale with the following possible responses: Strongly Disagree; Disagree; Neutral; Agree; and Strongly Agree. Interpretation of attitude based on responses for the four questions was as follows:
1. Strongly Disagree equates to the most negative attitude and Strongly Agree equates to the most positive attitude. Selection of Neutral equates to a null.
2. Strongly Disagree equates to the most negative attitude and Strongly Agree equates to the most positive attitude. Selection of Neutral equates to a null.
3. Strongly Disagree equates to the most positive attitude and Strongly Agree equates to the most negative attitude. Selection of Neutral equates to a null.
4. Strongly Disagree equates to the most positive attitude and Strongly Agree equates to the most negative attitude. Selection of Neutral equates to a null.
Research Design
Web-based
training for this study was developed by
Pre-training questionnaires and post-training questionnaires for Group A were compared for changes in attitude. The post-training questionnaires for two groups that received web-based training, Groups A and B, were compared to determine if there were differences in attitude. The questionnaire responses for Group C were compared to responses for Groups A and B in order to measure differences in attitude based on the dependant variable, web-based training. It was expected that responses for Groups A and B would show a more positive attitude when compared to results for Group C. Demographic differences could also cause some measured differences, and demographic data would be checked for positive or negative correlation to attitude as well as to compliance with computer usage policy. Detailed statistical results of this study will be available by emailing the author, Anthony P. Niemann at apniem01@louisville.edu or tony.niemann@elkcreek.net.
Future Research
Web-based training is under-utilized in a workforce that requires technical training to compete competently in the global economy. We have off-the-shelf products and training organizations that specialize in training users how to use Microsoft Office applications, but there are few off-the-shelf products available that can train users how to comply with requirements set forth by organizational policy when using these products. Few organizations have training departments within their organization that design their own systematic instruction for these products.
Many studies have been conducted that examine web-based training in association with attitude, student characteristics, and performance. Most have used university students as participants, but few are available that involve adults in the workforce. If it can be demonstrated that end-user behavior and attitude in a statewide government organization are significantly changed by a systematically designed web-based training program, additional studies might be implemented that measure the association of web-based training and self-efficacy in conjunction with their relation to compliance with organizational computer usage policies in the workplace, specifically, in a government organization. Future research could examine demographic data that correlates positively or negatively to changes in attitude and mailbox size to determine if factors other than web-based training can account for, or impede, changes.
Finally, additional research could compare the effects of web-based learning and instructor-led learning on attitude, self-efficacy, and performance. Demographic variables could also be factored into future research. It is critical that we discover the nature of the strengths and weaknesses of web-based learning if we are to take advantage of this important resource.
References
Bhattacherjee, A., & Premkumar, G. (2004). Understanding changes in belief and attitude
toward information technology usage: A theoretical model and longitudinal test [Electronic Version]. MIS Quarterly, 28(2), 229-254.
Bird, D. (2002). Effectively managing the GroupWise message store [Electronic Version].
Inside NetWare, 11(7), 1, 6p.
Desai, M. S., & Richards, T. (1999). End-user training: A meta model [Electronic Version].
Journal of Instructional Psychology, 26(2),
74, 11p.
Haverson, D. (2002). Records management: The digital dilemma [Electronic Version].
Transform Magazine, 11(3), 32, 6p.
House, J. D. (2002). The independent effects of student characteristics and instructional
activities on achievement: An application of the input-environment-outcome assessment
model. [Electronic Version]. International Journal of Instructional Media, 29(2), 225,
15p.
Jawahar,
end user performance. [Electronic Version]. Journal of End User Computing, 13,(2), 40,
6p.
M2 Presswire (2005). LAN 2 LAN: LAN 2 LAN takes the pressure off email. Retrieved
September 11, 2005, from http://proquest.umi.com.echo.louisville.edu.
Munger, P. D. (1996). A guide to high-tech training delivery: Part 1. [Electronic Version].
Training &Development, 50(12), 55, 3p.
Oliver, R. L. (1980). A cognitive model for the antecedents and consequences of satisfaction.
[Electronic
Version]. Journal of Applied Psychology, 17,
p. 460-469
Orr, C., Allen, D., & Poindexter, S. (2001). The effect of individual differences on computer
attitudes: An
empirical study. [Electronic Version]. Journal
of End User Computing,
13,(2), 26, 14p.
Wulf, K. (1996). Training via the
internet: Where are we? [Electronic Version]. Training &
Development, 50(5), 50, 6p.
|
Statement |
Strongly Disagree |
Disagree |
Neutral |
Agree |
Strongly Agree |
|
Web Based Training |
|||||
|
By using web-based training I can learn how to manage my
mailbox. |
|
|
|
|
|
|
Using web-based training to learn how to manage my mailbox
is a good idea. |
|
|
|
|
|
|
Email |
|||||
|
The problem is not the mail I keep, but the policy that
sets limits on it. |
|
|
|
|
|
|
I know how to manage my mailbox size. |
|
|
|
|
|
|
Additional Comments: |
|||||