F E A T U R E

A R T I C L E

The Technology
Acceptance Model
Predicting Nurses’ Intention to Use
Telemedicine Technology (eICU)

YANIKA KOWITLAWAKUL, PhD, RN

BACKGROUND/SIGNIFICANCE

Information technology has been used in healthcare de-
livery systems to improve patient safety and patient
care outcomes worldwide.1–3 The importance of infor-
mation technology was recognized by the Institute of
Medicine4 in 2000 when they released the report entitled,
To Err Is Human; the report recommended increased
efforts to incorporate information technology into the de-
livery of patient care, and since that time, there has been
a remarkable effort on the part of many organizations
(Leapfrog Group, the National Patient Safety Foundation,
the Institute for Healthcare Improvement, and the Joint
Commission) to improve patient safety by supporting the
use of information technology. All of these organizations
have encouraged the implementation of information tech-
nology to prevent human error.5

A high incidence of adverse events and medical errors
has been found in critical care settings (ICUs).6–8 The
Leapfrog Group, the National Quality Forum, and the
Agency for Healthcare Research and Quality have all
recommended that the ICUs be staffed exclusively with
board-certified critical care physicians (intensivists) who
will respond immediately to provide patient manage-
ment, thus decreasing medical errors and adverse events
and reducing hospital mortality rates.7,8

A shortage of critical care physicians and nurses makes
it difficult to comply with the recommendation. During
the night shift and weekend hours, it is more difficult
to have critical care physicians covering for the ICU
patients.8 Therefore, it has been proposed that tele-
medicine technology, eICU (remote ICU or electronic

ICU) be used as a possible alternative solution that allows
critical care nurses and physician intensivists to monitor
ICU patients from off-site locations. The rationale is that
patients can be more intensively monitored, thus decreas-
ing adverse events and improving patient outcomes.5,6,8

As of 2008, this technology, eICU, was implemented
in 28 states covering more than 200 hospitals and 40
healthcare systems.9 According to several studies,6,8–11

the eICU technology system has many advantages over

CIN: Computers, Informatics, Nursing & July 2011 411

CIN: Computers, Informatics, Nursing & Vol. 29, No. 7, 411–418 & Copyright B 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins

The purposes of this study were to determine

factors and predictors that influence nurses’ in-
tention to use the eICU technology, to examine
the applicability of the Technology Acceptance

Model in explaining nurses’ intention to use the
eICU technology in healthcare settings, and to
provide psychometric evidence of the measure-

ment scales used in the study. The study involved
117 participants from two healthcare systems. The
Telemedicine Technology Acceptance Model was
developed based on the original Technology Ac-

ceptance Model that was initially developed by
Fred Davis in 1986. The eICU Acceptance Survey
was used as an instrument for the study. Content

validity was examined, and the reliability of the
instrument was tested. The results show that per-
ceived usefulness is the most influential factor that

influences nurses’ intention to use the eICU tech-
nology. The principal factors that influence per-
ceived usefulness are perceived ease of use,
support from physicians, and years working in the

hospital. The model fit was reasonably adequate
and able to explain 58% of the variance (R

2
=

0.58) in intention to use the eICU technology with

the nursing sample.

K E Y W O R D S

Acceptance & eICU & Nurses’ attitude &

Nurses’ intention & Nursing & Technology

Author Affiliation: School of Nursing, The George Washington Univer-
sity, N.W. Washington, DC.

Corresponding author: Yanika Kowitlawakul, PhD, RN, School of
Nursing, The George Washington University, Virginia Campus, 45085
University Drive, Suite 201K, Ashburn, VA 20147 ([email protected]).

DOI: 10.1097/NCN.0b013e3181f9dd4a

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

the traditional ICU systems that exist today, such as
decreasing in-hospital mortality, decreasing ICU length
of stay, decreasing cost, and increasing high quality
of care.

The eICU unit is a secure telemedicine center where
a team of critical care physicians and nurses provides
oversight surveillance for the patients in off-site inten-
sive care units. This monitoring utilizes various technol-
ogies such as video assessment, two-way communication
directly into the patient’s room, access to hospital in-
formation systems, and the use of eCareManagement
(VISICU Inc, Baltimore, MD). Data, including vital signs
from bedside monitors, intake/output, blood glucose,
laboratory results, and current medications are all in-
terfaced with the computer database system. Thus, the
eICU team can review all of the medical data through the
computer system and have immediate communication
with and access to the on-site nurses and physicians.
Bedside nurses have the role of closely monitoring the
patient and cooperating with the eICU team in assessment
and management.

The intent of this study was to determine significant
factors and predictors of nurses’ intention to use (ITU)
the telemedicine technology (eICU) and to provide psy-
chometric data, further supporting the evidence for the
reliability and content validity of the measurement scale
strategy. Nurses have been identified as computer end
users in healthcare settings. In the 1980s, they were often
apprehensive about integrating computer systems into their
nursing practice.12 Recently, a common fear of nurses is
that the information technology will take over their roles,
and their roles might be replaced by machines.1 Also,
nurses’ frustrations when using a new technology system
have been associated with the fears of increasing work-
load and an unfriendly technical system.13 Stafford et al14

conducted an ethnography study and examined the col-
laborative communication between on-site (bedside) and
off-site (eICU) nurses. These researchers found that the
on-site nurses felt uncomfortable, as if they were being
watched; some were resentful, and these nurses ques-
tioned the eICU nurses about their commitment to nurs-
ing practice.14

Based on the author’s critical care experiences, nurses
complained when they received a telephone call from
the eICU nurses. On-site nurses felt as though someone
was watching over their shoulder. They complained about
having to do extra work, losing autonomy, receiving
contradictory orders from two different doctors (from
on-site and off-site), and creating duplicate documen-
tation. While several nurses had positive attitudes to-
ward eICU, they struggled when they had to deal with
technical problems with the computer program. As a
result, some nurses were not willing to communicate
with the eICU nurse and refused to take advice from the
eICU team.

The successful implementation of clinical information
technology systems is highly dependent on user accep-
tance.15 Patient safety and quality will not be achieved
if nursing staff are not willing to use the technologies.15

Therefore, it is critical to understand the readiness and
willingness of hospital nurses to implement new infor-
mation technology and efficiently integrate it into their
practice.16

One model that has attempted to explain acceptance of
technology systems is the Technology Acceptance Model
(TAM); according to TAM, the intention generates the
actual behavior.17 Thus, behavior should be predictable
from measures of behavioral intention and other factors
that influence intention directly and indirectly.18 Since re-
cognizing that the ITU new technology has become a very
significant issue, it is conjectured that TAM may be a useful
conceptual framework for examining nurses’ ITU tele-
medicine technology.19

CONCEPTUAL FRAMEWORK/
RESEARCH GAP

The TAM was initially developed by Davis et al in 1986.17

The TAM provides a framework for understanding the
determinants of computer acceptance that explain user
behavior with a variety of end-user populations. The the-
oretical framework has the potential to identify, explain,
and predict the factors, such as internal beliefs and atti-
tudes, which have an effect on the intentions of tech-
nology end users.17,18

The original TAM has five constructs: perceived use-
fulness (PU), perceived ease of use (PEOU), attitude toward
using, ITU, and actual system use.17 The key determinants
of computer acceptance in TAM are the belief that the
computer system will help to improve job performance
(PU) and the belief that using the computer system re-
quires only a minimal level of mental effort; in other
words, it is easy to use.17 Those two determinants are con-
sidered to be the basis for evaluating the attitudes toward
using particular computer systems and ultimately generat-
ing the ITU. The ITU a particular system then leads to
actual end-user behavior.

The TAM has evolved over time, being used with
different populations and various technology systems;
the framework has been used extensively in information
technology, education, and business.20–29 However, there
have been few studies in healthcare settings,20,22,25,27

and even fewer with regard to nursing practice.16,25,30

While nurse researchers have focused on developing and
testing instruments that measure nurses’ attitudes toward
using new computer technology,1,2,12,13,31–33 very few
studies have examined the ITU the technology in the
practice of nursing.16,30

412 CIN: Computers, Informatics, Nursing & July 2011

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

PROPOSED CONCEPTUAL MODEL

A number of nursing studies have demonstrated that the
primary factors influencing nurses’ attitudes toward
using computers are age, years of nursing experience,
and years of experience with computers.3,16,33 Based on
interviews with five nursing experts in the field and pre-
vious studies, three additional potential factors that in-
fluence the individual’s belief and attitudes toward
using telemedicine technology (eICU) were the number
of years working in the hospital, support from admin-
istrators, and support from physicians.16,34 According
to Shoham and Gonen,16 support from the nursing ad-
ministrator is a significant factor of influence on the
nurses’ intentions to use the computers. Taken together,
these six factors were selected as the external variables
in the proposed conceptual model that underpins the
current study.

The TAM provides the related constructs of PU,
PEOU, attitude toward using, and ITU. PU and PEOU
are the known primary key determinants for computer
acceptance behaviors. Both determinants influence atti-
tudes, leading to ITU and ultimately to actual individual
usage behavior.17

Interestingly, according to the Pearson correlation
results in this study, there were no statistically signifi-
cant correlations with the three constructs that were
drawn from the original TAM: age, years of experience
in nursing, and years of experience with computers.
The literature reveals that the relationships between the
latter variables and attitudes toward computerization
were unclear with most of the studies,3,15,32 having been
done in different settings and with different computer
programs. The eICU technology has been implemented
in nursing practice for only a few years.9,14 It would
appear that as technology changes and the demands of
practice change, the attitudes of nurses will be shaped

and formed by their earlier interface with technology,
as well as their experiences with technology in nursing
education.

In the current study, the author sought and received
permission from the creators of the TAM to revise the
original model, and renamed it the Telemedicine TAM
(TTAM; Figure 1). The TTAM utilizes four constructs
drawn from the original TAM (PU, PEOU, attitude
toward using, and ITU) and the three previously
identified external variables (years working in the hos-
pital, support from administrators, and support from
physicians). Age, years of experience in nursing, and years
of experience with computers were omitted from the
model because they were not statistically supported by
the data.

RESEARCH QUESTIONS

1. What are the relationships among the external variables,

the key constructs of TTAM, nurses’ attitude toward

computerization, and the ITU telemedicine technology

(eICU)?

2. Which variables are most influential in predicting ITU

telemedicine technology (eICU)?

3. Is the proposed hypothesized model consistent

with the empirical data in the study (in other words,

goodness-of-fit)?

METHODS

Measurement

The eICU Acceptance Survey was used as an instrument
for the study. The instrument was modified from three

FIGURE 1. TTAM with the results of R2 and ” values.

CIN: Computers, Informatics, Nursing & July 2011 413

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

original instruments, which are (1) PU and PEOU,35 (2)
Nurses’ Computer Attitudes Inventory,31 and (3) ITU.20

The content validity was examined by five nursing tech-
nology experts. The questionnaire for the experts con-
sisted of 10 questions. The experts were asked about the
content of each construct for clarity, appropriateness, the
relationship of overall items, and whether those items
measured the constructs. The experts were given 1 week
to complete the questionnaire, and then the researcher dis-
cussed the results with each expert. The results from five
experts showed that they all agreed or strongly agreed on
each item. Therefore, overall results of content validity for
this study were satisfied.

The internal consistency of the instrument constructs
was evaluated using coefficient ! (Cronbach !), which
showed ranges of .91 to .96 with a total coefficient ! of
.96 (Table 1).

Sample

Potential nursing participants were RNs, employed in
critical care units, where the eICU technology had not
yet been implemented. The nurses employed in these
units had a nurse-to-patient ratio of 1:2. Nurses in this
study were caring for patients who were in critical con-
ditions that might require life support, such as a ven-
tilator or vasopressive medications. Nurse managers or
directors of the critical care units and the nurses who
had worked in the unit that has implemented eICU tech-
nology before were excluded.

There were 139 potential participants in two metro-
politan healthcare systems; of these, 131 nurses partici-
pated for a 94% response. Of the 131 responses, three
(2.16 %) were excluded because of incomplete answers,
and 11 (7.19%) did not meet the criteria (two partici-
pants were managers, and nine participants were not
employed in the critical care units). The final sample was
composed of 117 nurses. Given a moderate effect size
(0.13) with an ! of .05 and a power level of 0.80, the sam-
ple of 117 was adequate for multiple regression and path
analysis based on Marsh and colleagues (1988) as cited in
Kelloway,36 Bollen,37 and Harris and Schaubroek.38

Data Collection/Data Analysis

The study was implemented after approval by the human
subjects review boards of the participating university
and the two healthcare systems. Nurses who indicated
a willingness to participate were provided a cover letter,
an informed consent form, and the study questionnaire
in a personal meeting with the researcher. During the
meeting, the researcher explained the purpose of the
study and reviewed the informed consent form and
the questionnaire. Signed participant consent forms were
placed in an envelope that was kept separate from the
questionnaire. Participants were instructed to complete
the questionnaire and return it to the researcher in a
sealed envelope. Once collected, both consent forms and
questionnaires were stored separately in a locked drawer.

Descriptive statistics, regressions, and multiple re-
gressions (for path analysis) were accomplished with
the SPSS version 15.0 data analysis software (SPSS Inc,
Chicago, IL). LISREL 8.8 (Scientific Software Interna-
tional, Lincolnville, IL) was used to test the goodness
of fit with the proposed model (TTAM). Statistical sig-
nificance for all of the analyses was set as P G .05. Data
screening was performed for missing data and outliers,
and the assumptions of multiple regression analysis meth-
ods were considered.39,40

RESULTS

Descriptive Analysis

The average age of the participants was 35.4 (SD, 9.37)
years. There were 107 women (91.5%) and 10 men
(8.5%). Most of the participants were employed on the
day shift (56 [47.9%]), whereas 35 (29.9%) were em-
ployed on the night shift, and 26 (22.25%) worked both
day and night shifts. Sixty-four of the participants
(54.7%) had bachelor degrees in nursing, 31 (26.5%)
had associate degrees in nursing, and three (2.6%) had
graduate degrees in nursing. Most of the participants
(54 [46.2%]) had heard about eICU from nurses who
had prior experience with eICU technology; however,
39 (33.3%) had heard about eICU from both nurses
who had prior experience and nurses who had not used
the technology. Only one participant had learned about
eICU technology from the Internet.

Most participants (101 [86.3%]) had never attended
a conference on eICU technology, and 98 (83.35%) had
never been trained to use eICU technology. There were
only 16 participants (13.7%) who reported that they
had attended a conference on eICU technology and 19
(16.2%) who had been trained to use the eICU tech-
nology before they worked in their current units. Most
of the participants (110 [94%]) reported that there were

T a b l e 1

Reliability in This Study

Constructs
No. of
Items

Item
Mean Coefficient !

PU 7 2.8 .96
PEOU 6 3.4 .94
Nurse attitudes
toward eICU (NATE)

21 3.3 .91

Intention to use 6 3.2 .95
Total 40 3.2 .96

414 CIN: Computers, Informatics, Nursing & July 2011

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

technology support personnel available in their units,
while seven (6.0%) reported there were no available sup-
port personnel.

The participants had worked in nursing an average
of 10.44 (SD, 9.13) years, and in the present hospital for
6.83 (SD, 6.98) years. The average number of years for
working in the critical care units was 7.64 (SD, 7.82) years.
The average number of years that nurses had known about
eICU was 2.41 (SD, 1.45) years, and the average number of
years working with any type of computer technology was
14.77 (SD, 5.97) years.

Path Analysis

A path analysis was performed to determine the causal
effects among the variables in the proposed model, TTAM.
Multiple regressions identified four paths based on the
assumptions of the causal closure of the path diagram
(Figure 1).

According to the results of regression 1, the follow-
ing path coefficients were statistically significant: years
working in the hospital to PU (” = j.200, P = .010),
support from physicians to PU (” = .270, P = .003), and
PEOU to PU (” = .420, P = .000). The path coefficient
of support from administrators to PU was not signifi-
cant (” = .051, P = .576).

The results of regression 2 provided a path coefficient
from support from administrators to PEOU that was sta-
tistically significant (” = .242, P = .009).

The results of regression 3 provided two path coef-
ficients that were statistically significant: from PU to atti-
tudes toward using (” = .297, P = .000) and from ease of
use to attitudes toward using (” = .466, P = .000).

Finally, the results of regression 4 provided two path
coefficients that were statistically significant: from PU
to ITU (” = .506, P = .000) and from attitudes toward
using to ITU (” = .364, P = .000).

Goodness-of-Fit/Fit Indices

LISREL provided an analysis of ‘‘fit’’ index values that
was used to examine and determine the model fit for the
data collected in this study. The overall model fit was
guided by using multiple fit indices as suggested in the
literature20,36,41–43 and is presented in Table 2. The re-
sults show that the model fit was reasonably adequate.
Furthermore, the LISREL program provided the results
of squared multiple correlations for structural equa-
tions (R2) that explain the power of the model for indi-
vidual constructs.

Together, years working in the hospital, support from
physicians, support from administrators, and PEOU ex-
plained 35% of the variance observed in PU. Support
from administrators explained only 6% of the variance

observed in PEOU. However, PU and PEOU were able
to explain 44% of the variance observed in nurses’
attitudes toward the eICU technology system. Finally,
PU and nurses’ attitudes toward using eICU technology
were able to explain 58% of the variance observed in
ITU in the eICU technology.

DISCUSSION/IMPLICATIONS

This study evaluated the usefulness of the TTAM in ex-
plaining nurses’ ITU eICU technology systems. The TTAM
was able to explain 58% of the variance in ITU eICU,
and the ITU was predicted by PU and attitude toward
using. Compared with a prior TAM study in nurs-
ing settings,30 the TTAM in this study appeared to be
more useful in explaining the intention of nurses to use
the technology.

According to the model fit results, the model fit in-
dices were within the reference range. Therefore, the
TTAM in this study appears to adequately specify the
intention of nurses to use the eICU technology system
and has the ability to reproduce a correlation matrix
with this nursing sample as did the original TAM. The
suitability and applicability of TTAM in the nursing
sample were confirmed as indicated by reasonable
model fit indices. Nevertheless, ‘‘it is important for re-
searchers to recognize that ‘model fit’ does not equate
to ‘truth’ or ‘validity.’ Finding the expected pattern of
correlation is a necessary but not sufficient condition
for the validity of the theory that generated the model
predictions.’’36(p40)

In terms of prediction, in agreement with the original
TAM and previous studies20,30,35,43 PU was found to be
a key determinant that has a statistically significant and
strong influence on nurses’ intentions to use the eICU
technology system. This suggests that nurses in criti-
cal care units tend to focus on the usefulness of this

T a b l e 2

Analysis of Overall Model Goodness-of-Fit Using
Common Fit Indices

Model Goodness-of-Fit Indices

Recommended

Value

Results Obtained
From This Study

Goodness-of-fit index Q0.90 0.95

Adjusted goodness-of-fit
index

Q0.80 0.85

Normalized fit index Q0.90 0.94

Non-normalized fit index Q0.90 0.91
Comparative fit index Q0.90 0.96
Root-mean-square

residual

e0.10 0.05

Root-mean-square error
of approximation

e0.10 0.10

CIN: Computers, Informatics, Nursing & July 2011 415

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

technology itself. In this study, PU was significantly im-
pacted by PEOU as TAM hypothesizes, contrary to what
Hu and colleagues20 found, namely, that PEOU had no
significant effect on PU of the telemedicine technology.
However, the population of the study of Hu et al20 was
not nurses, but physicians. The nature of the population
might well play a role for this contradictory result.

According to the results of this study, PEOU was found
to have more significant effect on nurses’ attitudes toward
using the eICU technology system than PU. This result
could reflect that most of the critical care nurses in the
current study were familiar with using computer technol-
ogy equipment (average number of years using computers
was 14.77). They had already been charging, ordering,
documenting, and using computers with various types
of software programs. Moreover, 94% of nurses in this
study reported that they had personnel support in their
facilities to help them solve technical problems whenever
they were struggling with new technology features or
operations.

Nurses’ attitudes toward the eICU technology system
were also a significant factor that predicted the ITU in
this technology, even though it contributed less to pre-
dicting the ITU in this technology than PU. The results
demonstrated that nurses’ attitudes were also relatively
important to nurses’ ITU in the eICU technology system.
Nursing practice requires particular knowledge and skill
in dealing with patient care within the critical care en-
vironment. Nurses are very focused on their patients
because those patients are acutely ill. Any new technol-
ogy that appears to take the nurses away from patient
care would lead to the belief that it would not be useful
or helpful. The TTAM does appear to explain the fac-
tors that influenced nurses’ attitudes toward using the
eICU technology system in a manner different from the
original TAM. This may be due to the unique character-
istics of the healthcare setting and the nature of the nurs-
ing profession.

According to the results, the numbers of years working
in the hospital had a negative statistically significant cor-
relation with PU, suggesting that nurses who worked in
the hospital longer might believe less in the usefulness of
the technology. Nurses who worked longer might need to
receive information at the early stage of implementation.
Therefore, an information and training program that out-
lines the purpose, usefulness, and features of eICU tech-
nology should be provided. This program would outline
effective communication strategies among healthcare pro-
viders to foster better communication and avoid conflicts.
The information and training program would primarily
focus on how the eICU technology system can help im-
prove patients’ safety and outcomes.

PU was also influenced by support from physicians.
Physicians had a great impact on nurses’ belief about how
this technology would be beneficial for their patients.

Therefore, to increase nurses’ ITU in the eICU technology
system, it is necessary to have support and cooperation
from the physicians as suggested by the findings. Com-
munication between physicians and nurses needs to be
clear and have precise direction. A physician who em-
braces computer technology must be selected as the one in
charge of patient care management and should be iden-
tified before the eICU technology system is implemented.
Protocols that outline the plan of escalation of treatment
must be clearly stated, thus enabling nurses to deliver
quality care and promote patient safety.

The support from administrators, an external variable
from this study, showed a significant influence on PEOU,
but not a significant influence on PU. Most of the ad-
ministrators or directors in critical care units were RNs.
They have similar backgrounds with nurses who work at
the patient’s bedside. However, the nursing administra-
tors had a different focus on the eICU technology system.
Their focus was on how to provide all nurses with user
support and proper training before the technology was
implemented. The administrators often reassure the nurses
that using this technology would not be difficult and that
it would improve patient outcomes.

Prior to introducing the new eICU technology sys-
tem to intensive care units, administrators can increase
the ITU the technology system by assessing nurses’ and
physicians’ perceptions. Nurses and physicians should be
involved in the planning and implementation process.
Since PEOU is a main factor in predicting attitude to-
ward using, nurse administrators could support this fac-
tor by having on-site user training and reassuring the
staff that they will have personnel support at the units at
all times.

LIMITATIONS AND RECOMMENDATIONS

Participants in this study were volunteers and subject
to self-selection biases. Additional research is needed to
address construct validity with a larger sample size and
improved model fit. The indirect path coefficient (ease of
use and ITU), which is statistically significant, and the
direct path coefficient (support from administrator and
usefulness), which is not statistically significant, may need
to have further investigation for model modification to
improve the ‘‘fit’’ of the research model.

The external variables (years working in the hospital,
support from administrators, and support from phy-
sicians) were the primary factors that influenced the
two key determinants (PU and PEOU). In the health-
care setting, there might have been more than three factors
that influenced those two key determinants specified in the
TTAM. More investigation on external variables, such as
knowledge, participation in the decision-making process,
peer support, and individual awareness, is needed.

416 CIN: Computers, Informatics, Nursing & July 2011

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

CONCLUSION

To improve the nurses’ ITU the eICU technology system,
cultivating PU and attitudes toward using this technol-
ogy are important. In critical care units, nurses have
high autonomy and are competent in patient care. Since
patients’ outcomes are of utmost concern, it behooves
administrators to support the autonomy and compe-
tency of the nurses by providing them with the educa-
tional opportunities that allow them to adapt to new
technology and new environments.

In addition, this nursing study used and adopted a
theoretical model and instruments that have been de-
veloped from the discipline of social psychology. The
TTAM shows promise as a valuable model for predict-
ing nurses’ ITU in the eICU technology in healthcare as
demonstrated by the fact that 58% of the variance (R2 =
0.58) in ITU the technology is explained by the model
in this study. Furthermore, the study suggests that the
TTAM has applicability in identifying, explaining, and
predicting the ITU telemedicine technology in nursing
practice. The replication of this study is also highly
recommended.

Acknowledgments

The author thanks Dr Catharine A Kopac, associate
professor at the George Washington University, and
Dr George Crossman for editing the article. Also, the
author thanks Dr Jean Burley Moore, Dr Heibatollah
Baghi, and Dr Gregory Guagnano, George Mason Univer-
sity, who have assisted me throughout the study.

REFERENCES

1. Toofany S. Nursing and information technology. Nurs Mang. 2006;
13(7):18–19.

2. Alquraini H, Alhashem AM, Shah MA, Chowdhury RI. Factors
influencing nurses’ attitudes towards the use of computerized health
information systems in Kuwaiti hospitals. J Adv Nurs. 2007;57(4):
375–381.

3. Davis PB, Duchene P. Putting change theory into practice with IT
adoption, part I. Program Apollo for Healthcare Professionals. 2006.
http://www.hhnmostwired.com/hhnmostwired/jsp/articledisplay.jsp.
Accessed January 12, 2008.

4. Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human: Build-
ing a Safer Health System. Washington, DC: Institute of Medicine;
2000.

5. Rothschild JM, Landrigan CP, Cronin JW, et al. The critical care safety
study: the incidence and nature of adverse events and serious medical
errors in intensive care. Crit Care Med. 2005;33(8):1694–1700.

6. Breslow MJ, Rosenfeld BA, Doerfler M, et al. Effect of a multiple-site
intensive care unit telemedicine program on clinical and economic
outcomes: an alternative paradigm for intensivist staffing. Crit Care
Med. 2004;32(1):31–38.

7. Spiegler P. Time of admission and mortality in the intensive care
unit. Clin Pulm Med. 2004;11(2):118–119.

8. Myers MA, Reed KD. The virtual ICU (vICU): a new dimension for criti-
cal care nursing. Crit Care Nurs Clin North Am. 2008;20:435–439.

9. Rajecki R. Electronic ICU monitoring: Big brother, great friend.
RN Web. 2008. http://rn.modernmedicine.com/rnweb/Nursing+
Top+Story/Electronic-ICU-monitoring-Big-Brother-great-friend/
ArticleStandard/Article/detail/561190. Accessed October 2, 2009.

10. Webber D. From a distance: saving lives through remote care: a
combination of eICU technology, intensivists and in-house staff-
ing delivers significant improvements in ICU patient care. Health
Manage Tech. 2007. http://archive.healthmgttech.com/archives/
0307/0307saving_lives.htm. Accessed January 12, 2008.

11. Berenson RA, Grossman JM, November EA. Does telemonitoring
of patients-the eICU-Improve intensive care? Health Aff. 2009.
http://content.healthaffairs.org/cgi/content/full/28/5/w937?maxto
show=&HITS=10&hits=10&RESULTFORMAT=&fulltext=eicu
&andorexactfulltext=and&searchid=1&FIRSTINDEX=0&
resourcetype=HWCIT. Accessed September 30, 2009.

12. Stronge JH, Brodt A. Assessment of nurses’ attitudes toward com-
puterization. CIN. 1985;3(4):154–158.

13. Smith K, Smith V, Krugman M, Oman K. Evaluating the impact of
computerized clinical documentation. CIN. 2005;23(3):132–138.

14. Stafford TB, Myers MA, Young A, Foster JG, Huber JT. Work-
ing in an eICU Unit: life in the box. Crit Care Nurs Clin North
Am. 2008;20:441–450.

15. Asaro PV. Measuring the effect of a computerized nursing docu-
mentation system with objective measures and reported percep-
tions. Ann Emerg Med. 2004;44:131–132.

16. Shoham S, Gonen A. Intentions of hospital nurses to work with
computers. CIN. 2008;26(2):106–116.

17. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer
technology: a comparison of two theoretical models. Manage Sci.
1989;35(8):982–1003.

18. Mathieson K. Predicting user intentions: comparing the Technology
Acceptance Model with the theory of planned behaviour. Inform Sys
Res 1991;2(3):173–191.

19. Igbaria M, Zinatelli N, Cragg P, Cavaye ALM. Personal comput-
ing acceptance factors in small firms: a structural equation model.
MIS Q. 1997;21(3):279–305.

20. Hu PJ, Chau PYK, Lie Sheng OR, Tam KY. Examining the Tech-
nology Acceptance Model using physician acceptance of telemedi-
cine technology. J Manage Inform Syst. 1999;16(2):91–112.

21. Venkatesh V. Determinants of perceived ease of use: integration
control, intrinsic motivation, and emotion into the Technology
Acceptance Model. Inform Syst Res. 2000;11(4):342–365.

22. Handy J, Hunter I, Whiddett R. User acceptance of inter-
organizational electronic medical records. Health Info J. 2001;7:
103–107.

23. Lu J, Yu CS, Liu C, Yao JE. Technology Acceptance Model for
wireless Internet. Internet Res. 2003;13(3):206–222.

24. Ammenwerth E, Mansmann U, Iller C, Eichstadter R. Factors af-
fecting and affected by user acceptance of computer-based nurs-
ing documentation: results of a two-year study. JAMIA. 2003;10(1):
69–84.

25. Chang BL, Bakken S, Brown SS, et al. Bridging the digital divide:
reaching vulnerable populations. JAMIA. 2004;11(6):448–457.

26. An JY. Correlates and Predictors of Consumers’ Health Informa-
tion and Services Usage Behavior on the Internet: A Structural
Equation Modeling Approach [dissertation]. New York: New
York University; 2005.

27. Cheung W, Huang W. Proposing a framework to assess Internet
usage in university education: an empirical investigation from a
student’s perspective. Br J Educ Technol. 2005;36(2):237–253.

28. Drennan J, Kennedy J, Pisarki A. Factors affecting student atti-
tudes toward flexible online learning in management education.
J Educ Res. 2005. http://findarticles.com/p/articles/mi_hb3507/is_
200507/ai_n15147680/. Accessed October 8, 2005.

29. Tung FC, Chang SC, Chou CM. An extension of trust and TAM
model with IDT in the adoption of the electronic logistics infor-
mation system in HIS in the medical industry. Int J Med Info. 2007;
77(5):324–335.

30. Chen J, Yang KF, Tang FI, Huang CH, Yu S. Applying the
Technology Acceptance Model to explore public health nurses’ in-
tentions towards Web-based learning: a cross-sectional question-
naire survey. Int J Nurs Stud. 2006;45:869–878.

31. Jayasuriya R, Caputi P. Computer attitude and computer anxiety

CIN: Computers, Informatics, Nursing & July 2011 417

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

in nursing: validation of an instrument using an Australian sam-
ple. CIN. 1996;14(6):340–345.

32. Moody L, Slocumb E, Berg B, Jackson D. Electronic health re-
cords documentation in nursing: nurses’ perceptions, attitudes,
and preferences. CIN. 2004;22(6):337–344.

33. Brumini G, Kovic I, Zombori D, Lulic I, Petrovecki M. Nurses’
attitudes toward computers: cross sectional questionnaire study.
Croatia Med J. 2005;46(1):101–104

34. Weiner M, Gress T, Thiemann DR, et al. Contrasting views of
physicians and nurses about an inpatient computer-based pro-
vider order-entry-system. JAMIA. 1999;6(3):234–244.

35. Davis FD. Perceived usefulness, perceived ease of use, and user ac-
ceptance of information technology. MIS Q. 1989;13(3):319–339.

36. Kelloway EK. Using LISREL for Structural Equation Modeling:
A Research’s Guide. Thousand Oaks, CA: SAGE Publications, Inc;
1998.

37. Bollen KA. Structural Equations With Latent Variables. New York:
Wiley Interscience; 1989.

38. Harris M, Schaubroek J. Confirmatory modeling in organiza-
tional behavior, human resource management: issue and applica-
tions. J Manage. 1990;16(2):337–360.

39. Mertler CA, Vannatta RA. Advanced and Multivariate Statistical
Methods. 2nd ed. Los Angeles, CA: Pyrczak Publishing; 2002.

40. Tabachnick AG, Fidell LS. Using Multivariate Statistics. 4th ed.
Needham Heights, MA: Allyn and Bacon; 2001.

41. Hartwick J, Barki H. Explaining the role of user participation in
information systems use. Manage Sci. 1994;40(4):440–465.

42. Hoyle RH. Structural equation modeling: concepts, issues, and appli-
cations. Thousand Oaks, CA: Sage; 1995.

43. Segars AH, Grover V. Re-examining perceived ease of use and
usefulness: a confirmatory factor analysis. MIS Q. 1993;17(4):
517–525.

418 CIN: Computers, Informatics, Nursing & July 2011

Copyright @ 201 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.1

Technological Forecasting & Social Change 78 (2011) 650–660

Contents lists available at ScienceDirect

Technological Forecasting & Social Change

Applying the Technology Acceptance Model to the introduction of
healthcare information systems

Fan-Yun Pai a,⁎, Kai-I Huang b

a Department of Business Administration, National Changhua University of Education, No. 2, Shi-Da Road, Changhua City, 500, Taiwan
b Department of Business Administration, Tunghai University, No. 181 Section 3, Taichung Harbor Road, Taichung, Taiwan

a r t i c l e i n f o

⁎ Corresponding author.
E-mail addresses: [email protected] (F.-Y. Pai)

0040-1625/$ – see front matter © 2010 Elsevier Inc.
doi:10.1016/j.techfore.2010.11.007

a b s t r a c t

Article history:
Received 27 May 2010
Received in revised form 23 October 2010
Accepted 18 November 2010
Available online 16 December 2010

With the rapid development of information systems and advancesinhealthcare technology paired
with current concerns arise over patients’ safety and how to cure them efficiently, the healthcare
information systems are attracting the attention of more and more people. The purpose of this
study is to propose a conceptual model, appropriate for the intention to use healthcare
information systems, by adopting the system, service, and information qualities covered in the
Information System Success Model proposed by DeLone and Mclean [1] as the external variables
and integrating the three dimensions of perceived usefulness, perceived ease of use, and intention
to use — referred to in Venkatesh and Davis’ updated Technology Acceptance Model, TAM [2]. This
study first analyzes relevant researches on the intention to use such systems as the basis for the
questionnaire design, then conducts questionnaire survey among district hospital nurses, head
directors, and other related personnel. After the questionnaires are collected, SEM is used to
analyze the data. The analysis shows that the proposed factors positively influence users’ intention
to use a healthcare system. Information, service and system quality influence user’s intention
through the mediating constructs, perceived usefulness and perceived ease-of-use. Managerial
implications are provided accordingly. Suggestions for introducing healthcare information system
are then provided as well.

© 2010 Elsevier Inc. All rights reserved.

Keywords:
Information system success model
Technology Acceptance Model (TAM)
Healthcare information system

1. Introduction

Currently, with the rapid development of information systems and the advancement of healthcare technologies, nurses are
often required to learn how to operate relevant care assistance equipments while providing clinical care for patients. As the
severity of patients’ illnesses increase, nurses must spend more time taking care of them, therefore, lots of scholars assert that how
to apply current information technology in assisting healthcare to effectively improve the quality of healthcare service and
promote electronic case history has currently become an important subject in healthcare information management [3,4].

In recent years, regarding the factors which may impact the implementation of a healthcare information system, the
questionnaire surveys conducted by Hsiao and Chang [5] among a total of 85 regional hospitals found that such factors include
among others, the support from the senior management level, the skills of the special committee, and the coordination of
organizational resources and user participation. However, Choe [6] claims in research that those factors are among others, user
participation, support from the senior management level, training, background of the special committee, and task type. In addition,
other scholars point out the following factors also have to be considered while implementing the system, including the nurses’
preparation, the coordination among each department and the evaluation on continuous supervising, the acceptance of computers
by nurses, and organization and management support. Therefore, how to use information technology to develop healthcare
systems is an important subject that deserves lots of attention [7–9].

, [email protected] (K.-I. Huang).

All rights reserved.

651F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

In the past, most researches on healthcare information were about the planning and discussion of hospitals as an entire unit, for
example, Tsai et al. [10] studied the factors that influence the information systems in hospitals; or about the brief introduction of
healthcare information systems, for example, Chang et al. [3] introduced the system and how to use it, in the context of a specified
hospital. None of them focused on the study of users’ actual use of the system, while those that come close mainly included
qualitative descriptions with a lack of quantity analysis. At the same time, systems concerning patients’ safety also deserve more
attention, such as the alerting system on patients’ life safety, recording system on vital signs and accident notifying system.
Therefore, by adopting the system, service and information qualities covered in the Information System Success Model proposed
by DeLone and Mclean [1] as the external variables and integrating the three dimensions of perceived usefulness, perceived ease of
use, and intention to use referred in Venkatesh and Davis’ [2] updated Technology Acceptance Model, TAM, this study is expected
to propose a evaluation model appropriate for healthcare information systems, in order to identify the cause and effect
relationships between the relevant factors affecting the intention to use information systems and provide reference for hospitals
equipped or unequipped with the system to evaluate, improve, and plan.

2. Literature review

2.1. Healthcare information system

This system is known as the healthcare planning system or hospital information system. Its development can be dated back to
1960 when its major functions were limited to administrative management only. After 1970, sizable hospitals gradually set up
internal information sectors, and private information companies started to develop high commercial value computer information
systems, which contributed to the prosperous development of the healthcare information system [10]. The creation of this system
is mainly a set of standards based on healthcare diagnosis, symptoms, cause, healthcare target and measurements. Such
computerized programs provide nurses with the necessary contents, healthcare plans, and additional functions including addition,
revision, inquiry and printing [11]. In order to get a more efficient system, Simpson and Weaver [12] believes that by integrating
the healthcare information system with the hospital system, clinical care and administrative management can be combined to
enhance the efficiency of the system.

To appropriately evaluate the efficiency of such systems, many scholars adopt different methods. For instance, Hortman and
Thompson [13] carried out open Q&A in both questionnaires and forms to identify users’ satisfaction and opinion, while Lee et al.
[14] used one-to-one or one-to-many quality interviews to analyze in depth the users’ opinion on a system. Lising and Kennedy
[15] mainly verified the quality of case history to figure out whether a healthcare process has been recorded completely as they
also used the behavioral observation method to get a better idea of the time allocation during the healthcare process. In the recent
5 years, healthcare information system use has mainly been evaluated in the forms of questionnaire surveys, in-depth interviews,
individual case studies, material collections. The questionnaire survey method is most widely used, generally targeted at system
use satisfaction and attitudes relevance with its major components as the nurses’ age, seniority, education, and user satisfaction
[16,17]. These researches show that nurses feel positively on the system in these aspects: it reduces paper work, provides
healthcare instruction, and is equipped with learning functions; in contrast, they feel negatively in terms of insufficient computers
and evaluation contents, disconnecting with other information system, complicated operation procedure, etc. [14]. In recent years,
the application of healthcare information systems and relevant research results are fruitful, which can be separated into four major
categories: the factors which can impact the input of the system, the structure of the system, the components of the system, and
the efficiency of the system.

2.2. Behavior theory and Technology Acceptance Model

In 1975, Ajzen and Fishbein [18] proposed the Theory of Reasoned Action, TRA, which mainly illustrates a person’s behavioral
tendency, for the purpose of predicting, changing and interpreting an individual’s particular behavior. TRA posits that individual
behavior is driven by behavioral intentions where behavioral intentions are a function of an individual’s attitude toward the
behavior and subjective norms surrounding the performance of the behavior. In this theory, attitude and subjective norms are
independent of each other and they could each exert indirect influence on an individual’s behavior through behavioral intention.
Attitude toward the behavior is defined as the individual’s positive or negative feelings about performing a behavior. Subjective
norm reflects social pressures when an individual is performing a behavior and his perception of whether people important to the
individual think the behavior should be performed.

In 1985, Ajzen [19] proposed the Theory of planned behavior, TPB. It is an extension of the Theory of Reasoned Action that
strived for a more appropriate prediction and interpretation of behavioral theory. The difference between TPB and TRA is that the
former predicts behavior under comparatively less controllable circumstances, while the latter predicts behavior based on the
assumption that all behaviors and behavioral motivations are under control. TPB also adds the concept of perceived behavioral
control as a third variable. It refers to an individual’s perceived ease or difficulty of performing a particular behavior [20]. It is
assumed that perceived behavioral control is determined by the total set of accessible control beliefs. In other words, if an
individual feels that he obtains more resources and opportunities while the difficulty of performing a behavior is comparatively
less, his perceived behavioral control would be stronger.

In order to explore the relationship between the perceived emotions factor and the use of science technology, Davis [21]
developed the Technology Acceptance Model, TAM that shows how users come to accept and use a technology and is based on the

652 F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

Theory of Reasoned Action (TRA) and Theory of Planned Behavior (TPB) [22]. TAM assumes that there are two specified beliefs that
determine computer usage: perceived usefulness and perceived ease of use, eliminating subjective norms and normative beliefs.
The model suggests that perceived usefulness and perceived ease of use influence users’ attitudes towards using a new technology.
User shows positive feelings about the new technology if he or she believes it is good for his or her job performance, thus users’
attitudes towards using a new technology will be more positive. Such attitudes will furthermore influence the user’s behavioral
intention and actual system use. In addition, external variables can also have some impact on users’ internal attitudes, beliefs, and
intentions, further influencing the Technology Acceptance Model [23]. Previous studies proved that different external variables
actually influence perceived usefulness and perceived ease of use. Hong et al. [24] asserted that the following five external
variables influence individual perception: the computer’s self-efficacy, the knowledge of the search domain, the relevance, the
terminology and the screen design. However, Lewis et al. [25] found the external variables to include the institutional factor, the
social factor, and the individual factor.

After a long period of research, TAM has been successfully tested across a wide range of computing technologies, organizational
settings, and user populations [26]. Although Hsu and Lu [27] mention that comparative results are mixed, TAM is still one of the
most frequently tested models in IS literature. Many scholars have revised the Technology Acceptance Model to enhance its
interpretation abilities. They not only revised the structure of TAM, but also added external variables and mediators. By studying
the relationships between all the variables, they have created better predicting models [2,28–32]. This study uses the revised TAM
proposed by Venkatesh and Davis [2] which includes Perceived Usefulness, Perceived Ease-of-Use and Intention of Use.

2.3. Information system success model

DeLone and McLean [33] created a multidimensional IS success model, which integrates the model of communication
developed by Shannon and Weaver [34], and the information impact theory found by Mason [35]. The updated model consists of
six interrelated dimensions of IS success: system and information quality, IS use, user satisfaction, individual impact and
organization impact. Studies show that system and information quality can influence user satisfaction. The degree of IS use can
influence the degree of user satisfaction directly, the individual’s performance indirectly, and eventually affect the whole
organization.

However, Pitt et al. [36] argue that DeLone and McLean’s information system success model did not include a measure of IS
service quality. They believe that it is necessary to include IS service quality, and assert that system, information, and service
quality together have an impact on IS use and user satisfaction.

Referring to many scholars’ arguments in the past, and agreeing with Pitt et al. [36] on the service quality perspective, DeLone
and McLean [1] proposed an updated IS success model, by adding the dimension of service quality into the original version.
Information, system and service quality may separately or simultaneously affect the two interrelated dimensions of IS use and user
satisfaction while these two dimensions directly affect net benefits. This is also the first time subsequence use is introduced into
the measuring of the IS success model [37].

3. Research design

3.1. Research model

Based on the purpose of the study, as well as the results of sorting relevant research articles, the factors that may affect the
healthcare information system are illustrated in Fig. 1. The study is mainly based on the external variables covered in the IS success
model proposed by DeLone and McLean [1], including system quality, information quality and service quality, together with

Fig. 1. Research model.

653F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

perceived usefulness, perceived ease of use, and intention to use as research dimensions demonstrated by Venkatesh and Davis [2]
in the updated Technology Acceptance Model (TAM). In addition to extensive literature review, we also conducted in-depth
interviews with experts in healthcare institutes and experts familiar with healthcare information systems or general information
system to understand the underlying factors influencing users’ usage intention of healthcare information system and to modify the
proposed model.

3.2. Research hypothesis

Ahn et al. [38] used the Technology Acceptance Model to explore the online and offline features of Internet shopping malls and
their relationships with the acceptance behaviors of customers. The results show that the external variables which affected the
online features included information, service and system quality. Meanwhile, these variables directly influence perceived
usefulness and perceived ease of use. Davis et al. [39] and Venkatesh and Davis [2] pointed out in relevant researches on
Technology Acceptance Model (TAM) that information quality positively affects perceived usefulness, in other words, if the
information quality of the knowledge management system is good, the output charts would be correct, the output knowledge
would be fruitful and could be reused, thus, users believe the system is capable of providing correct information and knowledge.
Consequently, the study puts forward hypothesis 1 (H1) based on above related researches.

H1. Information quality is positively related to IS user’s perceived usefulness.

In exploring the factors for a successful website, Chou [40] mentioned that service quality includes on-time, professional and
personalized service, and it influences perceived usefulness positively. The same was found in the studies of Gefen and Keil [41], as
well as Zhang and Prybutok [42]. They updated TAM to be in line with the context of online shopping. The results of their studies
show that service quality affects not only customer loyalty, but also the perceived ease of use of the online shopping system. Based
on above relevant articles, hypothesis 2 (H2) is put forward as follows:

H2. Service quality is positively related to IS user’s perceived usefulness.

Ahn et al. [38] used the Technology Acceptance Model to explore online and offline features of Internet shopping malls. The
results show that the external variables which affected the online features include information, service and system quality. These
variables positively influenced perceived usefulness and perceived ease of use. Thus, hypothesis 3 (H3) is represented follows:

H3. Service quality is positively related to IS user’s perceived ease of use.

Chiou and Fang [43] explored internet users’ behavior and found that system quality include design quality, response time, and
accessibility. Design quality refers to the inquiry function of the system and file transfer speed. Online response time means how
soon the response is and how long the response takes. Accessibility refers to whether the software and hardware of the website are
accessible. These have significant impacts on an IS user’s perceived ease of use. Thus, the following hypothesis 4 (H4) is put forth:

H4. System quality is positively related to IS user’s perceived ease of use.

Hung et al. [44] studied previous research articles on TAM, and found among 39 articles, more than 30 researches claim that
user’s perceived ease of use affects perceived usefulness positively, in accordance to what Mathieson [45] found in the research of
the use of word processing software. Lee and Kim [32] also found a positive relationship between perceived ease of use and users’
perceived usefulness. Based on above results, hypothesis 5 (H5) is presented as follows:

H5. Perceived ease of use is positively related to IS user’s perceived usefulness.

Tsai et al. [46] analyzed the association between individual motivation and user acceptance of a knowledge management
system. The results demonstrate that whether the use of a knowledge management IS can improve users’ work performance,
productivity and efficiency will affect users’ frequency in using the system. The results show a positive relationship between
perceived usefulness and users’ intention to use. Chiou and Fang [43] explored users’ behavior in using internet, and concluded
that frequently updating useful information on a website can affect users’ willingness to use the website. The study proved the
positive relation between perceived usefulness and users’ intention to use. Hence, this relationship is hypothesized as follows:

H6. Perceived usefulness is positively related to IS user’s intention to use.

Lee and Chao [47] explored researches on hospital employees’ use of electronic case histories, and found the users’ intention to
use electronic case histories were affected by their feelings about whether they are easier to use than the conventional method.
The study therefore concludes that perceived ease of use has a positive impact on user’s intention to use. Chen et al. [48] studied
electronic public service, and the results show that a simplified electronic public service system attracts user to reuse the system.
Thus, this leads us to establish the following hypothesis:

H7. Perceived ease of use is positively related to IS user’s intention to use.

654 F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

3.3. Sample and data collection

According to the list of healthcare nursing centers published by the Bureau of National Health Insurance (BNHI) in Taiwan in
2008, there are 23 medical centers, 70 regional hospitals, 359 district hospitals. In order to analyze in-depth how hospitals at
different levels use the healthcare information system and considering that the number of district hospitals alone meet the sample
quantity, the study adopted district hospitals as sample targets. Mainly based on the template proposed by Krejcie and Morgan
[50], the sampling survey was conducted in 100 randomly selected district hospitals. Each hospital was given 10 questionnaires on
the intention to use healthcare information systems and those were filled in by related nurses. It was not known beforehand
whether the hospital already has a functional healthcare information system in place. In the end, a total of 420 questionnaires were
returned. Eliminating the incompletely filled-in questionnaires, the remaining valid questionnaires came to a total of 366. The
response rate, based on the number of distributed, is 36.6%, however, it is 87.15% based on the number of questionnaires returned.
Tests for a non-response bias were carried out by comparing early respondents (responses received within the first 2 weeks) and
later respondents (responses received within the third week or later). The analysis indicated the absence of a non-response bias.

3.4. Questionnaire design

We undertook an intensive study of literature of interest to identity existing measures for related constructs. The questionnaire
was pilot tested with fifteen industry experts. And we conducted face-to-face discussions with these experts after they completed
the questionnaire. We modified, added and deleted questions to refine the survey based on their feedback.

Likert Scale is used in this study as it is the most commonly used measure in scale design, with the 3-point and 7-point Likert
scales generally enjoying the largest popularities. However, Berdie [49] addresses this questionnaire design in his study and
defends the 5-point Likert scale for the following three reasons. First, in most cases, a 5-point Likert Scale is the most reliable
measuring method. Once the questions are over five, it is hard for people to distinguish the right point. Secondly, a 3-point Likert
Scale depresses people’s strongest and mildest opinion, while a 5-point Likert Scale can express it ideally. Thirdly, a 7-point Likert
Scale causes confusion for those people with poor distinguishing ability. Hence, the study adopts the 5-point Likert Scale, with the
responses rated as follows: 1 as strongly disagree, 2 as disagree, 3 as somewhat agree, 4 as agree, and 5 as strongly agree.

4. Empirical study and discussion

4.1. Basic information

Among all the respondents, there are only 22 men (6.1%) compared to 344 women (93.9%). A majority of the respondents, a
total of 155 people (44.2%), are between 30 and 39 years old. Most of them are nurses, a total of 194 (55.3%), with head nurses
(23.4%). They are mainly bachelor holders. In terms of seniority, most of them have more than 5 years of experience, a total of 242
(68.9%). Regarding the source of the healthcare information system, most hospitals purchase the system from outside (66.7%).
There are a total of 125 respondents (35.6%) that have been using a healthcare information system for 3 years. A total of 140
respondents (39.9%) once accepted training on the system while another total of 172 respondents (49%) have never received any
training courses. Concerning familiarity with the operations of the system, a majority of the samples, a total of 180 (51.3%), are
good with it.

4.2. Reliability and validity analysis

The reliability of the questionnaire is measured by Cronbach’s coefficient alpha (α). The results from the study show a
Cronbach’s α score of each dimensional scale: information quality at 0.956, service quality at 0.940, system quality at 0.960,
perceived usefulness at 0.963, perceived ease of use at 0.943, intention to use at 0.944, while the construct as a whole is at 0.923.
This indicates that the questionnaire has the sufficient homogeneity (internal consistency) by exceeding the acceptable coefficient
alpha of 0.90. Table 1 shows the reliability analysis of construct.

The related construct and measurement of the questionnaire are based on the theories covered in previous relevant research
articles, meanwhile, the questionnaire is verified by a professor of the Information Management Department in National Chung

Table 1
Reliability and validity analysis.

Construct Reliability KMO

Information Quality 0.956 0.935
Service Quality 0.940 0.917
System Quality 0.960 0.938
Perceived Usefulness 0.963 0.953
Perceived Ease of Use 0.943 0.750
Intention to Use 0.944 0.760
Whole 0.983

655F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

Cheng University, a deputy professor of Medical Information Research Institution in Taipei Medical University, and experts, such as
the supervisor of the nursing department in Kuang Tien General Hospital, in order to guarantee the validity of the questionnaire.

Apart from the above, the study adopts factor analysis to measure the construct validity of the questionnaire, applying KMO
value in the factor analysis. More higher the KMO value is, more correlating factors the variables share, in turn, more appropriate it
is for factor analysis. A KMO value above 0.5 justifies the use of factor analysis, it is not fit for factor analysis otherwise [51].Hence,
as Table 1 shows, the KMO value of each variable is above 0.5, indicating each as appropriate for factor analysis as there are some
correlating factors among the variables. It also suggests that the questionnaire have sufficient construct validity, as all the factor
loadings exceed the acceptable 0.5.

4.3. Characteristics of construct

Table 2 demonstrates an average value of every construct variable. In the information system success model, variables include
information quality (mean=3.53), service quality (mean=3.43), and system quality (mean=3.35).Under the Technology
Acceptance Model (TAM), variables include perceived usefulness (mean=3.55), perceived ease of use (mean=3.46), and
intention to use (mean=3.61). At the study’s conclusion, healthcare information system users have responded positively in all the
information, service, and system qualities, which indicate the models applied in the study are appropriate.

4.4. Correlation analysis

To analyze the relationship between variables, matrices of Pearson product moment correlation coefficients are used to
measure the related index between variables. Samples to be tested are based on the questions of every construct. As Table 3 shows,
factors of healthcare information system are correlated with each other positively, with each Pearson correlation coefficient
ranging from 0.213 to 0.756.

4.5. Measurement model

Regarding the criteria for evaluating model fit, the study is based on the Bagozzi and Yi [52] proposed preliminary fit criteria,
overall model fit, and fit of internal structure of the model.

The evaluation of model fit covered in the study is based upon the following scholars’ suggestions on the ideal criteria, for
example, Byrne [53] proposed a goodness-of-fit model (as measured by the GFI, Goodness-of-Fit Index), claiming that GFI index
must exceed 0.80. According to Gefen et al. [41], it is a basic criterion that both indices of NFI and IFI exceed 0.90 for acceptable
model fitness, while the recommended fit values for CFI should be more than 0.90 and AGFI more than 0.80. In general, if the value
of χ2/df is smaller than 5, it is considered to be a good fit. Conversely, a RMSEA of less than 0.08 suggests a good fit.

This method is adopted to measure a series of model errors, input errors, or identification problems. We can know whether our
loadings are more than 0.50 and in the acceptable range or not from the measurement errors. Table 4 indicates that the loading of
each construct is more than 0.50 and there is no negative number, which indicates that all indices are within the acceptable range.

Based on what Hair et al. [54] suggested, the study examines the various goodness-of-fits of the overall model and information
observations in three types: Absolute Fit Measures, Incremental Fit Measures and Parsimonious Fit Measures.

4.5.1. Absolute fit measures
These measures determine the degree to which the overall model (structural and measurement models) predicts the observed

covariance or correlation matrix. The indices of measures commonly used include: the chi-square statistic, GFI, and RMSEA (Root
Mean Square Error Approximation). As Table 5 illustrates, the absolute fit index of the study’s overall theory models are:
χ2=2895.655, df=695χ2/df=4.166, GFI=0.871, and RMSEA=0.080, indicating that all of them are within acceptable range.

4.5.2. Incremental fit measures
The second class of measures compares the proposed model to some baseline model, most often referred to as the null model.

The null model should be some realistic model that all other models should be expected to exceed. The indices of these measures
are: NFI (Normal Fit Index), CFI (Goodness-of-fit Index) and AGFI (Adjusted Goodness-of-fit Index). Table 5 yields AGFI=0.884,
NFI=0.907,IFI=0.901, and CFI=0.901, indicating that all indices are within applicable range.

Table 2
Mean and variance analysis.

Variables Item Mean Variance

Information Quality 8 3.53 0.74
Service Quality 6 3.43 0.77
System Quality 10 3.35 0.75
Perceived Usefulness 9 3.55 0.76
Perceived Ease of Use 3 3.46 0.75
Intention to Use 3 3.61 0.75

Table 3
Correlation analysis.

Information Quality Service Quality System Quality Perceived Usefulness Perceived Ease of Use Intention to Use

Information Quality 1
Service Quality 0.596 ** 1
System Quality 0.756 ** 0.662 ** 1
Perceived Usefulness 0.244 ** 0.253 ** 0.262 ** 1
Perceived Ease of Use 0.267 ** 0.213 ** 0.306 ** 0.224 ** 1
Intention to Use 0.463 ** 0.378 ** 0.437 ** 0.234 ** 0.300 ** 1

*means pb 0.05; ** means pb 0.01;*** means pb 0.001.

Table 4
Measurement model.

Variables Factor Loadings(λ) CR AVE

Information Quality 0.861 0.854
1. The information covered in the healthcare information system meet my needs. 0.892
2. The healthcare information system can provide correct information. 0.753

Service Quality 0.850 0.855
1. When I am facing difficulty, service people from the information center can help me solve the problems. 0.888
2. Service people from the information center have good service attitudes. 0.771

System Quality 0.850 0.849
1. I can get related information while using the healthcare information system. 0.900
2. The healthcare information system can be linked to or integrated with information from other systems. 0.775

Perceived Usefulness 0.870 0.842
1. The healthcare information system can improve my professional skills. 0.873
2. The healthcare information system can reduce the paper work time. 0.777

Perceived Ease of Use 0.923 0.917
1. I think the healthcare information system is easy to use. 0.953
2. I think the interface of the system is clear. 0.876

Intention to Use 0.923 0.909
1. I am willing to use the healthcare information system. 0.943
2. I am glad to learn new healthcare information systems. 0.868

656 F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

4.5.3. Parsimonious fit measures
These measures are sometimes called adjusted fit measures. They can be used to compare models with differing numbers of

parameters to determine the impact of adding additional parameters to the model. Common parsimonious fit measures are the
parsimonious normal fit index (PNFI) and the parsimonious goodness of fit (PGFI). Table 5 yields PNFI=0.785, PGFI=0.817, and
PGFI=0.640, indicating that all indices are within applicable range.

These measures are mainly used to examine significant differences among the estimated parameters to the model and the
reliability of the potential variables of various indices. These can be judged by whether both individual item reliability and
potential variables composite reliability (CR) are more than 0.70, and average variance extracted (AVE) more than 0.50, the
acceptable range. As the Table 4 Measurement Model demonstrates, the CR of information, service, and system qualities, perceived
usefulness, perceived ease of use and intention to use are 0.861, 0.850, 0.850, 0.870, 0.923, and 0.923 respectively, while the AVE
for each item is 0.854, 0.855, 0.849, 0.842, 0.917, and 0.909 respectively. All of these values exceed the marginally acceptable range,
suggesting a good fit of the internal structure of the model.

4.6. Structural model

Before taking the next step, the study first confirms that every construction has certain reliability and validity, in other words,
the study establishes some hypotheses to analyze how variables of information, service and system qualities, perceived usefulness,
perceived ease of use and intention to use affect the intention to use healthcare information system together. According to Fig. 2,
the hypotheses are tested, and the tested results are sorted into Table 6.

Regarding the associations between information and service qualities and perceived usefulness, the standard coefficient of
information quality and perceived usefulness is 0.407 with a p-value of 0.05 which supports H1. Meanwhile, it is also concluded
that information quality positively affects the users’ perceived ease of use of the healthcare information system. In addition, the
standard coefficient of service quality and perceived usefulness is 0.172 with a p-value of 0.05, thus, H2 is supported, reflecting that
service quality positively affects the users’ perceived usefulness of a healthcare information system.

Concerning the relations between service quality, system quality, and perceived ease of use, the standard coefficient of service
quality and perceived ease of use is 0.196 with a p-value of 0.05, thus, H3 is proved, showing that service quality has a positive
impact on the users’ perceived ease of use of the healthcare information system. The standard coefficient of system quality and

Table 5
Measurement model.

Measures Index Ideal Suggest Value Overall Model N=366

Absolute Fit Measures
χ2 − 2895.655
df − 695
χ2/df b5 4.166
GFI N0.80 0.871
RMSEA N0.08 0.080

Incremental Fit Measures
AGFI N0.80 0.884
NFI N0.90 0.907
IFI N0.90 0.901
CFI N0.90 0.901

Parsimonious Fit Measures
PNFI N0.50 0.785
PCFI N0.50 0.817
PGFI N0.50 0.640

Fig. 2. Model path analysis.

657F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

perceived ease of use is 0.421 with a p-value of 0.05, hence, H4 is supported, displaying that system quality exert a positive
influence on users’ perceived ease of use of the system.

In terms of the relationship between perceived ease of use and perceived usefulness, the standard coefficient of β21 is 0.394,
thus, H5 is supported, translating into that perceived ease of use positively influences the users’ perceived usefulness of the
system.

As for the associations among perceived usefulness, perceived ease of use and intention to use, the standard coefficient of β31 is
0.387 with a p-value of 0.05, consequently, H6 is supported, showing that perceived usefulness positively impacts users’ intention
to use the system. The standard coefficient of β32 is 0.498 with a p-value of 0.05, therefore, H7 is supported, representing that
perceived ease of use positively affects users’ intention to use the system.

4.7. Direct and indirect effects

The effects of variables are grouped into three categories: direct, indirect and overall effect, while the last one refers to the
direct effects plus the indirect ones. According to hypotheses 1 and 6, information quality indirectly affects users’ intention to use,
through the path γ11β31, and the indirect influence is 0.158 (by multiplying the path coefficients — 0.407*0.387), without any
direct influence. As a result, the overall influence is 0.158. The indirect influence exerted by the service industry on users’ intention
to use have 3 paths, γ12β31, γ22β21β31, and γ22β32, with values respectively at 0.067, 0.030, and 0.098 (by multiplying the path
coefficients — 0.172*0.387, 0.196*0.394*0.387, and 0.196*0.498 respectively), without any direct influence, hence, the overall
influence is 0.195. System quality affects users’ intention to use indirectly though two paths γ23β21β31 and γ23β32, with values at
0.06 and 0.210 respectively (by multiplying the path coefficients — 0.421*0.394*0.387 and 0.421*0.498 respectively), without
any direct influences, as a result, the overall influence is 0.274. Perceived usefulness directly affects users’ intention to use, through
path β31, value at 0.387 (path coefficient 0.387), without any indirect effects, therefore, the overall influence is 0.387. Perceived
ease of use affects users’ intention to use both directly and indirectly, through paths β21β31 and β32, with values at 0.156 and 0.498
(by multiplying 0.394*0.387, plus the path coefficient 0.387), hence, the overall influences is 0.654. The study shows that

Table 6
Result of whole model hypotheses test.

Hypothesis Path Coefficient Result

H1: Information quality is positively related to IS user’s perceived usefulness 0.407 ⁎ Supported
H2: Service quality is positively related to IS user’s perceived usefulness 0.172 ⁎ Supported
H3: Service quality is positively related to IS user’s perceived ease-of-use 0.196 ⁎ Supported
H4: System quality is positively related to IS user’s perceived ease-of-use 0.421 ⁎ Supported
H5: Perceived ease-of-use is positively related to IS user’s perceived usefulness 0.394 ⁎ Supported
H6: Perceived usefulness is positively related to IS user’s intention to use 0.387 ⁎ Supported
H7: Perceived ease-of-use is positively related to IS user’s intention to use 0.498 ⁎ Supported

⁎ Means p b0.05.

658 F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

perceived ease of use has the largest impact on users’ intention to use, with an overall influence of 0.654, following by perceived
usefulness (overall influence of 0.387), next to system quality (overall influence of 0.274), as a result, it is clear that perceived ease
of use plays the most important role in healthcare information system.

5. Discussion and conclusion

This study proposes that information quality positively affects perceived usefulness, with a path coefficient of 0.497 and p-
value at 0.001, supporting H1. The result is in line with previous studies. DeLone and McLean [1] updated the Information System
Success Model based on the Technology Acceptance Model, believing that users’ behavioral intention to use will be impacted by
each individual’s perceived usefulness and attitudes towards the system. When the user’s attitude towards the information quality
is more positive, the perceived usefulness of information will be higher. As a result, this study asserts that while introducing
healthcare information systems, we should emphasize the following aspects: making sufficient information available, having good
interface design and ensuring on-time updating of information on the system.

In their research of online shopping, Zhang and Prybutok [42] updated TAM to be in line with the context of online shopping.
The results of their studies show that service quality affects not only customer’s loyalty, but also the perceived ease of use of the
online shopping system. The study once again proves that service quality of the healthcare information system positively
influences users’ perceived usefulness, with a path coefficient of 0.172 and p-value at 0.001, supporting H2. In addition, service
quality also has a positive influence on users’ perceived ease of use, with a path coefficient of 0.196 and p-value at 0.001,
supporting H3. The result is consistent with what Ahn et al. [38] concluded in their study on using the Technology Acceptance
Model to explore online and offline features of Internet shopping malls and their relationships with the acceptance behaviors of
customers. The above analysis suggests that when users feel more satisfied with the service quality of the healthcare information
system, their perceived usefulness and perceived ease of use will be higher. Therefore, medical centers should not only focus on
these influential forces during the system introduction period, but also continuously improve their service qualities. All of these
affect users’ feelings about the information system. By continuously enhancing its service qualities, the system would be able to
reach its potential full performance.

The results of the study demonstrate that system quality positively influences users’ perceived ease of use, with a path
coefficient of 0.421 and p-value at 0.001, supporting H4. This is in accordance with what Chiou and Fang [43] found in exploring
internet users’ behavior. Their study concluded that system quality has a significant relationship with perceived ease of use. Their
system quality included design quality, response time, and accessibility. Design quality refers to the inquiry function of the system
and file transfer speed. Online response time means how soon a response is given and how long the response takes. Accessibility
refers to whether the software and hardware of the website are accessible. According to the above analysis, the more the user
agrees with the system quality, the more he or she perceive its ease of use, consequently, hospitals should pays more attention to
the healthcare information system’s stability, its information provided, its information integration ability and its flexibility, so as to
improve users’ perceived ease of use of the information system.

Perceived usefulness and perceived ease of use significantly affect users’ intention to use, with a path coefficient of 0.498 and p-
value at 0.001, supporting H6 and H7. This is in line with the results of the Tsai et al. [46] analysis on the associations between
individual motivation and user acceptance of a knowledge management system. The results demonstrate that the use of a
management information system can improve users’ work performance, productivity, and efficiency. In turn, it will affect these
users’ frequency in using the system. These results show a positive relationship between perceived usefulness and users’ intention
to use. Lee and Chao [47] performed researches on hospital employees’ use of electronic case histories, and found that users’
intention to use electronic case histories were affected by their feelings about whether they are easier to use than the conventional
method. The study therefore concludes that perceived ease of use has a positive impact on a user’s intention to use. In addition, the
results show that perceived ease of use has a significant positive impact on perceived usefulness, with path coefficient of 0.394 and
p-value at 0.001, supporting H5. This is in line with the results of the Kwon and Wen [22] analysis on the factors affecting social
network service use. In short, users’ intention to use a system would be influenced positively and directly by their perceived
usefulness and perceived ease of use, which in turn displays the absolute importance of perceived ease of use. Therefore, while
introducing a healthcare information system into hospital, it is necessary to prompt users to use the system by making the
operation ways and interface of the system simple and easy to learn.

659F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

This study integrates TAM and the Information Systems Success Model to justify and extend the Technology Acceptance Theory
to healthcare information systems. Although this study makes significant contributions to both academia and practice, there are
several limitations which open up venues for further research. There are several factors not discussed that may influence the
constructs in TAM and the Information Systems Success Model. For example, speed of response may influence service quality. The
effects of the antecedences of these two models, therefore, can be investigated in detail. In addition, only district hospitals were
selected as samples to develop and test the proposed model. Future studies should further develop the proposed model and verify
the proposed model with broader samples such as medical centers, regional hospitals, and clinics.

References

[1] W.H. DeLone, E.R. McLean, The DeLone and McLean model of information systems success: A ten-years update, J. Manage. Inf. Syst. 19 (4) (2003) 9–30.
[2] V. Venkatesh, F.D. Davis, A theoretical extension of the technology acceptance model: four longitudinal field studies, Manage. Sci. 46 (2) (2000) 186–204.
[3] M. Chang, J.S. Lin, Y.M. Chang, T.T. Lee, Applications of nursing information systems: Sharing the experience of implementation in a hospital, J. Nurs. 55 (3)

(2008) 75–80.
[4] W. Raghupathi, J. Tan, Strategic uses of information technology in health care: A state-of-the-art survey, Top Health Manage 20 (1) (1999) 1–15.
[5] J.L. Hsiao, I.C. Chang, An empirical study of establishing nursing care plan systems, J. Inf. Manage. 12 (2) (2005) 27–43.
[6] J.M. Choe, The relationship among performance of accounting information systems: influence factors, and evolution level of information systems, J. Manage.

Inf. Syst. 12 (4) (1996) 215–239.
[7] T.Z. Li, R.H. Wang, S.S. Hsu, L.F. Chen, H.Y. Chang, Information technology and nursing: Using PDAs in clinical nursing care, J. Nurs. 45 (1) (1998) 69–76.
[8] H.L. Lai, L. Chen, Nurse satisfaction with the clinical use of personal digital assistant, TZU CHI Med. J. 15 (2) (2003) 97–103.
[9] T.W. Dillon, D. Lending, T.R. Crews, R. Blankenship, Nursing self-efficacy of an integrated clinical and administrative information system, CIN: Computers,

Inform. Nurs. 21 (4) (2003) 198–205.
[10] Y.J. Tsai, S. Wu, B.C. Chiang, Exploring factors affecting the performance of hospital information systems, J. Inf. Manage. 11 (2) (2004) 191–210.
[11] P.A. Mehmert, C.A. Dickel, R.J. Mckeighen, Computerizing nursing diagnosis, Nurs. Manage. 20 (7) (1989) 24–30.
[12] R.L., C.A. Simpson, Administrative application of information technology for nursing managers, in: V.K. Saba, K.A. McCormick (Eds.), Essentials of nursing

informatics, McGraw Hill, Boston, 2005.
[13] P.A. Hortman, C.B. Thompson, Evaluation of user interface satisfaction of a clinical outcomes database, Comput. Inform. Nurs. 23 (6) (2005) 301–307.
[14] T.T. Lee, C.H. Yeh, L.H. Ho, Application of a computerized nursing care plan system in one hospital: Experiences of ICU nurses in Taiwan, J. Adv. Nurs. 39 (1)

(2002) 61–67.
[15] M. Lising, C. Kennedy, An multi method approach to evaluating critical care information systems, Comput. Inform. Nurs. 23 (1) (2005) 27–37.
[16] T.T. Lee, T.Y. Lee, K.C. Lin, P.C. Chang, Factors affecting the use of nursing information systems in Taiwan, J. Clin. Nurs. 50 (2) (2005) 170–178.
[17] H. Alquraini, A.M. Alhashem, M.A. Shah, R.I. Chowdhury, Factors influencing nurse’s attitudes towards the use of computerized health information systemsin

Kuwaiti hospitals, J. Adv. Nurs. 57 (4) (2007) 375–378.
[18] M. Fishbein, I. Ajzen, Belief, Attitude, Intentions and Behavior: An Introduction to Theory and Research, Addison-Wesley, Boston, MA, 1975.
[19] I. Ajzen, From intention to actions: A theory of planned behavior, in: J. Huhl, J. Bechmann (Eds.), Action-Control: From Cognition to Behavior, 1985, Springer-

Verlag, Heidelberg.
[20] I. Ajzen, Attitude structure & behavior, in: A.R. Pratkanis, S.J. Breckler, A.G. Greenwald (Eds.), Attitude structure and function, Lawrence Erlbaum Associates,

Hillsdale, NJ, 1989.
[21] F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Q. 13 (3) (1989) 319–340.
[22] O. Kwon, Y. Wen, An empirical study of the factors affecting social network service use, Comput. Hum. Behav. 26 (2) (2010) 254–263.
[23] F.D. Davis, R.P. Bagozzi, P.R. Warshaw, User acceptance of computer technology: A comparison of two, Manage. Sci. 35 (8) (1989) 982–1003.
[24] W. Hong, J.Y.L. Thong, W.M. Wong, K.Y. Tam, Determinants of user acceptance of digital libraries: An empirical examination of individual differences and

systems characteristics, J. Manage. Inf. Syst. 18 (3) (2002) 97–124.
[25] W. Lewis, R. Agarwal, V. Sambamurthy, Sources of influence on beliefs about information technology use: An empirical study of knowledge workers, MIS Q.

27 (4) (2003) 657–678.
[26] B. Hasan, M.U. Ahmed, Effects of interface style on user perceptions and behavioral intention to use computer systems, Comput. Hum. Behav. 23 (6) (2007)

3025–3037.
[27] C.L. Hsu, H.P. Lu, Consumer behavior in online game communities: A motivational factor perspective, Comput. Hum. Behav. 23 (3) (2007) 1642–1659.
[28] M. Igbaria, N. Zinatelli, P. Cragg, A.L.M. Cavaye, Personal computing acceptance factors in small firms: A structural equation model, MIS Q. 2 (13) (1997) 279–305.
[29] C. Jackson, S. Chow, R. Leitch, Toward and understanding of the behavioral intention to use an information system, Decis. Sci. 31 (2) (1997) 291–311.
[30] H.C. Lucas, V.K. Spitler, Technology use and performance: A field study of broker workstations, Decis. Sci. 30 (2) (1999) 291–311.
[31] P. Luarn, H.H. Lin, Toward an understanding of the behavioral intention to use mobile banking, Comput. Hum. Behav. 21 (6) (2005) 873–891.
[32] S. Lee, B.G. Kim, Factors affecting the usage of intranet: A confirmatory study, Comput. Hum. Behav. 25 (1) (2009) 191–201.
[33] W.H. DeLone, E.R. McLean, Information systems success: The quest for the dependent variable, Inf. Syst. Res. 3 (1) (1992) 60–95.
[34] C.E. Shannon, W. Weaver, The Mathematical Theory of Communication, University of Illinois Pres. London, IL, 1949.
[35] R.O. Mason, Measuring information output: A communication system approach, Inf. Manage. 1 (5) (1978) 219–234.
[36] L.F. Pitt, R.T. Watson, C.B. Kavan, Service quality: A measure of information systems effectiveness, MIS Q. 19 (2) (1995) 173–188.
[37] T. McGill, V. Hobbs, J. Klobas, User-developed applications and information systems success: A test of DeLone and McLean model, Inf. Resour. Manage. J. 16 (1)

(2003) 24–45.
[38] T. Ahn, S. Ruy, I. Han, The impact of the online and offline features on the user acceptance of Internet shopping malls, Electron. Commer. Res. Appl. 3 (4)

(2004) 405–420.
[39] F.D. Davis, R.P. Bagozzi, P.R. Warshaw, Extrinsic and intrinsic motivation to use computers in the workplace, J. Appl. Soc. Psychol. 22 (14) (1992) 1111–1132.
[40] J.H. Chou, Using DeLone and McLean model to analyze determinants of portal sites success, J. Inf. Manage. Concepts Syst. Appl. 8 (1) (2006) 109–132.
[41] D. Gefen, M. Keil, The impact of developer responsiveness on perceptions of usefulness and ease of use: An extension of the technology acceptance model,

DATA BASE Adv. Inf. Syst. 29 (2) (1998) 35–49.
[42] X. Zhang, V.R. Prybutok, An empirical study of online shopping: A service perspective, Int. J. Serv. Technol. Manage. 5 (1) (2004) 1–13.
[43] Y.W. Chiou, G.D. Fang, A study of web portal user behavior, Web J. Chin. Manage. Rev. 8 (1) (2005) 43–60.
[44] S.Y. Hung, T.P. Liang, C.M. Chang, A meta-analysis of empirical research using TAM, J. Inf. Manage. 12 (4) (2005) 211–234.
[45] K. Mathieson, Predicting user intentions: comparing the technology acceptance model with the theory of planned behavior, Inf. Syst. Res. 2 (3) (1991)

173–191.
[46] C.H. Tsai, W.H. Chih, W.L. Hsu, Using tam to study the personal motivations and social context factors influence the acceptance of knowledge management

systems-based on high technology firms’ samples, Electron. Commer. Stud. 5 (1) (2007) 81–108.
[47] W.I. Lee, P.J. Chao, The research of information Technology Acceptance Model of electronic patient record system in health care industry’s employees-An

example of central and southern Taiwan, Health Manage. 5 (2) (2004) 243–269.
[48] Y.C. Chen, R.N. Shang, T.Y. Ho, S.C. Hesieh, The behavioral intention to use e systems for employees in public utility company: Analysis based on TAM and TTF,

J. E Bus. 10 (1) (2008) 305–327.

660 F.-Y. Pai, K.-I. Huang / Technological Forecasting & Social Change 78 (2011) 650–660

[49] D.R. Berdie, Reassessing the Value of High Response Rates to Mail Surveys, Mark. Res. 1 (3) (1994) 52–64.
[50] R.V. Krejcie, D.W. Morgan, Determining sample size for research activities, Educ. Psychol. Meas. 30 (1) (1970) 607–610.
[51] H.F. Kaiser, An Index of Factorial Simplicity, Psychometrika 39 (1) (1974) 31–36.
[52] R.P. Bagozzi, Y. Yi, On the evaluation for structural equation models, J. Acad. Mark. Sci. 23 (4) (1988) 272–277.
[53] B.M. Byrne, Structural equation Modeling with LISREL, PRELIS, and SMPLIS, Lawrence Erlbaum Associates, Mahwah, NJ, 1998.
[54] J.F. Hairs, R.E. Anderson, R.L. Tatham, W.C. Black, Multivariate Data Analysis, 5th ed., Prentice-Hall, Upper Saddle River, NJ, 1998.

O

H
W

a

A

R

R

1

A

K

F

H

E

O

M

U

T

T

E

O

1

H
o
h
t
f
t
m
o
t
f
u
u

1
d

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

j o u r n a l h o m e p a g e : w w w . i j m i j o u r n a l . c o m

rganizational framework for health information technology

elga E. Rippen ∗, Eric C. Pan, Cynthia Russell, Colene M. Byrne, Elaine K. Swift
estat, 1600 Research Blvd, Rockville, MD 20850, United States

r t i c l e i n f o

rticle history:

eceived 21 June 2011

eceived in revised form

0 October 2011

ccepted 26 January 2012

eywords:

ramework

ealth information technology

valuation

rganizational framework

odel

a b s t r a c t

Purpose: We do not yet know how best to design, implement, and use health information

technology (IT). A comprehensive framework that captures knowledge on the implemen-

tation, use, and optimization of health IT will help guide more effective approaches in the

future.

Methods: The authors conducted a targeted review of existing literature on health IT imple-

mentation and use, including health IT-related theories and models. By crosswalking

elements of current theories and models, the authors identified five major facets of an

organizational framework that provides a structure to organize and capture information on

the implementation and use of health IT.

Results: The authors propose a novel organizational framework for health IT implementation

and use with five major facets: technology, use, environment, outcomes, and temporality.

Each major facet is described in detail along with associated categories and measures.

se

echnology

emporal

nvironment

utcomes

Conclusion: The proposed framework is an essential first step toward ensuring a more con-

sistent and comprehensive understanding of health IT implementation and use and a more

rigorous approach to data collection, measurement development, and theory building.

© 2012 Elsevier Ireland Ltd. All rights reserved.

and to indicate gaps where further knowledge is needed.

. Introduction

ow can we maximize the benefits and minimize the risks
f health information technology (IT)? We do not yet know
ow best to design, implement, and use health IT. Although

here are stellar applications that are implemented success-
ully on all fronts within a given organization [1], it is too often
he case that applications are partially implemented, imple-

ented but never used, or implemented with disappointing
r even adverse health or business impacts. What are the fac-
ors that affect whether or not an application is a success or a

ailure? What measures can we use to assess success and fail-
re? How can we apply our understanding of how health IT is
sed to mitigate the risk of failure and maximize the benefits

∗ Corresponding author. Tel.: +1 240 453 2622.
E-mail address: [email protected] (H.E. Rippen).

386-5056/$ – see front matter © 2012 Elsevier Ireland Ltd. All rights res
oi:10.1016/j.ijmedinf.2012.01.012

of success? To answer these questions, we must go beyond
a piecemeal approach that captures only discrete aspects of
health IT such as the health IT product or the outcome. This
requires a comprehensive organizational framework to struc-
ture the array of information around the implementation and
ongoing use of health IT. This organizational framework can
provide the foundation for a more rigorous approach to data
collection, measurement development, and theory building.

A framework is an effective way to present a clear, par-
simonious, but comprehensive understanding of a complex
topic. It provides a road map to organize current knowledge

Because frameworks can effectively highlight key dimen-
sions, relationships, and research needs, they are often used
to guide data collection, measurement development, and

erved.

i c a l

e2 i n t e r n a t i o n a l j o u r n a l o f m e d

theory building. For example, the Institute of Medicine’s
landmark report, Crossing the Quality Chasm [2] presented six
aims and ten rules for quality improvement as a framework to
guide the redesign of the health care system. The framework
was subsequently used and refined to guide data collection,
measurement development, and theory building across a
range of patient care processes, health care settings, and
patient populations [3–5]. Frameworks are ideally suited to
elucidate a complex field such as health IT. Although the
beginnings of medical informatics can be traced back to at
least the 1950s, its development as a formal discipline took
place more recently with a focus on information and how
to collect, analyze, and disseminate it within the health
care delivery process [6]. By 1990, medical informatics was
defined as “a rapidly developing scientific field that deals with
resources, devices and formalized methods for optimizing the
storage, retrieval and management of biomedical information
for problem solving and decision making” [7].

The development of medical informatics as a scientific
field owes much to attempts to understand the use of IT in
non-health business areas and in consumer markets, espe-
cially its rapid growth and winning and losing applications
and investments that resulted. In the 1980s, major theories
and approaches to IT included classic diffusion of innovation
theory [8], organization assimilation of innovation analysis
[9], socio-technical theory [10], the behavioral intention model
[11], socio-cognitive theory [12] and change management [13].
From the 1990s through the 2000s, these models were applied
to technology in a health care setting, with the focus on
the technology alone. In the mid to late 1990s, it became
increasing clear that the success of health IT implementa-
tion and use involved more than just technology since health
care organizations implementing health IT often encountered
high failure rates and other significant challenges. However,
few probed other factors [14]. Since then other fields such as
change management [13] and usability [15] have contributed
to a richer understanding of health IT [16].

2. Theories related to health IT

Publications on health IT implementation are often based on
case studies that report before-and-after outcomes assess-
ments of health IT as an intervention. Although they can
provide rich detail on particular examples, they are often so
focused on the specific aspects of the cases at hand that they
are difficult to use as building blocks for constructing more
generalizable theory. In addition, because of their focus on
the process and impact of implementation, they offer limited
insight into the underlying factors and conditions that shaped
the outcomes [17].

To begin to build a more robust approach to the study of
health IT, some researchers are assessing the applicability of
major theories and models developed outside of health IT
to better predict outcomes, to identify the important factors
relating to success, and to determine how to mitigate risk.

Table 1 lists the aims and major components of several of
these theories. The last column lists major aspects of health
IT addressed in these theories and will be discussed in more
detail in the following section.

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Individually and collectively, these approaches make valu-
able contributions by calling attention to the role of a range
of key factors influencing the implementation and use of
health IT beyond the features of the technology itself. For
example, some perspectives such as sociotechnical theory and
social-cognitive theory focus on the important impact that
individuals can have on health IT mediated through social sys-
tems such as incentive and value structures, organizational
processes, and organizational cultures. Other perspectives,
such as technology diffusion and change management, seek
to assess health IT use in a broader context of the relation-
ship of individuals, groups, organizational features and other
elements to the technology. These perspectives underscore
the complex, interactive, and often subtle range of influences
that shape health IT use and that must be considered in
evaluating its initial use and ultimate outcomes. Still other
perspectives, such as PRECEDE/PROCEED and multi-method,
underscore temporal dimensions as initial health IT imple-
mentation and use over time is affected by change over time
in the environment or other factors.

While these theoretically driven approaches are broader
and often richer than case studies, they are still highly focused,
which allows them to deeply explore the impact of a limited
number of factors. However, this prevents them from explain-
ing the effects of others. For example, change management
theory can be used to address environmental variables criti-
cal for successful implementation, but it will neither predict
nor explain an implementation that fails because the technol-
ogy does not work (e.g., shuts down unexpectedly or does not
scale). In addition, many of the measures used to substantiate
them have not been validated in the context of health IT as
indicated by a paucity of validation studies in the literature.

3. The organizational framework for health
IT

An organizational framework for health IT would provide
a critical step toward the development of a comprehensive
model of implementation by supplying a structure to organize
and capture information around its use, the relevant mea-
sures and tools, and the relationships between and among
different factors. Based upon our understanding of the health
IT field, a targeted review of the health IT implementation
literature [12–14,17,18,27–39], and the key theory-based com-
ponents highlighted in Table 1, we have identified five major
facets of an organizational framework. These facets are:

1. Technology—elements relevant to the specific health IT;
2. Use—elements relating to the actual use of the technology;
3. Environment—elements relating to the context influencing

the use of the technology;
4. Outcomes—elements capturing the end results of the tech-

nology in use in that environment;
5. Temporality—time and the developmental trajectory of

other elements such as implementation and clinical dis-
ease processes.

The following explains each of the major facets in more
detail, along with associated categories and measures.

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e3

Table 1 – Health IT related theories, their components and overlap to framework facets.

Theory Exploratory aim Major components Corresponding organizational
framework facet

Technology
diffusion
[8,18,19]

How diffusion of an
innovation/technology
spreads across a social
system, including
individuals, groups and
organization

Innovation characteristics
Compatibility
Relative advantage
Complexity
Cost
Communicability
Divisibility
Profitability
Social approval
Trialability
Observability

Technology
Use
Environment
Outcomes

Change
management [13]

Relationship of people and
organizational issues to the
change process;
Four stages around change

Organizations
Individuals
Groups
Management of change process

Use
Environment

Precede/proceed
[12]

Integrated framework for
implementing health IT

Five phases with levels of assessment
– Organizational needs and goals
– IT specifications and match with goals
– Behavior and environmental
– Educational and organizational
– Points for system use
Variables of interest
Evaluation phase
– Implementation
– Process evaluation
– Impact evaluation
– System evaluation
– Outcome evaluation

Use
Environment
Outcomes
Technology
Temporality

Sociotechnical
theory [12]

How individuals interact
with technologies relating
to a task

Technical work processes
Social systems within organization (users, their
practices, their mental constructs and their
interactions, management values)

Technology
Use
Environment

Multi-method
[20]

Computerized provider
order entry (CPOE) systems

Work and communication pattern
Organizational culture
Safety and culture

Environment
Use
Temporality

Social-cognitive
theory [11,12,21]

Learning theory and
behavioral change

Environment
Situation
Self-efficacy
Outcome-expectation
Reciprocal determinism
Reinforcement

Environment
Use
Outcomes

Task-technology
fit model [22,23]

Use of technology and how
well it fits

Task characteristics
Technology characteristics
Performance impacts
Utilization

Technology
Use

Technology
acceptance
model [24]

Individual intention to use
the system

Perceived usefulness
Perceived ease of use
Behavioral intention to use
Actual use of system

Use
Environment

Theory of
planned behavior

Intention to use and
human behavior

Attitudes
Perceived behavioral control

s

Use
Environment

4

T
d
r
o
h

[25,26] Subjective norm
Intention
Behavior

. Technology

he health IT implementation literature often does not

escribe the technology in a detailed way. However, details
elating to a technology can be critical to the success
r failure of an implementation. The following examples
ighlight the importance of capturing information around

health IT implementation in a more detailed and uniform
manner:

• Papers published on Computerized Provider Order Entry

systems invariably display screen shots of what a user sees.
However, they often do not depict or otherwise examine the
extent of the technology’s functionality and the smooth-
ness of its incorporation into workflow. For example, order

i c a l

e4 i n t e r n a t i o n a l j o u r n a l o f m e d

entry without integration into the physician documentation
process interrupts a clinician’s workflow.

• Information relating to performance and system downtime
such as one to five minute click delays are often not men-
tioned in studies, even when they may significantly affect
the use of the system.

• Emergency department use of a health information
exchange application to look for patient information does
not note that for every 1000 patients seen, only 10 patients
have some information available. This will influence the use
of that application.

• An application to access patient information was built for
phone X, but 90% of the physicians have phone Y that does
not support the application, affecting the adoption rate.

Table 2 provides examples of categories associated with the
technology facet. Ideally, each category, such as functionality,
should be composed of subcategories. Currently, there is no
standard set of categories and subcategories for technology.

Table 2 – Examples of categories associated with the technolog

Category Characteristics Current state of

Functionality • Describes the
functionality and design
purpose of the technical
application

• Detailed specifica
systems are often c
but not made avail
outside of developm
• Can use taxonom
concepts

Non-functional
Requirements

• Indicates how well the
system performs

• Reliability
• Availability
• Performance
• Security
• Scheduled downt
• Update schedule

Data and
interoperability

• Captures the attributes
related to the data and their
ability to be shared with
other systems
• Includes concepts of
validation and data
integrity, quality, currency,
semantic interoperability
and health IT-related
standards

• Wide range of con
with varying levels
specificity

User-based
design

• Includes user interface
design but also the
workflow that the health IT
was designed to support

• Wide range of opt
formats

Cost • There are several layers
relating to cost: hardware;
software; operation and
maintenance;
implementation costs

• Some costs are m
difficult to measure
others

Product • Describes the specific
technology product (i.e.,
hardware, software)

• Can be very speci
complex and inclu
hardware and softw
e.g., operating syst
coding language, P

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Ideally, categories should also be associated with measures.
For technology, measures range from percentages (e.g., avail-
ability) to detailed technical specifications/artifacts. Although
measures should have specified formats, currently there is
also no standard set of categories or standard data formats
for measures.

5. Use

Categories that are tied to the actual use of a technology are
captured under the use facet (Table 3). This facet covers not
only the “individual” user but also the “group” user discussed
in many of the models. Also included are the individual mea-
sures relevant to many of the models such as ownership,
usability, motivation, workflow, perception of usefulness, ade-

quate training, and comfort with a technology. As with the
other facets, many of these components are not adequately
assessed in the literature. Many studies focus on one aspect of
the facet, for example, an analysis of the usability of a system,

y facet.

measures Examples

tions for
reated,

able
ent

ies to tag

• Drug-drug alert compares an entered drug in a
CPOE application against an XX product interaction
list. If a positive match is identified, an alert based
on that interaction is triggered and text from the list
is presented to the user with an auditory alert such
as a beep. The user is required to select one of two
choices: (1) “Ignore and continue,” which progresses
the ordering process with the information entered;
and (2) “Edit,” which brings the user back to the drug
entry screen.
• Concepts: alerts; CPOE; alert lists; XX product

ime

• Availability is the percent of projected up time over
total time that has passed, e.g., 99.99%, including
scheduled downtime
• Time delay between clicks is 200 ms

cepts
of

• RxNorm used for medication
• 50% of clinical data is shared with the system
• The data is refreshed every 24 h rather than
“immediately”

ions and • Developed based on user-centered design
principles Detailed workflow mapping to support
use in setting with assessment of match

ore
than

• Cost for initial license and recurring yearly cost
such as operations and maintenance

fic but
des

are,
em,
DA type

• Microsoft 7 operating system; iPad 2; Product Z
ambulatory Electronic Health Record (EHR)

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e5

Table 3 – Examples of categories associated with the use facet.

Category Characteristics Current state of measures Examples

User attitudes • Cover a wide range of concepts
such as user satisfaction,
perceived usefulness and usability,
and user acceptance

• Some measures are standardized
while others are not

• User satisfaction with the
drug-drug alert is low (scale)

Usability and
Workflow

• Covers usability and actual
workflow of the user

• Some usability constructs but
significant variations. Workflow
should be detailed and consistent
to ensure comparison over time

• Physician not using CPOE so the
drug-drug alert is seen by the
pharmacist; doctor is called to
question the prescription several
hours later

Ownership/buy-
in

• Captures the amount of user
involvement and participation in
the health IT implementation
process

• No standard measures • Physicians were asked to help set
the alert threshold for the
drug-drug alerts

Knowledge • Includes concepts around adult
learning, training, capability

• Training and effectiveness of
training are related

ainin
al p

ner, s

• Physicians, while being trained to
use CPOE, were also trained about

b
o

•

•

•

n
i
u
i
t
s
s
w

• Tr
form
trai

ut fail to account for the others. Examples of the importance
f these details include:

A nursing documentation system was implemented with a
goal of ensuring real time data capture of vitals for physi-
cians to access on the computer. However, the nurses’
workflow involved documentation of vitals on a piece of
paper in the patient’s room for the physician. The final doc-
umentation in the chart took place at the end of the shift.
The implementation of the system changed nothing related
to the timing of the final documentation, which was now
entered into the computer at the end of the shift. This meant
that physicians did not have ‘real time’ access to the vitals
on the computer.

Physicians were told they had to use the CPOE system and
were not involved in the selection of the system or the
development of order sets. When the system was imple-
mented, many of the physicians did not use the predefined
order sets, ordering took a significant time, and resistance
dramatically increased when errors were discovered. There
was no ownership or sense of responsibility to solve prob-
lems that arose, and the CPOE system was subsequently
abandoned.

Some of the older physicians were uncomfortable with com-
puters. When the hospital moved to EHRs and a paper-free
record, many physicians considered leaving. These physi-
cians were assigned a physician “buddy” who trained them
and answered questions. Discomfort with the technology
was decreased as a result of this intervention.

It is clear that the categories relevant to use of the tech-
ology are important to understanding the success of an

mplementation. Capturing details around user attitudes,
sability and workflow, ownership, and knowledge provides

nsights critical to understanding how best to implement

hese applications. Some categories have validated mea-
urement tools, e.g., user satisfaction survey. Others have a
tandard approach, e.g., cross-functional flowcharts to assess
orkflow that may have variations.

g modalities such as
resentation, train the
uper-user support

the alerts, the threshold settings,
how to address them (e.g., override
or edit)

Table 3 provides examples of categories associated with the
use facet. As was true for the technology facet, each category,
such as user attitudes, should be composed of subcategories.
Moreover, there are no standard set of categories and sub-
categories for use or related measures and data formats. For
use, measures range from surveys of user attitudes to user
feedback on training experiences/value. Additionally there are
metrics that capture the percent use of a specific functionality
within a technology as in the case of physician acknowledg-
ment of drug alerts.

6. Environment

The environment facet captures categories that influence the
implementation and use of the technology. Some examples
that highlight the importance of capturing information related
to this facet include:

• The financial incentives being offered by the federal gov-
ernment to implement health IT in a meaningful way,
combined with the ability of hospitals to provide support to
clinicians because of changes to STARK [40], have enabled
hospitals to more freely provide EHRs at a time when addi-
tional incentives to physicians are being offered.

• The IT infrastructure and the challenges to integrate the
hospital EHR with the physician’s outpatient EHR are barri-
ers to implementation.

• CEO and physician champions can reinforce the commit-
ment of the hospital to implementation of a CPOE system
and its importance to patient safety. This can enhance
communications around the initiative and help clinicians
prepare for the change.

Many of these categories contain numerous elements or
subcategories. However, as with other facets, there are no
established standard measures used to capture and describe

them. The “measures” range from the presence or absence of
an element to detailed description of federal and local policies.

Table 4 provides examples of categories associated with
the environment facet. These categories describe variables

e6 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 4 – Examples of categories associated with the environment facet.

Category Characteristics Current state of measures Examples

Cultural/organizational • Captures teamwork climate,
values, culture

• Some measures exist, others are
descriptive

• The culture supports innovation
and teamwork

Business drivers • This includes governmental
policies and regulations that
influence the organization and
business factors, e.g., competition

• These are often descriptive but
can be summarized in a
categorical measure such as
favorable, neutral or negative on
the health IT implementation

• STARK enhanced the ability and
desire of hospitals to provide EHRs
to physicians

Leadership • Senior leaders and champions fit
into this category

• Leadership also is descriptive,
but can be captured in a scale to
indicate level of support and
relevant attributes of leadership

• Physician champion is involved,
visible, engaging, and trusted

Setting • Which environment the health IT
is being used

• Includes traditional descriptors
such as inpatient, outpatient

• 400 bed hospital associated with
an academic medical center

Resources • This includes not only the
resources available to support the
implementation of the health IT,
but also the IT infrastructure that
can enable it

• Resources cover a broad range
from financial and human
resources to the state of the IT
infrastructure (e.g., broadband)

• Availability of broadband within
the hospital; presence of dead
zones within the hospital

• Sup
man
impl

Support • Many aspects of implementation
management fit under this
category, including training

that influence how health IT is used. For example, environ-
mental categories such as leadership can influence whether
or not the health IT will be successfully implemented [13].
Environment related measures range from surveys of cli-
mate/organizational cultural to asset inventories. As was true
for the each facet, each category, such as business drivers,
should be composed of subcategories with associated mea-
sures.

7. Outcomes

The outcomes facet provides the categories that are the indi-
cators for failure and success of the health IT implementation.
Three categories are highlighted as examples in Table 5. Com-
pared to the other facets, outcomes has the most well-defined
measures [36,41]. Despite the maturity of outcome measures,
the measures captured tend to be very narrowly defined and
at times are not aligned with the other facets. Examples that
highlight these challenges are described below:

• A disease management system with functionality to sup-
port diabetics is implemented to help patients better
maintain their blood sugar by using an online personal
health record to monitor and report back to the case man-
ager. Most patients did not have access to a computer, and
the intervention had no effect.

• Physicians taking care of patients within an HMO were given
reminders to provide patients with pneumococcal vaccines.
After one year, 98% of the appropriate patients had received
the vaccine. Cost savings from avoidance of hospitalizations
for pneumococcal vaccine were only captured for the first
year. True savings are greater since the benefits last beyond
one year.

Table 5 provides examples of categories associated with the
outcome facet. Ideally, each category, such as clinical, should
be composed of subcategories. Currently, there is no standard

port for training, users, and
agement of the
ementation project

• Help desk to assist with forgotten
passwords

set of categories and subcategories or standardized measures
for the outcome facet. For outcome, measures range from clin-
ical and business measures that are predefined to compliance
rates which vary in calculation. Also captured under the out-
comes facet is the methodology of the study. Although some
outcome measures have specified formats, currently this is
not consistent among all the categories.

8. Temporality

Temporality is a final and critical facet for the organizational
framework [8,18,42,43]. The principal temporal category is
time. Time as an independent variable allows for linking mea-
sures across all facets over prescribed periods. The facet also
includes two other temporal categories, the implementation
cycle and the outcomes lifecycle, summarized in Table 6.

Table 6 provides examples of categories associated with the
temporality facet. Ideally, each category, such as implementa-
tion cycle, should be composed of subcategories. Currently,
there is no standard set of categories and subcategories
for temporality. Ideally, categories should also be associated
with measures. For temporality, measures range from the
unambiguous measurement of time to the less structured
measurement associated with the health IT implementa-
tion cycle. Although measures should have specified formats,
currently there is also no standard set of categories or stan-
dard data formats for measuring planning or implementation
cycles.

The temporality facet provides a critical axis for evalu-
ation. Over time, the characteristics captured in the other
facets will change. Linking these changes to time (and/or other
temporal categories) allows for comparisons and understand-
ing of relationships between and across facets. Information

captured across all facets at the start of a health IT
implementation provides a baseline “snapshot.” Information
collected across the facets in the middle of an implementa-
tion captures another “snapshot.” At another phase of the

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e7

Table 5 – Examples of categories associated with the outcomes facet.

Category Characteristics Current state of measures Examples

Clinical • Covers the clinical outcomes
related to the use of the health IT
• Quality measures

• There are many clinical outcome
measures in the literature, and
they need to be mapped to the
specific intervention

• Reduction in pneumococcal
pneumonia (reminders to
immunize)

Business/financial • Cost savings or expenditures are
part of the business outcomes

• Business measures include
increases in efficiencies and
reductions in cost or hospital days

• Hospital stays for senior patients
dropped due to reduced iatrogenic
pneumococcal pneumonia

Adoption • Includes the number of users and
the depth of their use

• Captured as a percentage of users
to potential users; level of use of a
system can be quantified a variety
of ways

• 95% of physicians using CPOE to
enter their orders

Methodology • Covers the details associated
with their study and analysis

• Study type (e.g., case study,
case–control, prospective,
retrospective)

uanti
ualita
rview

• Case–control study of two similar
practices, with one implementing
an EHR (case) and another

“
l
w
t
“
d
f

9

E
t

• Q
• Q
inte

implementation,” another snapshot can be captured. The col-
ection of these “snapshot” data sets over time across facets
ill enable a more robust evaluation of health IT implemen-

ation and the factors relevant to success or failure. The more
snapshots” that are assessed, more fidelity and insight can be
eveloped on the dynamics between the different facets and
actors.

. Cross-walk of facet with five papers

arlier we evaluated health IT-related models to inform
he development of the major facets. To further assess the

Table 6 – Examples of categories associated with the temporali

Category Characteristics Curre

Time • External anchor
• Construct for understanding
duration
• Independent variable

•Unamb

Implementation
cycle

•Characterizes a step in a lifecycle
of health IT implementation
•Not dependent upon time
•Supports comparison across
different implementations
independent of time

•No sing
phases
•Categor

Outcome
lifecycle

•Characterizes the lifecycle of
when an intervention can be
expected to generate a given
outcome
• Provides the ability to account for
findings when a study has not
provided for sufficient time for
evaluation

•Interven
• Need to
effective
•Comple
delivered
•Outcom
long tim
•Tied to

tative (time, cost)
tive (e.g., focus groups,
s)

continuing paper-based practice
(control)

comprehensiveness of our proposed major facets, we also
conducted a detailed crosswalk with five different
types of papers summarized in Table 7 and listed
below:

1. A comprehensive international literature review of EMR
studies Häyrinen et al. [32];

2. A systematic review study of Health IT impact based on

English-language publications by Chaudhry et al. [44];

3. A model of organizational change across several disciplines
applied to medical informatics systems by Lorenzi et al.
[13];

ty facet.

nt state of measures Examples

iguous •Date (e.g., month/day/year);
duration (e.g., days, months, years)

le standard definition of

ical variable

•Planning, implementation,
evaluation, and optimization

tion specific
account for variance in

ness of intervention
x if intervention is

over time
es can continue across a

e span
other facets

•CPOE drug-allergy alerts:
outcome is avoidance of event;
outcome is “immediate”: the
presence or absence of an
avoidable drug-allergy event can
be realized without a lag
•Reminders for pneumococcal
vaccine: outcome is reduced costs
for pneumococcal pneumonia;
outcome is realized across the
lifetime of the patient
•Smoking cessation reminders and
tools: outcome is occurrence of
acute myocardial infarction;
outcome would be expected in a
time frame of months to years

e8 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 7 – Crosswalk between the facets of the organizational framework and components identified in five different types
of health IT papers.

Source Have Organizational Framework Facets

Technology Use Environment Outcome Temporality

Häyrinen et al.
[32]

EHRs classification:
International
Organization for
Standardization
(ISO) definition
Medical component:
Referral, present
complaint (e.g.
symptoms), past
medical, history, life
style, physical
examination,
diagnoses, tests,
procedures,
treatment,
medications,
discharge
Data elements:
Diagnoses codes
(ICD, ICPC),
procedure (CPT),
medications (ATC),
pathological findings
(SNOMED), nursing
problems (NANDA,
ICPN)
System quality: (e.g.,
ease of use, ease of
learning, usability*)
Information use: (e.g.,
retrievability)
Cost

User: Nurse,
physicians, patients,
pharmacy, others
Usability:
Timesaving, clinical
work patterns,
documentation
habits, user
satisfaction,
attitudes,
acceptance

Settings: Inpatient
tertiary, secondary
care, home health,
outpatient

Information quality:
(e.g., completeness
and accuracy)
System quality: (e.g.,
usability*,
timesaving)
User effects:
information use,
user satisfaction,
individual impact
(e.g., clinical work
patterns, changed
documentation
habits, decision
effectiveness or
altered policies to
allow patients to see
their own records)**

Organizational impact:
(e.g., communication
and collaboration,
impact on patient
care)
Patient effects: Patient
satisfaction,
physician–patient
interactions, length
of patient stay,
effects on patient
care, consumer
reactions
Cost

Chaudhry et al.
[44]

Broad range of Health
IT: e.g., CPOE,
decision
support-stand-alone
systems, electronic
results reporting,
electronic
prescribing,
consumer health
informatics/patient,
decision support,
mobile computing,
telemedicine, data
exchange networks,
knowledge retrieval
systems
Human factors: (e.g.,
user-friendliness)

User: Broad range of
users
Organizational process
change: (e.g.,
workflow redesign)

Settings: Many
settings including:
inpatient tertiary,
secondary care,
home health,
outpatient,
academic, health
system (e.g., VA)
Project management:
(e.g., achieving
project milestones)

Effects on Quality:
Reduction of
medication errors,
adherence to
protocol-based care,
enhanced capacity
to perform
surveillance and
monitoring for
disease conditions
Effects on Efficiency:
Decreased rates of
utilization of
redundant or
inappropriate
therapies, provider
time savings
Effects on Costs:
Related to
reductions in
utilization
Effect on time
utilization: Secondary
preventive care
based on time-series
studies (e.g., reduced
hospitalizations,
reduced pressure
ulcers)

i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e9

– Table 7 (Continued)

Source Have Organizational Framework Facets

Technology Use Environment Outcome Temporality

Disease progression:
(primary preventive
care)

Lorenzi et al. [13] Ease of use of the
technology

User acceptance and
satisfaction
User involvement and
participation during
implementation

Organizational
cultural: Leadership,
politics, strategy,
psychology,
intelligence,
decision making
Organizational
structure: Simple,
bureaucracy,
adhocracy,
networked, federal,
divisional

Ash et al. [45] Interoperability: The
system’s ability to
communicate

Workflow: Impact on
added computer
activities on work
performance

Financial Incentives
Safety improvement to
work processes:
Organizational
Culture
Driven top down
verse provider
promoted

Quality of care and
outcomes

Wright et al. [46] System: Brigham
Integrated
Computing System,
MEDIECH MAGIC,
EpicCare, Physician
Order Entry,
Allscripts EMR,
Siemens INVISION,
McKesson Horizon
Expert Orders
Order sets: General
admission orders,
surgical/anesthesia
orders, critical care
admission, clinical
pathways,
obstetrics, diabetes
orders

Setting: Hospital Size
(Staffed beds)
Discharges
Location: City, State
Type: Academic
medical center,
community, health
system

Patient Days
Discharges
Order set utilization:
Total order sets,
order set usages/yr,
frequency of use of
order sets

∗ Some overlap between technology and use exists; for example, usability is an attribute of a technology but when assessed within the context
of use, e.g., workflow, it is captured under use.

ome.

4

5

r
p
t
t
S
e
a

∗∗ Behavior is captured as part of use; the change, over time is in outc

. An assessment based on medical informatics directors of
gaps in EHR adoption and reasons for the gaps and possible
solutions by Ash et al. [45]; and

. A comparison of technology use across seven sites by
Wright et al. [46].

Specific components described in each paper were catego-
ized under one of the facets in our framework. This crosswalk
rovided insightful findings. It became immediately clear that
here is little consistency in how components are concep-

ualized and in how terminology is used to describe them.
ome of the papers used in the crosswalk, along with oth-
rs reviewed for our study, describe contextual factors that
ffect health IT implementation, but do not use a framework

to organize those factors. Other studies provide organizational
frameworks, although each one is different.

An encouraging finding from this crosswalk is that our
framework encapsulates all of the important health IT study
components identified in the five studies. Further, we found
that some facet components such as workflow, organizational
change, and usability are commonly discussed in the stud-
ies. However, other facet components such as those related to
temporality are not often explicitly discussed, although they
are inherently important to the health IT implementation.

Chaudhry et al. [44] noted that the heterogeneity in reporting
on categories such as functionality (technology facet) made it
difficult to assess whether system capabilities were absent or
simply not reported.

e10 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

Table 8 – Illustrative example of two CPOE implementations is related to outcomes over the course of an implementation.

Temporal

Pre-implementation Implementation Post-implementation

Comparison Tool 1 Tool 2 Tool 1 Tool 2 Tool 1 Tool 2

Technology + +++ + +++ ++ +++
Use ++ − ++ + +++ −

++
+

Environment + −−
Outcome n/a n/a

10. Conclusion

Given the complex nature of health IT implementation, an
organizational framework is an essential first step toward
ensuring more consistent and more comprehensive data col-
lection. Ensuring that the framework can support the data
collection of different models will facilitate the development
of more comprehensive models of effective health IT imple-
mentation. Technology, use, environment, outcomes, and
temporality facets are the core features of this organizational
framework. These facets were developed based on our knowl-
edge of the field, a limited literature review, and a crosswalk
of the data elements tied to current theories/models used in
health IT and the literature.

Although this framework is preliminary, the five facets
and the category examples may be able to provide a high
level checklist of data categories to consider in designing a
study. The framework can also be used to begin exploring the
interrelationship between the different facets across studies.
An illustrative example is highlighted in Table 8 where two
CPOE implementations projects, Tool 1 and Tool 2, are fol-
lowed across implementation stages (temporality facet). In
this example, each facet is reduced to a global measure with
a +++ to −−− scale (+ is positive; − is a negative). Tool 1
has relatively positive technology attributes, with challenges
around the performance of the system. This is contrasted by
Tool 2, which is a “top of the line” tool, with great perfor-
mance. On the attributes around the use facet, Tool 1 was
favorable; it fits into the clinician’s workflow, they have been
involved in its selection and the staff participated in well-
designed training. On the other hand, Tool 2 did not support
the workflow of the clinical staff, they did not participate in the
selection of the application (no ownership) and the training
was mediocre. Environmental characteristics for Tool 1 were
positive—resources and technical support available and lead-
ership was supportive. For Tool 2, insufficient resources were
provided for with minimal support. The leadership assumed
not much was needed given that it was a top of the line
tool. The implementation and post-implementation course
for both of the tools were different. For Tool 1 more resources
and leadership support was rallied during implementation
and tool performance was improved over time. Use continued
to increase as positive outcomes were demonstrated. On the
other hand, Tool 2 was used by some during the implemen-
tation push, with an increased involvement in the leadership

team. However, post-implementation, things reverted to pre-
implementation, as there was no perceived value to sue
it.

− + −−
0 ++ 0

From Table 8, it is clear that although the technology in Tool
2 was more favorable than Tool 1, Tool 1 implementation was
successful and Tool 2 was not. Each facet can play a critical role
in the success or failure in the successful design, implemen-
tation, and use of health IT. Pushing the use of a technology
when it cannot meet the requirements may result in more
harm than good. For example, when laboratory results are not
updated in real time in an inpatient EHR system, the lag time
may result in duplication of orders or worse, delaying a treat-
ment that could result in significant adverse consequences for
a patient. If the data is not reliable or available, similar negative
consequences may result and a lack of confidence in these sys-
tems by the users will ensue. However, in the example above,
the technology was not the core factor in the success or failure
of the use of the health IT.

The technology is irrelevant without the users and the con-
text around its use. Even when the technology is still “a work
in progress”, users will use the system if the benefits outweigh
the costs. Nuances such as ownership, matches in workflow
between what the system was designed for versus what is
needed for clinical care, attitudes of the users and training are
important to success. In the example above, it was the lack of
fit and buy-in at the clinical level that resulted in failure. The
environmental conditions around implementation can also
play a significant part in the success or failure of a technology.
Without leadership support, policies, resources, and business
drivers the use of a technology cannot be implemented or sus-
tained. Often the use and environmental conditions influence
each other either positively or negatively. The outcome of the
implementation and use of a technology can be effected by the
technology, use and environmental. When temporal shifts are
not accounted for, findings may not be possible even when the
health IT was effective, e.g., reduced healthcare expenditures
for complications of diabetics cannot be expected for a num-
ber of years (not medically possible). Hence, the temporality
facet provides the ability to track changes in the other facets
over time and events.

There are limitations in examining and aggregating infor-
mation into global scores for each facet. There may be bias in
how scores are assessed unless there is some validation and
assessment of inter-rater reliability. Moreover, “global” scores
may mask more important categorical effects. For this and
other reasons, a “scoring” approach at the facet level for pre-
dicting outcomes would be inappropriate. However, given our
limited understanding of the field, exploration of tools and

approaches to quantify these facets of health IT may yield
some valuable insights. Given the complexity of the factors
relevant in the success or failure of an implementation and use
of a health IT, it is more likely that categories or subcategories

a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e11

w
T
g
c
I

b
o
m
e
a
t
c
a
t
a

t
t
t
m
I
f
t
t
b
r

1

W
f
p
i

•

•

•

•

•

Summary points
What was already known

• Health IT implementation is complex; “wicked”.
• Several theories are being explored to better under-

stand health IT implementation.
• There is a need to address our limited understanding.

What this study added to our knowledge

• No model provides a comprehensive construct for
health IT implementation.

• Five facets of technology, use, environment, temporal,
and outcomes provide the constructs for an organiza-
tional framework.

• An organizational framework for health IT should
organize data, tools and measures in a consistent man-
ner for the field of health IT, facilitating data collection,
measurement development, and theory building.

i n t e r n a t i o n a l j o u r n a l o f m e d i c

ithin the facets could be found to be predictors of success.
he exploration of the facets and their corresponding cate-
ories and subcategories will provide insight for researchers to
ontinue to build and refine predictive models around health
T.

However, in order for evaluations across studies to
e meaningful, each of the facets should be composed
f categories and subcategories that can be consistently
easured—in standard formats—to facilitate cross-study

valuations. The resulting, more detailed framework with
ssociated standard measures and formats will prove essen-
ial to providing a mechanism to make the organization and
ollection of data and observations uniform across studies
nd technologies. This is critical if we are to pool information
hat can be used to develop and evaluate models and theories
round implementation.

Although evaluation of theories that capture the concep-
ualization, development and marketing of a new innova-
ion/technology [23] have not been included in the scope of
his paper, we agree with Chaudhry et al. that the develop-

ent of uniform standards for the reporting of research health
T implementation must be a high priority [44]. The findings
rom the crosswalk suggest that our framework can serve as
he basis for organizing and reporting research on implemen-
ation of health IT, and reducing heterogeneity in reporting
y ensuring that all important facets and their components
elated to health IT are identified using common terminology.

1. Next steps

e have proposed five facets for an organizational framework
or health IT. It is the first step in the development of a com-
rehensive organizational framework for the field. Next steps

nclude:

Further refinement of the categories and subcategories
obtained from the literature and expert panels. The addition
of categories and subcategories will allow a more granular
understanding of health IT.

Identification of measurement tools and measures associ-
ated with the categories or subcategories. Associating tools
and measures with categories and subcategories are a crit-
ical step in enabling researchers to compare studies, and
test models. These measures will likely include text descrip-
tions, categories, scales, and numbers.
Construction of a health IT taxonomy from a variety of
sources to ensure consistency in use of terms. A taxonomy
will facilitate a common understanding of terms and the
consistent use of categories and measures. Taxonomies are
living concepts that change over time and require a main-
tenance process and infrastructure.

Development of a database that supports the collection of
complex data, which can be expanded as we learn more.
A data-mart to adequately support the complex nature of
the data and maintain data relationships required by the

organizational framework will enable the collection of the
data and its analysis.

The development of a publicly available tool that
researchers can access to obtain and/or add tools and

measures and that they can download and/or upload study
data. An online data-mart available to researchers in health
IT will accelerate our understanding of health IT and lever-
age the multitude of studies currently underway. Moreover,
this approach will reinforce the use of the measures and
the associated tools to capture them.

• The development of tools to explore the data and rela-
tionships. Once the data are available, tools that can help
explore complex data sets will facilitate the evaluation of
models and our understanding of health IT.

Author contributions

All authors contributed toward (1) the conception and design
of the study, or acquisition of data, or analysis and interpre-
tation of data, (2) drafting of the article or revising it critically
for important intellectual content, and (3) the final approval
of the version to be submitted.

Conflict of interest

None of the authors have any conflicts of interest that could
bias this work.

Role of funding source

Westat employees the authors of this manuscript and Westat
was not involved any aspects of the study.

Acknowledgements

We would like to thank Susan Crystal-Mansour and Ejim Mark
for help in editing the document.

i c a l

r

e12 i n t e r n a t i o n a l j o u r n a l o f m e d

e f e r e n c e s

[1] D.W. Bates, The road to implementation of the electronic
health record, JAMIA 19 (4) (2006) 311–312.

[2] Institute of Medicine (IOM), Crossing the Quality Chasm: A
New Health System for the 21st Century, National Academy
Press, Washington, DC, 2001.

[3] Institute of Medicine (IOM), Envisioning the National Health
Care Quality Report, National Academy Press, Washington,
DC, 2001.

[4] Institute of Medicine (IOM), Improving the Quality of Health
Care for Mental and Substance-Use Conditions, National
Academy Press, Washington, DC, 2006.

[5] U.S. Department of Health and Human Services (HHS),
National Strategy for Quality Improvement in Health Care,
2011, available at:
http://www.healthcare.gov/law/resources/reports/nationalq
ualitystrategy032011.pdf, accessed 10/05/2011.

[6] C.W. Hanson, History of Computing in Medicine, February 6,
2002, available at:
http://www.cs.princeton.edu/courses/archive/spr02/cs495/
History%20of%20Computing%20in%20Medicine.pdf, cited
16/6/2011.

[7] R.A. Greenes, E.H. Shortliffe, Medical informatics: an
emerging academic discipline and institutional priority,
AMA 263 (1990) 1114–1120.

[8] E.M. Rogers, Diffusion of Innovations, 3rd edition, Free Press,
New York, 1983.

[9] A.D. Meyer, J.B. Goes, Organizational assimilation of
innovations: a multilevel contextual analysis, Academy of
Management Journal 31 (1988) 123.

[10] W. Pasmore, Social science transformed: the sociotechnical
perspective, Human Relations 48 (1995) 1–21.

[11] A. Banduar, Organisational applications of social cognitive
theory, Australian Journal of Management 13 (December)
(1988) 275–302.

[12] R. Kukafka, S.B. Johnson, A. Linfane, J.P. Allegrante,
Grounding a new information technology framework in
behavioral science: a systematic analysis of the literature on
IT use, Journal of Biomedical Informatics 36 (2003) 218–
227.

[13] N.M. Lorenzi, R.T. Riley, A.J. Blyth, G. Southon, B.J. Dixon,
Antecedents of the people and organizational aspects of
medical informatics: review of the literature, AMIA 4 (1997)
79–93.

[14] R.M. Braude, People and organizational issues in health
informatics, JAMIA 4 (1997) 150–151.

[15] Koppe lR, D.A. Kreda, Healthcare IT usability and suitability
for clinical needs: challenges of design, workflow and
contractual relations, Studies in Health Technology and
Informatics 157 (2010) 7–14.

[16] F.C.G. Southon, C. Sauer, C.N.G. Dampney, Information
technology in complex health services: organizational
impediments to successful technology transfer and
diffusion, JAMIA 4 (1997) 112–124.

[17] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach,
Improving clinical practice using clinical decision support
systems: a systematic review of trials to identify features
critical to success, British Medical Journal 330 (April) (2005)
765.

[18] J. Ash, Organizational factors that influence information
technology diffusion in academic health sciences centers,
JAMIA 4 (1997) 102–111.

[19] K.J. Klein, R.S.A. Ralls, Organizational dynamics of
computerized technology implementation: a review of the
empirical literature, Advances in Global High-Technology
Management 5 (1995) 31–79.

i n f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13

[20] J.I. Westbrook, J. Braithwaite, A.W. Georgiou, A. Ampt, N.
Creskwick, E. Coiera, et al., Multimethod evaluation of
information and communication technologies in health in
the context of wicked problems and sociotechnical theory,
JAMIA 14 (2007) 746–755.

[21] M.D. Slater, Integrating application of media effects,
persuasion, and behavioral change theories to
communication campaigns: a stages-of-change framework,
Health Communications 11 (1999) 335–354.

[22] D. Goodhue, Development and measurement validity of a
task-technology fit instrument for user evaluation of
information systems, Decision Sciences 29 (1998) 105–138.

[23] E. Ammenwerth, C. Iller, C. Mahler, It-adoption and the
interaction of task, technology and individuals: a fit
framework and a case study, BMC Medical Informatics and
Decision Making 6 (2006) 3.

[24] F.D. Davis, Perceived usefulness, perceived ease of use, and
user acceptance of information technology, MIS Quarterly 13
(1989) 319–339.

[25] I. Ajzen, The theory of planned behavior, Organizational
Behavior and Human Decision Processes 50 (1991) 179–211.

[26] P.M. Archambault, F. Légare, A. Lavoie, M.P. Gagnon, J.
Lapointe, S. St-Jacques, J. Poitras, K. Aubin, S. Croteau, M.
Pham-Dinh, Healthcare professionals’ intentions to use
wiki-based reminders to promote best practices in trauma
care: a survey protocol, Implementation Science 5 (2010)
45.

[27] C. Paglian, Design and evaluation in eHealth: challenges and
implications for an interdisciplinary field, Journal of Medical
Internet Research 9 (2007) e15.

[28] J.G. Anderson, Social, ethical and legal barriers to e-health,
International Journal of Medical Informatics 76 (2007)
480–483.

[29] W.J. Winkelman, K.J. Leonard, Overcoming structural
constraints to patient utilization of electronic medical
records: a critical review and proposal for an evaluation
framework, JAMIA 11 (2004) 151–161.

[30] B.E. Dixon, A. Zafar, J.M. Overhage, A framework for
evaluating the costs, effort, and value of nationwide health
information exchange, JAMIA 17 (2010) 295–301.

[31] T. Greenhalgh, D. Swinglehurst, Studying technology use as
social practice: the untapped potential of ethnography, BMC
Medicine 9 (2011) 45.

[32] K. Häyrinen, K. Saranto, P. Nykänen, Definition, structure,
content, use and impacts of electronic health records: a
review of the research literature, International Journal of
Medical Informatics 77 (2008) 291–304.

[33] D.S. Bell, S. Cretin, R.S. Marken, A.B. Landman, A conceptual
framework for evaluating outpatient electronic prescribing
systems based on their functional capabilities, JAMIA 11
(2004) 60–70.

[34] J.I Westbrook, J. Braithwaite, K. Gibson, R. Paoloni, J. Callen,
A. Georgiou, et al., Use of information and communication
technologies to support effective work practice innovation
in the health sector: a multi-site study, BMC Health Services
Research 9 (2009) 201.

[35] G. Pare, Enhancing the rigor of qualitative research:
applications of a case methodology to build theories of IT
implementation, The Qualitative Report December (2002) 7.

[36] C.M. Cusack, C.M. Byrne, J.M. Hook, J. McGowan, E. Poon, A.
Zafar, Health Information Technology Evaluation Toolkit,
AHRQ AHRQ Publication No. 09-0083EF, Rockville, 2009.

[37] R. Amarasingham, L. Plantinga, M. Diener-West, D.J. Gaskin,
N.R. Powe, Clinical information technologies and inpatient
outcomes: a multiple hospital study, Archives of Internal

Medicine 169 (2009) 108–114.

[38] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach,
Improving clinical practice using clinical decision support

a l i n

[46] A. Wright, D. Sittig, J.D. Carpenter, M.A. Krall, J.E. Pang, B.
Middleton, Order sets in computerized physician order entry
systems: an analysis of seven sites, in: AMIA 2010

i n t e r n a t i o n a l j o u r n a l o f m e d i c

systems: a systematic review of trials to identify features
critical to success, British Medical Journal 330 (7494) (2005)
765.

[39] A.X. Garg, N.K.J. Adhikar, H. McDonald, M.P. Rosas-Arellano,
P.J. Devereaux, J. Beyene, J. Sam, R.B. Haynes, Effects of
computerized clinical decision support systems on
practitioner performance and patient outcomes: a
systematic review, JAMA 293 (10) (2005) 1223–1238.

[40] Centers for Medicare and Medicaid Services, Physician
Self-referral Overview, Centers for Medicare and Medicaid
Services, February 23, 2011, available at:
http://www.cms.gov/PhysicianSelfReferral/, cited 18/6/2011.

[41] L.M. Kern, R. Dhopeshwarkar, Y. Barron, A. Wilcox, H.
Pincus, R. Kaushal, Measuring the effects of health
information technology on quality of care: a novel set of
proposed metrics for electronic quality reporting, The Joint
Commission Journal on Quality and Patient Safety 35 (2009)

359–368.

[42] S.P. Robbings, Organizational Behavior: Concepts,
Controversies, Applications, 7th edition, Prentice Hall,
Englewood Cliffs, 1996, p. 230.

f o r m a t i c s 8 2 ( 2 0 1 3 ) e1–e13 e13

[43] A. Georgiou, G.A. Westbrook, J. Braithewaite, Time
matters—a theoretical and empirical examination of the
temporal landscape of a hospital pathology service and the
impact of e-health, Social Science and Medicine 72 (2011)
1603–1610.

[44] B. Chaudhry, J. Wang, S. Wu, M. Maglione, W. Mojica, E. Roth,
S.C. Morton, P.G. Shekelle, Systematic review: impact of
health information technology on quality, efficiency, and
cost of medical care, Annals of Internal Medicine 144 (10)
(2006) 742–752.

[45] J.S. Ash, D.W. Bates, Factors and forces affecting EHR system
adoption: report of a 2004 ACMI discussion, Journal of the
American Medical Informatics Association 12 (2005)
8–12.

Symposium Proceeding, 2010, pp. 892–896.

‘‘Learning’’ From Other Industries
Lessons and Challenges for Health
Care Organizations

Amer Kaissi, PhD

Although it is true that health care has several distinguishing characteristics that set it apart,
analysts both within and outside the industry point to several similarities with other fields and
suggest opportunities for health care to learn from other industries. Applications from other
industries have been described in the literature, but the transfer of learning at health care industry
level has not been examined. This article investigates health care learning from other industries in
the recent decade, focusing on aviation, high-reliability organizations, car manufacturing, tele-
communication, car racing, entertainment, and retail; evidence suggests that most innovative
practices originate with these fields. The diffusion of innovations from other industries appears to
start with a few early adopter organizations (hospitals and health systems) and influential other or-
ganizations (The Joint Commission, Institute of Medicine, Agency for Healthcare Research and Quality,
or Institute for Healthcare Improvement) pushing for the innovations. Once the trend becomes
accepted, consultants and copying behavior seem to contribute to its spread across the industry. An
important question to explore is whether the applications in the early adopter organizations are
different (in terms of their effectiveness) from those in the rest of the industry. Another intriguing issue
is to examine whether other industries learn from health care organizations. Key words: health care,
innovation, knowledge transfer, learning

HEALTH CARE MANAGERS typically de-scribe their organizations as ‘‘unique’’
and ‘‘different’’ and rave about the distinctive
aspects of working in an industry where

actions directly impact patients’ lives.
1

Man-

agement guru Peter Drucker
2

described

health care as the most difficult, chaotic,

and complex industry to manage and sug-

gested that the hospital is ‘‘altogether the

most complex human organization ever

devised.’’
2(p118)

Although it is true that health
care has several distinguishing characteristics

that set it apart, analysts both within and

outside the industry point to several similar-

ities with other fields and suggest opportu-

nities in which health care can learn from
other industries.

In the last 10 years, a number of popular

books have stressed similarities with other

industries and recommended learning oppor-

tunities for health care organizations. In Why

Hospitals Should Fly? Nance
3

stressed prin-

ciples from aviation that health care organiza-

tions must instill as a foundation for safety.

Lately, Gawande
4

in The Checklist Manifesto

investigated the finance, construction, restau-

rant management, and aviation industries and

suggested that checklists can significantly

reduce errors in surgery. Whereas these books

have mainly focused on quality and patient

safety, others have addressed different aspects

of health care. For example, in If Disney Ran

Your Hospital, Lee
5

proposed ways to bring

the Disney culture of customer service to

health care.
Christensen has argued that health care can

learn from other industries in implementing

The Health Care Manager
Volume 31, Number 1, pp. 65–74
Copyright # 2012 Wolters Kluwer Health |
Lippincott Williams & Wilkins

Author Affiliation: Department of Health Care

Administration, Trinity University, San Antonio, Texas.

The author has no conflicts of interest.

Correspondence: Amer Kaissi, PhD, Associate

Professor, Department of Health Care Administration,

Trinity University, One Trinity Pl, # 58, San Antonio,

TX 78212 ([email protected]).

DOI: 10.1097/HCM.0b013e318242d399

65

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

‘‘disruptive innovations’’ that will result in

cheaper, simpler, and more accessible health

care services.
6

At the same time, other

examples have surfaced in newspapers and

news reports. Virginia Mason Medical Center

in Seattle has been praised for its efforts to
implement ‘‘Lean manufacturing.’’

7
Boston

Medical Center has been admired for ‘‘keep-

ing the patient moving,’’ a concept borrowed

from airports and restaurants.
8

Massachusetts

General Hospital has implemented lessons

from the Ritz-Carlton in order to create a

‘‘patient experience,’’
9

and Great Ormand

Street Hospital for Children in Britain has
been celebrated by the Wall Street Journal

for collaborating with the Formula One racing

team Ferrari to improve patient handover

techniques from the operating room (OR) to

the intensive care unit (ICU).
10

However, it is not clear why health care

should adopt such innovations from other

industries. The previously mentioned exam-

ples assume that they lead to improved

performance, but there is little to no evidence

to support that argument. One might even

question how much ‘‘learning’’ is occurring

when health care organizations adopt prac-

tices from other industries without much

evidence to support their effectiveness. This

is especially true given that there are essential

differences that exist between health care

organizations and those in other industries,

such as providing a service that directly affects

life and death, legal requirements to stabilize

patients and provide charity care,
11,12

deal-

ing with nonemployed physicians,
13

and

operating in a highly regulated and complex

environment.
14

PURPOSES AND METHODS

The purpose of this article was to review

the evidence on health care ‘‘learning’’ from

other industries. Abundant examples of ap-
plications from other industries have been

described in the health care literature, but a

comprehensive review and assessment of the

positive and negative aspects of this learn-

ing are still lacking. This article will contribute

to the health care management literature by

providing a thorough summary of the types

of innovations that health care organizations

have adopted from other industries and there-

fore can serve as a guide to better understand

what innovations work and what innovations
do not work and under what circumstances.

In the following sections, we inspect

health care learning from other industries in

the recent decade (2000-2010). Although ex-

amples of transfer of knowledge have been

documented before that, we have observed

that the trend has reached its peak in the last

10 years. We elected to focus on aviation,
high-reliability organizations (HROs), car manu-

facturing, telecommunication, car racing, enter-

tainment, and retail because evidence suggests

that most innovative practices originate from

these fields.

LITERATURE REVIEW

Aviation

The aviation industry has long been admired

by health care organizations, especially be-

cause of the dramatic improvements in safety

that it has undergone.
15

Helmreich
16

was

among the first to suggest that lessons from

aviation can help reduce errors in the OR.
Of all the techniques that health care has

borrowed from aviation, crew resource manage-

ment (CRM) and team training techniques are

arguably the most widespread. Crew resource

management emphasizes developing skills in

briefing, inquiry, assertion, workload distribu-

tion, vigilance, and conflict resolution.
17

Appli-

cations of CRM to health care settings are
widespread in the literature: from 2000 to 2010,

there were at least 35 discussion, review, or

research articles that deal with CRM application

in health care organizations. Substantial evi-

dence is also available on the application of

aviation-based simulation training in various

health care disciplines and settings.
18

Another important concept in aviation
safety is the focus on systems and cultures,

rather than blaming individuals for failures.

In the last decade, health care organizations

started to borrow aviation concepts to change

66 THE HEALTH CARE MANAGER/JANUARY–MARCH 2012

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

their cultures and focus on system factors that

can prevent or trap errors before they reach

patients.
19

Moreover, increased attention to

nonpunitive error reporting systems has

been observed.
20

Other concepts and tech-

niques that have crossed over from aviation
include checklists

21
and standardized tools

and behaviors,
22

among others. At a more

macro level, experts have called for the

formation of a public-private partnership

modeled on the Commercial Aviation Safety

Team to improve safety in health care.
23

Applications of CRM and other team

training techniques started in anesthesia but
then spread to other ‘‘high-risk’’ health care

areas such as the emergency department,
24

the OR, the ICU, obstetrics and perinatal

services, and neonatal resuscitation.
25

Other

aviation-based practices such as improved

communication and standard protocols have

been regularly applied to hospital medication

administration processes.
26

The transfer of knowledge from aviation to

health care has not been immediately em-

braced by everyone. Randell
27

argued that

the comparison to aviation ‘‘is not always

useful, on the basis that (i) the type of work

and technology is very different in the 2

domains, (ii) different issues are involved in

training and procurement, and (iii) attitudes
to error vary between the 2 domains.’’

27

Others have warned against the blind copy-

ing of aviation techniques by noting that

‘‘it is not sufficient to take aviation training

materials and simply delete ‘pilot’ and re-

place with ‘nurse’ or ‘anesthetist.’’’
28

What these views seem to suggest is that a

solid understanding of the inherent differ-
ences and similarities between the 2 indus-

tries is needed. A common argument is that

‘‘patients are not airplanes.’’ While advocating

for applications from aviation, Helmreich
16

has advised that the OR ‘‘. . .is a milieu more
complex than the cockpit, with differing

specialties interacting to treat a patient whose

condition and response may have unknown
characteristics. Aircraft tend to be more pre-

dictable than patients.’’ In Table 1, the simi-

larities and differences between health care

and aviation are summarized.
29

High-reliability organizations

High-reliability organizations are organiza-
tions that function in hazardous, fast-paced,

and highly complex technological systems

while operating with no errors for long pe-

riods.
30

They include nuclear power plants,
air traffic control systems, petrochemical plants,
naval aircraft carriers, as well as commercial
and military aviation (aviation was discussed
separately because of the extensive evidence).
The concept of HROs has been around for a
long time, but its applications in health care
organizations have started only around the
year 2000, with the Institute of Medicine
(IOM) (and the Agency for Healthcare Re-
search and Quality [AHRQ] mainly pushing
the idea).

31,32
General applications from

HROs are similar to those from aviation and
include a focus on safety systems, error re-
porting, and simulation training.

33
Specifically,

nuclear power plants represent a successful
model to emulate. It is suggested that this
industry represents a better analogy for an-
esthesia than aviation because of its high levels
of complexity.

34

Car manufacturing

Up until 2010 when news of grave accidents

and major recalls broke, Toyota was widely

recognized as one of the most successful

companies in the world. Through relentless
dedication to continuous quality improvement,

it was famous for the quality of its cars and its

focus on employee safety and well-being, as

well as its efficiency and high profit margins.
35

At the heart of these remarkable results is

the Toyota Production System (TPS), which

emphasizes frequent rapid problem solving

and work redesign, with the goal of ‘‘deliv-
ering to customers exactly what they need,

when they need it, every time, defect-free,

in a safe environment at the lowest cost

without waste.’’
36

In the last 10 years, this philosophy started

to gain a following among health care or-

ganizations with an increasing number of

hospitals and health systems adopting a ver-
sion of TPS as their systematic approach to

enhancing quality and improving efficiency.
37

Lessons and Challenges for Health Care Organizations 67

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

Once widespread across several industries,

TPS became known as Lean Systems, Lean Man-

ufacturing, or just Lean. Several early adopters

of ‘‘Lean health care’’ emerged.
37-40

As these

success stories became publicized in the health

care literature, numerous organizations started

to learn and implement Lean principles. A

recent review of the literature between 1999

and 2009 (which almost coincides with the

time frame of this assessment) illustrated

Table 1. Summary of Similarities and Differences between Health Care and Aviation

Similarities

Complexity Complex procedures and processes with a series of critical steps that must occur to
ensure the safe outcome for the patient/passengers.

Time-critical events Time-critical event flows and actions.
Unpredictability Element of unpredictability (for aviation, unexpected weather events, and other

external operating conditions; for medical, patient response to treatment).
Rare deviations Most days have normal procedure and process flows, but a variety of deviations

may occur requiring urgent response; some of these deviations are extremely rare.
Lengthy training Highly trained professionals involving many years of training a team of professionals,

with a gradient of authority present in the team. Often a single person is designated
as the final authority for the safe outcome of the flight/procedure/process.
As an industry, the practices result in highly visible public safety implications.

Differences

Personal risk The pilot’s fate is tied to the fate of passengers; in the doctor-patient relationship,
only the patient’s safety is at risk.

Public perception Passengers are not often aware of the errors that pilots make. Medical errors are
more frequently visible to the patient or their family.

Litigation Doctors are more often the principal target of litigation when errors occur in
medicine. This impacts how voluntary reporting systems are used: NASA’s
Aviation Safety Reporting System effectively results in immunity for the reporter
in most cases and is very widely used by pilots. Doctors are often reluctant to report
errors in their systems because of the potential for litigation.

Level of training
and roles

Most surgical procedures are performed by 1 surgeon (with resident, etc, assisting)
resulting in a single high-authority figure. Air carriers are 2-crew, the captain
and first officer trade the ‘‘pilot flying’’ role on alternate flights. The other pilot
role is ‘‘pilot monitoring,’’ and their job is to catch and report errors. This yields
a high probability of catching errors.

Authority structure
within team

Flight crew authority gradient is improving over time with crew resource management
adoption. Medical teams are generally autocratic, with even more extreme authority
gradient in some developing countries, so there is little opportunity for error
catching because of crosschecking.

Culture of
standardization

Pilot culture generally accepts standard operating procedures (SOPs), with disciplined
use of procedures and checklists; medical culture values extreme level of knowledge,
judgment, and expertise but is resistant to imposed SOPs, rigor, or discipline.

Oversight Pilots’ performance is subject to random checks by Line Check Airmen, regulatory
observation, recurrent training and checking, and flight operations quality assurance
performance data gathering. Doctors are subject to less formal oversight and
less frequent mandatory ongoing training.

Labor unions Pilots are represented by labor unions in many parts of the world, so feedback or
discipline due to errors is often done with the union as intermediary. Health care
workers are sometimes members of unions, but these unions do not usually act
as intermediaries in incidents involving patient harm.

Litigation If a pilot violates the federal aviation regulations (FARs), they are subject to the
provisions of the FARs, which are federal laws. Litigation is a potential consequence
of medical error, but very few errors committed by health care workers are a
direct violation of the law.

Outside authority Pilot procedures have an original authoritative source with the aircraft manufacturer
(OEM, or original equipment manufacturer). The medical community is a group of
peers, generally without a single authoritative source of procedures and standards.

68 THE HEALTH CARE MANAGER/JANUARY–MARCH 2012

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

around 40 articles related to Lean in health

care.
41

Of these, 8 were empirical studies that

examined the implementation of a Lean proj-

ect to various health care settings. The review

concluded that there is weak evidence be-

hind the assertion that Lean techniques lead
to performance improvement in health care.

This conclusion is in line with the views

of many experts who are skeptical of the

transfer of car manufacturing principles into

health care. It is suggested that ‘‘Lean thinking

has been applied, largely uncritically, to the

hospital sector.’’
42

Some observe that there

is limited literature on the failure of Lean
techniques in health care, which may suggest

a publication bias,
4

whereas others warn that

these principles have been applied in health

care with no theoretical foundation.
43

Other observers have taken a more flexible

approach and propose that Lean techniques

can be successfully implemented in health

care but that ‘‘cultural and practical barriers
likely will have to be overcome.’’

37
Just like

with aviation, the argument is that ‘‘people

are not automobiles.’’ At a deeper level, a

serious organizational barrier is that health

care organizations have fragmented units

that operate as autonomous silos, although

Lean tools emphasize that the entire work-

flow with cooperation of multiple operating
units must be improved. Other challenges

include the fierce adherence to physician

autonomy in health care, which contrasts

with the standardization of practice advocated

by Lean.
44

These barriers and challenges

notwithstanding, and despite the recent failings

at Toyota, recent evidence suggests that the

health care industry is moving toward more
widespread application of Lean principles.

Telecommunication

In the mid-1980s, Motorola, the multina-

tional telecommunication company, devel-

oped Six Sigma, a quality improvement concept

that focuses on error reduction by establishing
aggressive goals.

45
Although Six Sigma was first

applied to manufacturing processes, Motorola,

GE, and other companies have extended the

applications to customer service. Therefore,

several experts started calling for the appli-

cation of Six Sigma in health care. In 1998,

Mark Chassin,
46

a nationally recognized expert

in health care quality improvement declared:

‘‘We can learn a good deal from industries that

are working toward the Six Sigma goal. Let’s try
it in health care and see how close we can get.’’

Soon after the Chassin
46

article, applica-

tions of Six Sigma in health care started to

proliferate. Between 1999 and 2009, around

124 studies relating to Six Sigma appeared in

the health care literature. Of these, 26 were

empirical studies using statistical methods

to evaluate the effectiveness of Six Sigma
projects in health care settings.

41
Similarly

to Lean applications, the review demon-

strated that there are significant gaps in the

literature and very weak evidence that Six

Sigma actually improves quality of care.

The implementation of Six Sigma in health

care has also been fraught with skepticism

and concern. One reason is the fear of health
care executives to completely overhaul their

existing quality improvement initiatives. Other

barriers include the nursing shortage, govern-

mental regulations, long-standing professional

group silos of nonemployed physicians in

hospitals, and the risk of Six Sigma being

used in only marginal projects.
47

Other industries

Although not as extensive as the previously

mentioned applications, examples of transfer

of learning from other industries have also

been documented in the health care literature.

As previously mentioned, to address com-

munication failures in patient handovers
from surgery to the ICU, surgeons at Great

Ormond Street Hospital for Children in

London got inspiration from an unlikely

source: the pit-stop techniques of the Italian

Formula One racing team Ferrari. Using ex-

pertise from car racing, the new handover

protocol focused on leadership, task allocation,

rhythm, standardized processes, checklists,
awareness, anticipation, and communication.

As a result, technical errors, information hand-

over omissions, and duration of handovers

were reduced.
48

Lessons and Challenges for Health Care Organizations 69

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

In a recent article, experts called on health

care organizations to learn lessons from mass

retail to streamline layers in the supply chain

and use purchasing volume to reduce prices.

They used examples of in-store health clinics

and low-cost generic drugs as examples to be
followed to cut costs in health care.

49

Other more straightforward examples

have included the application of hotel-style

room service in hospitals. Practices such as

meal delivery within 30 to 45 minutes, a

restaurant-style menu, tray assembly on de-

mand, scripting, and waitstaff uniforms have

been successfully implemented to allow pa-
tients more control over their food choices.

50

Recently, many hospitals and health systems

have joined the trend of creating a ‘‘customer

experience,’’ just like Starbucks or Disney has

done. Especially after the publication of the

book If Disney Ran Your Hospital, some

pioneering health care organizations have

shifted from a narrow focus on customer
service to engaging ‘‘patients on an emotional,

physical, intellectual, and, yes, spiritual level.’’
5

Hospitals are introducing hotel- and spa-like

amenities such as waterfalls, fireplaces, gar-

dens, aquariums, larger windows, more natural

light, private rooms, better waiting areas, re-

duced environmental stressors, and calming

music. Models such as ‘‘Planetree’’ that stress
‘‘healing, nurturing environments’’ have been

adopted by numerous hospitals.
51

Table 2 pro-

vides a summary of the practices and their in-

dustry, as well as the evidence and the methods

of transfer, as discussed in the following section.

DISCUSSION

Several applications from other industries

to health care organizations have been de-

scribed. These applications can be thought

of as ‘‘innovations’’
52

diffusing across indus-

tries at first (car manufacturing to health

care, for example) and then within the same

industry (across hospitals, for example). The

first issue to consider is how the innovations
are transferred from one industry to another.

The case of health care organizations imple-

menting the TPS can provide a good under-

standing of that development. Executives at T
a
b
le

2
.

S
u

m
m

a
ry

o
f

T
ra

n
sf

e
r

o
f

P
ra

c
ti

c
e
s

In
d
u
s
tr
y

P
ra
c
ti
c
e
s

M
e
th
o
d
s
o
f
T
ra
n
s
fe
r

E
v
id
e
n
c
e

A
v
ia

ti
o

n
C

re
w

re
so

u
rc

e
m

an
ag

e
m

e
n

t/
te

am
tr

ai
n

in
g
/s

im
u

la
ti

o
n

sy
st

e
m

/c
u

lt
u

re
/e

rr
o
r

re
p

o
rt

in
g

In
st

it
u

te
o
f

M
e
d
ic

in
e

(I
O

M
)

re
p

o
rt

,
Jo

in
t

C
o
m

m
is

si
o
n

/A
g
e
n

c
y

fo
r

H
e
al

th
c
ar

e
R

e
se

ar
c
h

an
d

Q
u

al
it

y
(A

H
R

Q
)

c
o
n

su
lt

an
ts

/t
ra

in
in

g
M

ix
e
d

H
ig

h
-r

e
li
a
b

il
it

y
o

rg
a
n

iz
a
ti

o
n

s
S
im

u
la

ti
o

n
sy

st
e
m

/c
u

lt
u

re
/e

rr
o

r
re

p
o

rt
in

g
IO

M
re

p
o

rt
,

Jo
in

t
C

o
m

m
is

si
o

n
/A

H
R

Q
c
o

n
su

lt
a
n

ts
/t

ra
in

in
g

M
ix

e
d

C
a
r

m
a
n

u
fa

c
tu

ri
n

g
(T

o
y
o

ta
)

T
o

y
o

ta
P

ro
d

u
c
ti

o
n

S
y
st

e
m

/L
e
a
n

C
o

ll
a
b

o
ra

ti
o

n
w

it
h

o
th

e
r

e
x

e
c
u

ti
v
e
s

In
st

it
u

te
fo

r
H

e
a
lt

h
c
a
re

Im
p

ro
v
e
m

e
n

t
c
o

n
su

lt
a
n

ts
W

e
a
k

T
e
le

c
o
m

m
u

n
ic

at
io

n
(M

o
to

ro
la

)
S
ix

S
ig

m
a

C
o

ll
a
b

o
ra

ti
o

n
w

it
h

o
th

e
r

e
x

e
c
u

ti
v
e
s,

c
o

n
su

lt
a
n

ts
W

e
a
k

O
th

e
r

in
d

u
st

ri
e
s

C
a
r

ra
c
in

g
P

a
ti

e
n

t
h

a
n

d
o

v
e
rs

/s
a
fe

ty
C

o
ll
a
b

o
ra

ti
o

n
S
tr

o
n

g
H

o
te

ls
M

e
a
l

d
e
li
v
e
ry

—
E

n
te

rt
a
in

m
e
n

t
C

u
st

o
m

e
r

se
rv

ic
e

H
ir

in
g

fo
rm

e
r

e
x

e
c
u

ti
v
e
s

—

70 THE HEALTH CARE MANAGER/JANUARY–MARCH 2012

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

Virginia Mason Medical Center first became

interested in TPS when they heard about the

benefits of the approach from local business

executives in the Seattle region. Once they

became convinced of the value of TPS, the

hospital chairman, president, and other key
leaders went on a 2-week visit to Toyota

factories.
37,39

Around the same time in the

Pittsburgh area, hospitals, major insurers, and

corporate and civic leaders joined efforts to

form the Pittsburgh Regional Health Initiative,

a nonprofit community consortium focused on

‘‘perfecting the health care system.’’ As the

former Alcoa (a global producer of Aluminum)
chairman was recruited to help spearhead the

effort, health care executives came in contact

with Alcoa’s own adoption of TPS.
34

As a

result, several health systems started learning

from Alcoa’s experience and implemented TPS

in their own organizations.
36,38

A similar situation

took place at the University of Michigan Health

System, where contact was established with
General Motors (GM), an expert in both Lean

and Six Sigma approaches. This collaboration

resulted in GM providing University of Michigan

Health System with facilitators for initial Lean

projects, helping with training coaches, and

giving access to GM’s own training materials.
40

In brief, the pattern of initial diffusion of TPS/

Lean innovations seems to be collaboration
between health care executives and other local

business executives, which results in transfer

of knowledge across industries.

A different method of innovation diffusion

appears to take place in the case of patient
safety practices from aviation and HROs. The

main precursor for that trend was the pub-

lication of the IOM
31

report that estimated
that up to 98 000 people die each year in

the United States because of medical errors

and that noted that ‘‘health care is decades

behind other industries in terms of creating
safer systems.’’ The report called on health

care organizations to derive lessons from

aviation and HROs. Soon after, powerful or-

ganizations in the health care arena, such as

The Joint Commission and the AHRQ, started

pushing for practices such as CRM, team train-
ing, simulation, and other safety practices.

25,53

For example, AHRQ established the HRO net-

work to provide health care organizations

with a forum for learning about promising

practices and identifying new and innovative

ways to implement research findings. Early

adopters such as the Veterans Health Admin-

istration, Kaiser Permanente, and Vanderbilt
University Medical Center, among others, started

sending their clinical teams to attend training

courses, as well developing their own courses

‘‘in-house.’’
54-57

Another method of diffusion of innovation

that is also worth mentioning is health care

organizations’ hiring of executives who have

worked in other industries. For example,
Henry Ford West Bloomfield Hospital in

Michigan recently hired a former Ritz-Carlton

executive as its chief executive officer. The

newly built hospital boasts private rooms,

a chef, a concierge, and weekly classical

concerts, all concepts imported from the

hotel industry.
58

The second issue to examine is, once the
innovation is brought to the industry by early

adopter organizations, how does it spread to

other organizations in health care? In the

case of TPS/Lean, once the initial phase of

adoption was underway by a few organiza-

tions, an influential organization jumped ship

and endorsed the trend. The Institute for

Healthcare Improvement, an independent
not-for-profit organization, published in 2005

a powerful white paper that proposed that

‘‘adoption of lean management strategies—

while not a simple task—can help health care

organizations improve processes and out-

comes, reduce cost, and increase satisfaction

among patients, providers, and staff.’’
59

As a

result, the TPS/Lean trend started to gain
‘‘legitimacy’’ in health care. Lean (and Six

Sigma) consultants started offering their ser-

vices. For example, the previously mentioned

Pittsburgh Regional Health Initiative now of-

fers education and training courses in TPS and

Lean.
60

Many health systems implemented

these new approaches by hiring consultants

and/or by copying successful early adopters
in their markets.

Similarly, for innovations from aviation and
HROs, health care organizations, hoping to

catch up with the new trend, started purchasing

Lessons and Challenges for Health Care Organizations 71

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

(at considerable expenses) prepackaged CRM
and other team training approaches. Con-
sultants began to sell health care–tailored team

training, and several standardized team train-
ing curricula emerged: MedTeams (adopted
from US army rotorcraft safety experience),

TeamStepps (developed by the Department of
Defense Safety Program in collaboration with
AHRQ), and Medical Team Training, devel-

oped by the Veterans Health Administration.
30

Helmreich and Sexton
61

likened the surfacing

of team training consultants in health care
to what took place in aviation several years
ago: ‘‘After CRM had gone through the pro-

cess of being recognized, acknowledged, and
formally mandated, a number of consultants
emerged from ‘under rocks’ offering packaged

programs that promised to cure all an orga-
nization’s ill.’’ They cautioned that some of
these packages may be ill suited for hospital

staff and other medical teams.

Implications

In summary, the diffusion of innovations
from other industries appears to start with a
few early adopter organizations (hospitals
and health systems) and influential other
organizations (The Joint Commission, IOM,
AHRQ, or Institute for Healthcare Improve-
ment) pushing for the innovations. Once the
trend becomes accepted, consultants and
copying behavior contribute to its spread
across the industry. An important question to
explore is whether the applications in the
early adopter organizations are different (in
terms of their effectiveness) from those in
the rest of the industry. Institutional Theory
suggests that innovative practices that im-
prove performance in early-adopting organi-
zations are legitimized in the environment.

Ultimately, these innovations reach a level of
acceptance where failure to adopt them is
seen as ‘‘irrational and negligent.’’ At this point,
other organizations will adopt the new prac-
tices even if they do not improve perfor-
mance.

62
For example, in the early 1990s,

hospitals that have adopted Total Quality

Management principles early on were driven

by efficiency concerns, whereas those that

adopted it later on were mainly driven by le-
gitimacy concerns.

63
It will be important to

assess whether the same applies to the inno-

vative practices described in this article. To

take this a step further, Institutional Theory

can be used as a basis for developing a model

of transfer of innovations from other indus-

tries to health care. More specifically, we plan

to assess coercive and mimetic isomorphism
forces that affect health care executives and

organizations when decisions to adopt prac-

tices from other industries are made.
64

Another intriguing issue is to examine

whether transfer of learning happens both

ways: do other industries learn from health

care organizations? In their book: Manage-

ment Lessons From Mayo Clinic,’’ Berry and
Seltman

65
claim that lessons of organizational

efficiency and interpersonal relationships

learned at the Mayo Clinic can be applied

to any organization in any industry.

In conclusion, innovative practices have

been imported from various industries to

health care organizations in efforts to ad-

vance patient safety, enhance quality of care,
reduce waste and inefficiency, and improve

customer service and satisfaction. Although

several success stories have been documented,

serious questions remain about the value of

these imported practices in improving the

performance of health care organizations.

REFERENCES

1. Coyle C. Hospital CEO pay: Maryland Hospital Associa-

tion responds. The Baltimore Sun. 2010. http://articles

.baltimoresun.com/2010-09-02/news/bs-ed-hospital-assn-

letter-20100902_1_hospital-ceo-hospital-leaders-maryland-

hospital-association. Accessed March 12, 2010.

2. Drucker P. Managing in the Next Society. New

York: St Martin Press; 2002.

3. Nance J. Why Hospitals Should Fly? Bozeman, MT:

Second River Healthcare Press; 2008.

4. Gawande A. The Checklist Manifesto: How to Get

Things Right? New York: Metropolitan Books; 2010.

5. Lee F. If Disney Ran Your Hospital: 9½ Things You

Would Do Differently. Bozeman, MT: Second River

Healthcare Press; 2004.

72 THE HEALTH CARE MANAGER/JANUARY–MARCH 2012

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

6. Smith M. Disruptive innovation: can health care

learn from other industries? A conversation with

Clayton M. Christensen [published online ahead of

print March 13, 2007]. Health Aff (Millwood). 2007;

26(3):w288-w293.

7. Black C. To build a better hospital, Virginia Mason

takes lessons from Toyota plants. Seattle PI. March

15, 2008. http://www.seattlepi.com/local/article/To-

build-a-better-hospital-Virginia-Mason-takes-1267283.php.

Accessed March 12, 2010.

8. Allen S. Busy Boston Medical Center eases delays by

keeping ‘‘customers’’ moving. Boston Globe. July 8,

2004. http://www.boston.com/globe/search/stories/

reprints/emergencyroom070804.html. Accessed March

12, 2010.

9. Saver C. How can and OR be like the Ritz-Carlton?

OR Manag. 2008;24(7):1-3.

10. Naik, G. Hospital races to learn lessons of Ferrari

crew. Wall Street Journal. November 14, 2006. http:

//www.post-gazette.com/pg/06318/738252-114.stm.

Accessed March 12, 2010.

11. Testa PA, Gang M. Triage, EMTALA, consultations,

and prehospital control. Emerg Med Clin North Am.

2009;27(4):627-640.

12. Altman S, Shactman D, Eilat E. Could hospitals go the

way of U.S. airlines? Health Aff (Millwood). 2006;

25(1):11-21.

13. Kaissi A. Manager-physician relationships: an orga-

nizational theory perspective. Health Care Manag.

2005;24(2):165-176.

14. Begun J, Kaissi A. Uncertainty in health care en-

vironments: myth or reality? Health Care Manag

Rev. 2004;29(1):31-39.

15. Bagian J. Patient safety: lessons learned. Pediatr

Radiol. 2006;36:287-290.

16. Helmreich RL. On error management: lessons from

aviation. Br Med J. 2000;320:781-785.

17. Helmreich RL, Wilhelm JA. Outcomes of crew

resource management training. Int J Aviat Psychol.

1991;1:287-300.

18. Aggarwal R, Undre S, Moorthy K, Vincent C, Darzi A.

The simulated operating theatre: comprehensive

training for surgical teams. Qual Saf Health Care.

2004;13(1):i27-i32.

19. Gore DC, Powell JM, Baer JG, et al. Crew resource

management improved perception of patient safety in

the operating room. Am J Med Qual. 2010;25(1):60-63.

20. Hunt GJ, Callaghan KS. Comparative issues in aviation

and surgical crew resource management: (1) are

we too solution focused? ANZ J Surg. 2008;78(8):

690-693.

21. Penson RT, Svendsen SS, Chabner BA, Lynch TJ Jr,

Levinson W. Medical mistakes: a workshop on per-

sonal perspectives. Oncologist. 2001;6(1):92-99.

22. Leonard M, Graham S, Bonacum D. The human

factor: the critical importance of effective teamwork

communication in providing safe care. Qual Saf

Health Care. 2004;13(1):i85-i90.

23. Pronovost PJ, Goeschel CA, Olsen KL, et al. Reducing

health care hazards: lessons from the commercial avi-

ation safety team. Health Aff (Millwood). 2008;28(3):

w479-w489.

24. Morey JC, Simomn R, Jay GD, et al. Error reduction

and performance improvement through formal team-

work training: evaluation results of the MedTeams

project. Health Serv Res. 2002;37(6):1553-1581.

25. Thomas EJ, Sexton JB, Helmreich RL. Translating

teamwork behaviors from aviation to healthcare: de-

velopment of behavioral markers for neonatal resusci-

tation. Qual Saf Health Care. 2004;13:i57-i64.

26. Pape TM. Applying airline safety practices to medi-

cation administration. Medsurg Nurs. 2003;12(2):

77-93.

27. Randell R. Medicine and aviation: a review of the

comparison. Methods Inf Med. 2003;42(4):433-436.

28. Flin R, Maran N. Indentifying and training non-

technical skills in acute medicine. Qual Saf Health

Care. 2004;13(suppl 1):i8-i84.

29. Weiser TM, Haynes AB, Lashoher A, et al. Perspec-

tives in quality: designing the WHO surgical safety

checklist. Int J Qual Health Care. 2010;22(5):365-370.

30. Roberts KH. Managing high reliability organization.

Calif Manag Rev. 1990:101-113.

31. Corrigan JM, Kohn LT, Donaldson MS, eds. To Err Is

Human: Building a Safer Health System. Washington:

National Academies Press; 1999.

32. Alonso AD, Baker R, Day A, et al. Simulation-based

team training at the sharp end: a qualitative study of

simulation-based team training design, implementa-

tion, and evaluation in healthcare. Hum Resour

Manage Rev. 2006;16(3):396-415.

33. Pronovost PJ, Berenholtz SM, Goeschel CA, et al.

Creating high reliability in health care organizations.

Health Serv Res. 2006;41(4 pt 2):1599-1617.

34. Webster CS. The nuclear power industry as an

alternative analogy for safety in anesthesia and a

novel approach for the conceptualization of safety

goals. Anesthesia. 2005;60(11):1115-1122.

35. Liker JK. The Toyota Way. Madison, WI: McGraw-

Hill; 2004.

36. Thompson DN, Wolf GA, Spear SJ. Driving improve-

ment in patient care: learning from Toyota. JONA.

2003;33(11):585-595.

37. Kim CS, Spahlinger DA, Kin JM, Billi JE. Lean health

care: what can hospitals learn from a world-class

automaker? J Hosp Med. 2006;1(3):191-199.

38. Shannon RP, Frndak D, Grunden N, et al. Using real-

time problem solving to eliminate central line in-

fections. Jt Comm J Qual Patient Saf. 2006;32(9):

479-487.

39. Spear SJ. Fixing health care from the inside. Harv

Bus Rev. 2005;83(9):78-91.

40. Kim CS, Spahlinger DA, Kin JM, Coffey RJ, Billi JE.

Implementation of lean thinking: one health sys-

tem’s journey. Jt Comm J Qual Patient Saf. 2009;

35(8):406-413.

41. DelliFraine JL, Langabeer JR II, Nembhard IM. As-

sessing the evidence of Six Sigma and Lean in the

health care industry. Q Manage Health Care. 2010;

19(3):211-225.

Lessons and Challenges for Health Care Organizations 73

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

42. Winch S, Henderson AJ. Making cars and making health

care: a critical review. Med J Aust. 2009;191:28-29.

43. Linderman K, Schroeder R, Zaheer S, Choo A. Six

Sigma: a goal-theoretic perspective. J Oper Manag.

2003;21(2):193-203.

44. Betzbee P. Testing the limit of lean. HealthLeaders

Magazine. February 4, 2010.

45. Harry M, Schroeder R. Six Sigma. New York, NY:

Doubleday Publishers; 2000.

46. Chassin MR. Is health care ready for Six Sigma

quality? Milbank Q. 1998;76(4):565-591.

47. Tucci R. Six Sigma. . . without the pain. iSixSigma

Healthcare Newsletter. 2003.

48. Catchpole KR, De Leval MR, McEwan A, et al. Patient

handover from surgery to intensive care: using For-

mula 1 pit-stop and aviation models to improve safety

and quality. Pediatr Anesth. 2007;17:470-478.

49. Agnuwobi J, London PA. Removing cost from the

health care supply chain. Lessons from mass retail.

Health Aff (Millwood). 2009;28(5):1336-1342.

50. Sheehan-Smith L. Key facilitators and best practices

of hotel-style room service in hospitals. J Am Diet

Assoc. 2006;106(4):581-586.

51. Planetree. http://www.planetree.org. Accessed December

10, 2010.

52. Rogers EM. Diffusion of Innovations. New York:

Free Press; 2003.

53. Baker DP, Day R, Salas E. Teamwork as an essential

component of high-reliability organizations. Health

Serv Res. 2006;41(4 pt 2):1576-1598.

54. Heget JR, Bagian JP, Lee CZ, Gosbee JW. John M.

Eisenberg Patient Safety Award. System innovation:

Veterans Health Administration National Center for

Patient Safety. Jt Comm J Qual Improv. 2002;28(12):

660-665.

55. Dunn EJ, Mills PD, Neily J, Crittenden MD, Carmack

AL, Bagian JP. Medical team training: applying crew

resource management in the Veterans Health Ad-

ministration. Jt Comm J Qual Patient Saf. 2007;33(6):

317-325.

56. Uhlig PN, Brown J, Nason AK, Camelio A, Kendall E.

John M. Eisenberg Patient Safety Award. System in-

novation: Concord Hospital. Jt Comm J Qual Improv.

2002;28(12):666-672.

57. Malec JF, Torsher LC, Dunn WF, et al. The mayo high

performance teamwork scale: reliability and validity

for evaluating key crew resource management skills.

Simul Healthc. 2007;2(1):4-10.

58. Taylor B. One Hospital’s Radical Prescription for

Change. HBR Blog. June 2, 2010.

59. Institute for Healthcare Improvement. Going Lean

in Health Care. IHI Innovation Series White Paper.

Cambridge, MA: Institute for Healthcare Improve-

ment; 2005 (www.IHI.org).

60. PRHI. http://www.prhi.org/about_mission.php. Accessed

December 31, 2010.

61. Helmreich R, Sexton B. Managing threat and error to

increase safety in medicine. In: Dietrich R, Jochum

K, eds. Teaming Up: Components of Safety Under

High Risk. Aldershot, Hampshire: Ashgate; 2004.

62. Meyer J, Rowan B. Institutional organizations: for-

mal structure as myth and ceremony. ASJ. 1977;83:

340-363.

63. Westphal JD, Gulati R, Shortell SM. Customization or

conformity? An institutional and network perspective

on the content and consequences of TQM adoption.

ASQ. 1997;42:366-394.

64. DiMaggio PJ, Powell WW. The iron cage revisited:

institutional isomorphism and collective rationality

in organizational fields. Am Soc Rev. 1983;48:147-160.

65. Berry LL, Seltman KD. Management Lessons From

Mayo Clinic Inside One of the World’s Most Ad-

mired Service Organizations. New York: McGraw-Hill;

2008.

74 THE HEALTH CARE MANAGER/JANUARY–MARCH 2012

Copyright @ 2012 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

RESEARCH ARTICLE Open Access

Understanding innovators’ experiences of barriers
and facilitators in implementation and diffusion of
healthcare service innovations: a qualitative study
Julie Barnett1, Konstantina Vasileiou1*, Fayika Djemil2, Laurence Brooks1 and Terry Young1

Abstract

Background: Healthcare service innovations are considered to play a pivotal role in improving organisational
efficiency and responding effectively to healthcare needs. Nevertheless, healthcare organisations encounter major
difficulties in sustaining and diffusing innovations, especially those which concern the organisation and delivery of
healthcare services. The purpose of the present study was to explore how healthcare innovators of process-based
initiatives perceived and made sense of factors that either facilitated or obstructed the innovation implementation
and diffusion.

Methods: A qualitative study was designed. Fifteen primary and secondary healthcare organisations in the UK,
which had received health service awards for successfully generating and implementing service innovations, were
studied. In-depth, semi structured interviews were conducted with the organisational representatives who
conceived and led the development process. The data were recorded, transcribed and thematically analysed.

Results: Four main themes were identified in the analysis of the data: the role of evidence, the function of inter-
organisational partnerships, the influence of human-based resources, and the impact of contextual factors. “Hard”
evidence operated as a proof of effectiveness, a means of dissemination and a pre-requisite for the initiation of
innovation. Inter-organisational partnerships and people-based resources, such as champions, were considered an
integral part of the process of developing, establishing and diffusing the innovations. Finally, contextual influences, both
intra-organisational and extra-organisational were seen as critical in either impeding or facilitating innovators’ efforts.

Conclusions: A range of factors of different combinations and co-occurrence were pointed out by the innovators
as they were reflecting on their experiences of implementing, stabilising and diffusing novel service initiatives. Even
though the innovations studied were of various contents and originated from diverse organisational contexts,
innovators’ accounts converged to the significant role of the evidential base of success, the inter-personal and
inter-organisational networks, and the inner and outer context. The innovators, operating themselves as important
champions and being often willing to lead constructive efforts of implementation to different contexts, can
contribute to the promulgation and spread of the novelties significantly.

Background
The ability to innovate is considered as a major compe-
titive advantage in organisations, enhancing their effec-
tiveness, efficiency, and thus their potential for long
term sustainability [1]. The concept has been strongly
identified with manufacturing, where innovations con-
cern products and artefacts, while the service sector has,

by contrast, been seen as a “laggard” [2]. However, the
rapid expansion of the service sector in modern econo-
mies and the increasing “servicisation” of many, pre-
viously pure, manufacturing industries [3] have shifted
the focus of attention to new forms of behaviour and
activities, expressed as service innovations.
The healthcare domain, and its ability to implement

and diffuse innovations has generated intense scientific
interest (for reviews see [4,5]). The need for innovation
in service delivery and organisational functions has been
emphasised since the early 1980s [6]. Nevertheless,

* Correspondence: [email protected]
1Department of Information Systems and Computing, Brunel University
London, Uxbridge, Middlesex, UB8 3PH, UK
Full list of author information is available at the end of the article

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

© 2011 Barnett et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited.

healthcare systems in developed countries continue to
encounter considerable difficulties in implementation,
and experience major delays in diffusing novel initia-
tives, despite the perception that healthcare organisa-
tions are arguably among the most knowledge-rich and
scientifically-based institutions [7].
The imperative to innovate in healthcare has recently

intensified under the economic challenges and the
increasing demands of an ageing population. In the UK,
this has resulted in calls for reform of the National
Health Service (NHS), recently crystallised in the Equity
and Excellence: Liberating the NHS White Paper [8].
This aligned healthcare policy around the Quality, Inno-
vation, Productivity, and Prevention (QIPP) programme
initiative with the redesign of health services aimed at
improving quality while making significant efficiency
savings. The notion of innovation occupies a core posi-
tion within this reformed policy framework now placed
at the heart of the healthcare agenda.

Conceptualising healthcare innovations
Innovation can be defined as “the intentional introduc-
tion and application within a role, group or organisation
of ideas, processes, products or procedures, new to the
relevant unit of adoption, designed to significantly bene-
fit the individual, the group, the organisation or the
wider society” [9] (p.209). Anderson et al. [10] suggest
that this definition presents several advantages: firstly, it
demarcates innovation from creativity, secondly, it con-
ceptualises innovation as a deliberate effort, aiming at
benefit, and thirdly, it highlights the relativity of novelty.
Following this definition, Länsisalmi et al. [5] suggested
that healthcare innovations typically comprise “new ser-
vices, new ways of working and/or new technologies” (p.
67). These novelties “are directed at improving health
outcomes, administrative efficiency, cost effectiveness, or
users’ experience and are implemented by planned and
coordinated actions [4] (p.582). Healthcare innovations
constitute particularly complex outputs, since they fre-
quently combine both product and process novelties, or
embodied and disembodied components [11] with diver-
sified levels of materiality or tangibility.

Theoretical perspectives in healthcare innovation research
A large body of research on healthcare innovation has
been strongly influenced and shaped by Rogers’ seminal
work on diffusion of innovation [12]. Within this
approach, healthcare innovations are adopted and dif-
fused more easily when certain conditions are favour-
able. For example, the perceived relative advantage, the
potential to trial and observe the effects of the innova-
tion, and its compatibility with the values, norms and
beliefs of the adopting system are all believed to facili-
tate the diffusion [13]. Furthermore, innovations are

more likely to spread when they are consistently sup-
ported by key opinion leaders and when homogenous
groups of people sharing common values are involved.
Social networks and inter-organisational partnerships
are also recognized as highly significant forces [13-15].
Other theoretical approaches propose a more complex

and often turbulent process of innovation diffusion that
follows “a nonlinear cycle of divergent and convergent
activities that may repeat over time and at different
organizational levels”[16] (p.16). For example, research-
ers have studied the impact of scientific information on
the diffusion of healthcare innovations [17-19] and have
found that robust scientific evidence does not necessa-
rily lead to innovation adoption in a linear or direct
way. The relative advantage of healthcare innovations,
and even the evidence of benefit, is not a judgement
rooted in pure rationalistic reasoning, but rather is sub-
ject to debate and negotiation. This is partly linked to
multi-professionalism [20] characterising healthcare
where the different professional groups adhere to and
value different types of evidence, and also to the
demands stemming from the context in which evidence
is embedded and interpreted [21].
Finally, adopting a more systemic approach the theory

of disruptive innovation [22,23] has also been proposed
as a conceptual framework to enhance our understand-
ing concerning the difficulty the healthcare systems have
to lead and sustain innovation. According to this theory,
disruptive technological advancements in healthcare are
not embedded in corresponding business-model innova-
tions which would allow healthcare organisations to
take full advantage of these technological enablers and
to deliver ‘pure’ value propositions to their users. This
happens because healthcare organisations, structured
historically in the form of hospitals and physicians’ prac-
tices, conflate fundamentally different business models.
For disruptive innovations to become embedded in
healthcare, several factors need to be aligned including
suitable business models and regulatory reform while
the inherent technological characteristics of the change
and specifically, its ability to simplify are also critical
[22,23].

The aims of the present study
This study sought to shed further light on the question
of process-level innovation and to understand more
about why diffusion is apparently so poor in healthcare.
By interviewing award winners of service innovations,
we sought to understand innovation from inside the
organisation by exploring the perspectives of those who
drive it. Technology was not excluded from these case
studies but accommodating a new technology was not
the principle driver for innovation. Specifically, we were
interested in examining how healthcare innovators

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 2 of 12

perceived and made sense of their experiences of factors
that facilitated or constrained their success. We adopted
a critical realist perspective [24] which allowed us to
consider how these factors were constructed and inter-
preted in innovators’ accounts. A qualitative study, with
semi-structured interviews, was conducted among NHS
employees highly engaged with particular innovations.

Methods
Sample and recruitment strategy
Healthcare organisations in the UK, which had gener-
ated and implemented service innovations and whose
efforts had been recognized through an award, were the
study population of this research. Specifically, primary
and secondary healthcare organisations who were win-
ners of Health Service Journal (HSJ) [25] awards from
2007 to 2009 were invited to participate. The HSJ is the
premier weekly journal read by NHS managers and
healthcare professionals with a print circulation of
17.680 copies [26]. The organisations submitted an
application for inclusion in the contest, and following a
favourable assessment, presented their cases to a panel
judges in a face-to-face presentation, following which
final decisions were made. In this way, we secured a
measure of innovation that was external-to-the-innova-
tors and which provided a relatively comparable metric
of their success. HSJ awards reasonably affirmed a suc-
cessful implementation status, at least within a single
organisation. Additionally, winning these awards indi-
cated that these innovations were valued positively
within the NHS system, and that broader adoption
would be judged as desirable. Moreover, a variety of ser-
vice innovations with different aims and from diverse
contexts were targeted; all initiatives were process-
based, even though product-based innovation was some-
times part of the overall initiative.
Fifty-one organisations (20 primary care organisations,

30 secondary care organisations, and 1 educational insti-
tution in collaboration with the NHS) of HSJ service
innovation awards were approached and informed about
the study. Twenty four of these expressed an interest in
knowing more, and of these 15 agreed to take part in
interviews (response rate: 29.41%). Table 1 summarises
the participating innovations, providing a short descrip-
tion and the award title for each of them. The classifica-
tion of the innovations presented in table 1 was
developed by the researchers based on the key theme of
the innovations and their relationship to the main func-
tion of the NHS system, namely the provision of health-
care. Among the participating organisations, 5 provided
primary care and 10 secondary care. Thirteen out of
23 HSJ award categories are represented in our sample
(see table 1), while the following are not: 1. Acute Health-
care Organisation of the Year, 2. Chronic Disease

Management, 3. Communications, 4. Cost Effective Part-
nership, 5. Implementing NICE Guidance, 6. Informa-
tion-based Decision Making, 7. Patient Safety, 8. Primary
Care Innovation, 9. Reducing Health Inequalities, and 10.
Data-driven Service Improvement.
Finally, the people interviewed were directly engaged

with the service innovation; in most of the cases, they
had been involved with generating the initial concept
and had led the processes of implementation and diffu-
sion. All interviews were conducted with one innovator,
except for one case (Good Corporate Citizenship; see
table 1) where four interviewees took part since they all
had contributed to different aspects of the innovation.
As the focus of our research was on examining a range
of service innovations rather than seeking different per-
spectives around the same innovation, we chose only to
interview a single key informant around each innovation
and as part of the interview explored the role attributed
to others in directly or indirectly contributing to its
implementation and diffusion.

Data collection and ethical considerations
Semi-structured telephone interviews, lasting around 45
minutes, were conducted at scheduled times during Sep-
tember-October 2010 with key representatives of the
healthcare service innovations. Initially, people were con-
tacted by e-mail, informed about the aims of the study
and invited to take part in a telephone interview. Pro-
spective interviewees were provided with an information
sheet explaining the aims, procedure and ethical aspects
of the project, as well as the consent form in which they
confirmed their willingness to participate. Latterly, inter-
viewees were able to indicate their agreement with the
information disclosed and the validity of our analysis (i.e.
testimonial validity [27]) in a pre-publication draft paper.
Interviews included questions about the generation

and development of the service innovation, the barriers
and facilitating factors experienced, and the wider
uptake of the innovation. Rather than directing the
interviewees to discuss specific themes previously identi-
fied in the literature, we deliberately created open-ended
questions. This allowed the interviewees to pick up
spontaneously on those factors that were perceived as
most important for their innovation and then they were
prompted to discuss the issues in more depth. More
specifically, the interview schedule was comprised of the
following questions:
A. Conception and development of the innovation
Could you please tell me about (name of the

innovation)?
Where did the idea come from?
Can you tell me about the process of developing this?
How much opposition/interest did you meet along the

way?

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 3 of 12

How was the service innovation viewed by your orga-
nisation/in your local context? Did that change at all
over time?
B. Diffusion of innovation
Has your innovation been picked up by others?
How did this come about?
What have they done with it?
What do you think were the main reasons for their

interest?

C. Perceived barriers and facilitators of innovation
diffusion
What factors do you think have encouraged/con-

strained its uptake by others?
What do you think are the main reasons for this?
How could this have been different?
Participants also provided information about their role

in the organisation and the innovation process as well
as their participation in the actual HSJ competition. The

Table 1 Classification, award titles, and descriptions of innovations

Innovations directly related to healthcare provision

Categories Award titles Short description

I. Innovations of specified healthcare
provision

1. World Class
Commissioning

1. Initiative that developed a timely and coordinated hyper-acute service provision
to stroke patients with early care and fast access to services, and educated
organisations to implement stroke care according to national guidelines

2. Clinical Service
Redesign

2. Initiative that redesigned the acute stroke service offering rapid assessment and
care provision, and rehabilitation closer to home

3. Mental Health
Innovation

3. Initiative that provided evidence-based interventions to chronic respiratory
patients helping them to manage their synchronic mental health conditions (e.g.
depression)

4. Best Social
Marketing Project

4. Initiative that targeted pregnant women to access smoking cessation services,
designed on the basis of their needs

II. Innovations related to the overall
organisational function

1. Primary Care
Organisation of the
Year

1. Initiative that concerned a variety of organisational functions, from several health
care provision programmes and sound financial management, to the development
of local partnerships and the positioning of the organisation as a leader

III. Innovations related to patients’
safety*

1. Acute & Primary
Care Innovation

1. A reliability checklist that ensured the conduction and revision of appropriate
medical checks in round-wards

2. Improving Care
with Technology

2. A technologically-based innovation that secured adherence to evidence-based
guidelines during the blood transfusion process, reducing errors/omissions,
paperwork, process time per patient and staff capacity requirements

IV. Innovations of patients’ access and
reception of healthcare

1. Improving Patient
Access

1. Initiative that improved prisoners’ access to healthcare services through their
involvement and the development of the scheme of “prisoner healthcare
representatives”

2. Patient Centred
Care

2. Initiative that applied a patient-centred model of care giving patients choice,
involvement and control over their care experience

V. Innovations of educational services 1. Mental Health
Innovation

1. A training programme that educated professionals in evidence-based family
interventions, increasing their awareness to the needs of carers of mentally ill
patients

2. Skills Development 2. A training programme that educated diabetes diagnosed patients to better self-
manage their condition and provided continued professional development to
trainers and educators.

Innovations less related to healthcare provision

Categories Award titles Short description

VI. Innovations related to human
resources

1. Recruitment &
Retention

1. A human-resources and workforce development initiative that reformed the
recruitment and retention practices through the provision of training programmes
and employment opportunities to local unemployed and excluded groups

2. Workforce
Development

2. A workforce transformation programme that prepared staff to provide care
closer to home

VII. Innovations related to other
organisational functions (e.g. logistics)

1. Good Corporate
Citizenship

1. Initiative that introduced a transportation scheme, IT developments and new
methods of food procurement and waste management which increased
organisational sustainability and supported the local economy

2. Best Social
Marketing Project

2. Initiative that provided free access to leisure facilities to disadvantaged groups of
citizens in collaboration with the city council

* Technological products were incorporated

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 4 of 12

study received ethical approval from the Research Ethics
Committee of Brunel University London.

Analytic procedure
A thematic analysis, informed by a critical realist stance
[24] was conducted on the transcribed interviews. A cri-
tical realist epistemological position offers the possibility
of accounting for the ways in which the social world is
constructed through language, whilst simultaneously
recognizing the existence of an external-to-discourse
reality. This reality is constituted by the material and
institutional world and is considered to influence the
meanings and the discourses which people invoke to
construct the objects at stake [28]. The employment of
a critical realist perspective allowed us to locate innova-
tors’ experiences within the broader material and insti-
tutional contexts they operated.
Thematic analysis was considered to be an appropriate

technique identifying as it does, “repeated patterns of
meaning” across a set of data [29,30] (p.86). An induc-
tive approach with a full presentation of themes was
selected and the data were analysed semantically,
emphasising the explicit meaning of the themes. In
addition to description, an interpretation of the themes
is provided.
There were four stages in the analysis. Initially, there

was a familiarisation process with the data; then, an
initial coding took place, where particular extracts were
named and defined. Next, themes and subthemes were
developed by aggregating the respective coded segments.
The construction of the comparative analytical cate-
gories was assisted by the use of computer software
[31]. Finally, the themes and subthemes were revised
and refined, ensuring that the criterion of internal
homogeneity and external heterogeneity [32] was met
satisfactorily, and that the themes reflected the data.
Initial coding of the data was applied by KV, and the
themes and sub-themes were developed, revised and
refined by JB and KV.

Results
Four themes were identified in the analysis and are pre-
sented in the following sections: (a) the role of evidence,
(b) the role of partnerships, (c) the influence of cham-
pions and other human-based resources, and (d) the
impact of contextual factors both organisational and
external. All of these played a role, alone or in combina-
tion, to facilitate or block implementation or diffusion
of the innovation.
(In order to protect the participants’ anonymity, as

they are drawn from a narrow range of award years,
after each quote below, the innovation is identified by
the extent to which it is related to the healthcare provi-
sion-see Table 1 for the relevant categorisation).

A. The role of evidence
Evidence was considered to play a crucial role. This was
visible at multiple time points in the initiation, imple-
mentation and diffusion of service innovations. Specifi-
cally, there was a focus on quantitative evidence and
this was seen to operate in three main ways: (a) as a
proof of effectiveness, (b) as a means of diffusion and
(c) as a precondition for the initiation of the innovation.
A1. Evidence as a proof of effectiveness
“Hard” evidence, in the form of quantitative data, was
perceived as the ‘gold standard’ demonstration of effec-
tiveness, constituting an optimal and credible base for
assessing an innovation. It equipped innovators to attest
to the usefulness and success of their initiatives, and to
persuade prospective adopters that they were valuable.
A participant characteristically commented:

“So our outcomes are important to us, because they
give us sort of a language by which we can sort of
articulate the success of it.” (Innovation less related
to the healthcare provision)

Even when the innovators considered that “soft”
aspects of the impact were valuable and significant for
the demonstration of usefulness, they also attempted to
corroborate this with numerical evidence. The latter
represented the indisputable metric needed to buttress
accounts of “soft outcomes”.

“All the guys that work for me work for nothing,
and the job satisfaction of seeing these guys change
and…build confidence…is just absolutely amazing,
and I think, if nothing else, if people want to call it
soft outcomes-I mean, we have got figures of how
we’ve improved stuff, you know…I can give you
some statistics on that, on how we’ve improved stuff
in 2009.” (Innovation directly related to the health-
care provision)

Lack of quantitative evidence was seen as a notable
shortcoming not only for the sustainability and diffusion
of the innovation but also for its initiation. This percep-
tion was particularly pronounced where there were
other barriers and when the initiative was peripherally
linked to the core business of the NHS. In such cases,
quantitative evidence that would link the innovation to
health-related outcomes was imperative.

“One of the things which we need to expand is the
health impact of schemes like this (social inclusion
schemes), whilst we have a lot of anecdotal evidence
we need to gather empirical evidence to demonstrate
the long term benefits. In an increasingly challenging
fiscal climate, supporting evidence may encourage

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 5 of 12

organisations to make the investment in this type of
development.” (Innovation less related to the health-
care provision)

Within certain professional groups, almost notably clini-
cians, “hard” evidence which would be obtained from
scientific methods was construed as the necessary prere-
quisite for the demonstration of the innovation impact,
without which any persuasive effort was doomed to failure.
Anecdotal or experiential testimonies were unable to exert
any significant influence; scientific data were seen as the
only basis for a process of persuasion, and on which pro-
spective adopters could make informed decisions.

“For some reason, with doctors, if you haven’t got
some data, and maybe a p-value, then it’s really hard
to convince them that something works. What we’ve
found it is, if we just give it to people and tell them
it’s a good idea, they don’t believe us; whereas, if we
give it to people and we show them our chart, where
we can show that, when we use the checklist, we
have a massive change, and suddenly we’ve got a p-
value, that seems to win hearts and minds.” (Innova-
tion directly related to the healthcare provision)

A2. Evidence as a means of diffusion
“Hard” evidence was also a sound means of dissemina-
tion, both intra-organisationally and inter-organisation-
ally. Encouraging health outcomes based on empirical
data led organisations, in some instances, to propose the
adoption of innovation to other departments. Moreover,
the availability of evidence enabled innovators to neutra-
lise resistance and opposition.

“…So instead of taking an aggressive approach to
that, we backed it up with data…So we simply just
sent the audit data round, which picked up a few
people.” (Innovation directly related to the health-
care provision)

Similarly, evidence was seen to help innovators obtain
human and financial resources needed to expand geo-
graphically and inter-organisationally.

“…and then we actually audited the results of their
intervention…and actually evidenced that the team
were having an effect on patients’ wellbeing, and
that led to the team being enhanced staff-wise, and
the service then rolling out city-wide.” (Innovation
directly related to the healthcare provision)

A3. Evidence as a precondition
Quantitative data were often seen as a base and precon-
dition for the initiation of the innovation, indicating the

need for change or the scope for benefit. Pre-existing
evidence, which was available before the initiation of the
innovation, was used to argue, justify and bolster the
case for the developing initiative and it clarified innova-
tors’ incentives and intentions, particularly when resis-
tance was anticipated. In this way, uncertainty was
reduced and the stakeholders involved could estimate
the potential risks and benefits.

“I personally was surprised that there wasn’t more
opposition really to closure than that, but I think
people were genuinely, staff included, genuinely con-
vinced by the evidence supporting why this was the
right move and what it would mean.” (Innovation
directly related to the healthcare provision)

B. The role of partnerships
Inter-organisational connections, either formalised as
partnerships or loosely linked, constituted an integral
part in the process of developing, establishing and dif-
fusing the innovations. In some instances, the partner-
ships were seen to be part of the essence of the
innovation itself.
Existing working relationships between partners’ orga-

nisations were often identified as the starting point of
the innovation and the driving force for its development.
Trust and mutual support were vital prerequisites for
cooperation, since they ensured that the decisions and
commitments made would be adhered to by all parties.
The importance of trust was amplified when there was
high uncertainty around what would follow.

“But I think why it worked for us was that we had a
combination of a good working relationship in an
environment where the decisions could be made by
those people, so you already had some trust, you
already had confidence that partners were going to
pull their weight…I mean what the PCT [Primary
Care Trust] didn’t know was whether people would
come and use their offer, but what they did know was
that if they gave the city council in our area the
money, we’d do with it what we said we’d do with it.”
(Innovation less related to the healthcare provision)

The building of partnerships was a goal of innovators
in their attempt to initiate and establish their services,
and the lack of them was often assumed to be a reason
for failure of previous initiatives, even when these might
be well supported financially. The quote below illus-
trates this point.

“Then, as we were looking around, we realised we
hadn’t really got any formal links with other

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 6 of 12

services…and actually, whilst we’d got a good level of
resource going into the [name of previous service],
we weren’t getting the impact that we wanted. So,
that’s where it started from.” (Innovation directly
related to the healthcare provision)

Having built supportive partnerships, the innovation
gained more chances of sustainability in the long term,
since partners represented a securing mechanism, war-
ranting and endorsing the continuation.
Dissemination of service innovations was also seen to

be contingent upon partnerships. Partners who had an
interest in the novelty were perceived as significant in
promoting and publicising the message beyond organi-
sational boundaries.

“…and we’ve got some very good relationships with
voluntary organisations anyway, but to just obviously
build on that, do more promotions out in the com-
munity, working with organisations in obviously pro-
moting what we’re doing and how that could fit into
the outside world.” (Innovation directly related to
the healthcare provision)

Importantly, partnerships were sometimes seen as one
of the beneficial side-effects that enabled people to con-
struct a common communication framework and a
mutually shared agenda, potentially useful for future
interactions and collaborations. Interestingly, in some
instances the emergent partnerships themselves were
identified as a novel characteristic of the service, usually
when the collaborations were believed to be exceptional
and rare in the field of healthcare.

“It is the process and, you know, great credit to the
PCTs across [city name] to agree to work together.
That isn’t the same in lots of other areas. And so the
structures that were put in place and the way that
they worked closely with the clinicians and the wider
stakeholder group certainly was innovative.” (Innova-
tion directly related to the healthcare provision)

Finally, and in retrospect, success was perceived to
have been unlikely without the a priori consensus of the
involved partners. Proactive engagement and dialogue
were vital because organisations had the opportunity to
agree on the principal elements of the initiative, and in
turn the perceived risks were minimised, the undertak-
ing was legitimised and the partners were committed to
support the initiative and to follow the rules.

“…I think, you know, because we had great stake-
holder engagement, and we’d started off with a big
consensus event where we’d agreed some basic

principles that we would abide by, then I think
everybody felt comfortable.” (Innovation directly
related to the healthcare provision)

Proactive engagement with partners was also seen as
an effective strategy against future resistance and able to
mitigate potential obstructions, particularly when the
innovation was perceived to be radical and to diverge
significantly from the existing norms.

“So we knew that we’d have to come up with some-
thing completely different, so we engaged key stake-
holders at the outset… because we involved them at
the beginning we didn’t receive any opposition.”
(Innovation directly related to the healthcare
provision)

C. People-based resources
People within and outside the organisations were per-
ceived as particularly significant, either in facilitating or
inhibiting the innovation journey.
Most interviewees highlighted the importance of

champions, who could be employees in various organi-
sational positions, people in the local community or the
users of the innovation. Importantly, the innovators
themselves, in being passionate about and committed to
their initiatives, ultimately became champions.
The role of top and senior management was critical,

since the financial support of the initiative, and thus its
sustainability and success, was often contingent upon
their decisions. Public espousal of the core ideas of an
innovation was also represented as a key resource, able
to transfer new knowledge necessary for the advance-
ment of the initiative. The interviewee below, comment-
ing on the significance of champions, said:

“It is important. We’re fortunate, we’re led from the
top, our Chief Executive and our Chairman are very
passionate, as are our Trust Board, about sustainabil-
ity. We’ve also got an environmental awareness cam-
paign that we started again in the New Year that’s
just gone. We’ve now got 147, I think it is, environ-
mental champions across the Trust, that are driving
it forward, and we’re recruiting all the time. These
are obviously voluntary posts, but the passion and
the commitment out there is absolutely fantastic. So
I don’t think it would stop. I think it would take it
to new levels, because people are bringing in things
that we’ve never even thought of.” (Innovation less
related to the healthcare provision)

The users of the novel services could also constitute
powerful champions, as they were able to circulate their

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 7 of 12

experience to the local community, thus promulgating
the new service and counteracting possible resistances.

“The students themselves who have come through
the Academy have demonstrated behaviours which
are exemplary and they have been the greatest advo-
cates for the Academy,-the reputation of the Acad-
emy has allowed it to grow, we do still have people
who are anti the Academy, which is reflective of
working with excluded groups, but we have far more
supporters.” (Innovation less related to the health-
care provision)

Employees were seen as a vital channel for intra-
organisational and inter-organisational diffusion, since
they can persuade their colleagues informally or influ-
ence decisions directly, especially when they occupy
key positions. They were considered as powerful advo-
cates when they experienced beneficial results in their
daily working routines and regarded the innovation as
advantageous. By contrast, employees affected nega-
tively can put up barriers that require effort to sur-
mount through proactive engagement and timely
information provision.

“…the way that we did that was to just start off with
very small pilots in non-acute areas, and develop a
process which was so much better for the nursing
staff that they were, you know, they were so
delighted to have it that they would, you know, they
would work with us and become advocates for us
when we went to new clinical areas.” (Innovation
directly related to the healthcare provision)

Barriers to the implementation and diffusion of inno-
vations were perceived to arise when the innovators and
the decision-makers belonged to different professional
groups. Different educational backgrounds, organisa-
tional roles, and diverse worldviews resulted in different
priorities, which could delay or obstruct the spread of
innovation. In such cases, innovators had to devote
much effort to persuading decision makers of the useful-
ness of the initiative.

“…sometimes the people who are in charge of the
budgets are not necessarily very familiar with clinical
priorities. So they might be somebody who’s actually
an accountant, who’s responsible for helping the
PCT decide which disease areas and which services
to put their money in. So you have to…be prepared
to almost educate people about why what you’re
doing is important.” (Innovation directly related to
the healthcare provision)

D. Contextual factors
The context, both intra-organisational and extra-organi-
sational, was also perceived to decisively influence the
life-cycle of the healthcare innovations.
D1. Intra-organisational context
three basic subthemes were identified, relating to: (a)
organisational receptiveness, (b) available resources, and
(c) organisational capability to promote the innovation.
a. Organisational receptiveness A series of long-term
changes often preceded implementation of the service
innovation, especially when the latter was large-scale
and system-wide. These changes were believed to pre-
pare the organisation structurally and functionally to
receive the novelty smoothly, especially when the
impending initiative was complex and multifaceted. In
this case, the process of implementation and spread was
constructed as an incremental change, embedded in an
already changing system.

“…we’d been developing our pathway for stroke and
aspects of it, year on year, since 2000, and I think it
was that foundation that truly enabled us to respond
in the way that we did and deliver that [innovation].”
(Innovation directly related to the healthcare
provision)

Organisational culture was also perceived to be a criti-
cal factor. Specifically, the openness of the organisation
to trial new ideas and carry the associated risks was
seen as significant, particularly when the change was not
triggered by external factors, such as policy initiatives,
or an obvious and urgent organisational need.

“…there was an environment of being prepared to
take a risk, with the right kind of conditions to sup-
port that.” (Innovation less related to the healthcare
provision)

Equally important for diffusion was the fit between the
innovation and the organisational ethos. When innova-
tors’ values were perceived to be congruent with prevail-
ing organisational norms and beliefs, the diffusion was
facilitated, since the novelty affirmed the cultural organi-
sational orientation. By contrast, when the innovation
collided with basic organisational principles, resistance
emerged and dissemination was impeded.

“You know, it’s just changing-it’s a big culture
change, and it does meet with controversy and it
does meet with people who still feel that prisoners
shouldn’t have any rights at all, and so you are con-
stantly coming up against that.” (Innovation directly
related to the healthcare provision)

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 8 of 12

b. Available resources Sufficient human and financial
resources were of paramount importance not only for
the proper implementation of service innovations but
also for their diffusion to other organisations, sectors
and fields of practice. Shortage of resources, or fear of
this, could block innovators’ efforts and led to stagna-
tion. An interviewee, employing the metaphor of
“paralysis”, commented:

“It’s very clear what we need to do in stroke, and
sometimes, just the paralysis is just that the money
isn’t there to develop early supported discharge.”
(Innovation directly related to the healthcare
provision)

c. Organisational capability to promote the innova-
tion Innovators’ own capability to promote their initia-
tives within and beyond their organisation was
considered to facilitate diffusion. Specifically, awards,
media attention and the possibility of academic publica-
tions were viewed as a powerful means of communicat-
ing the innovation to the wider publics.

“When we got that far, and, crucially publicised what
we’d been doing-enter awards, produced journal
publications-which was very helpful to us. You
know, having the picture of the process on the front
page of the premier international transfusion journal,
things like this.., trying to develop some momentum
behind it.” (Innovation directly related to the health-
care provision)

Winning awards was experienced as a crucial social
recognition in three main ways: firstly, awards were per-
ceived as an external and unbiased validation of the
innovation, against which it was difficult for doubters to
argue. Secondly, awards were seen to raise the profile of
organisation and its reputation of being innovative,
which in turn helped it to build further networks and
inter-organisational collaborations. It also identified the
organisation as an early adopter. Thirdly, innovators
regarded the awards as an effective means of promoting
an agenda, which in turn could attract further resources.

“…if you do have a good profile nationally, regionally,
things come to you. You know, you are invited to
participate in things, to be early adopters and, that’s
for the benefit of the people that we serve really, so
it’s not just about the glory-it’s about being at the
table…” (Innovation directly related to the healthcare
provision)

Active promotion of the innovation with other organi-
sations was presented as essential and innovators

believed that they should communicate and publicise on
an on-going basis. In this way an extrovert organisa-
tional culture is cultivated. This sharing-behaviour
enabled innovators to identify pitfalls and advise pro-
spective adopters of the most promising ways of imple-
mentation.

“You know, that’s one of the big ethos behind kind
of the networks really is, it’s around sharing. It’s not
about keeping it to yourself. It is definitely around
sharing good practice, sharing what’s learnt, and
also, hopefully things that went wrong for us, shar-
ing that as well and saying this is how…don’t do
that, don’t go down off that road because we tried
that and it didn’t work, as much as it is about saying
this route worked really well.” (Innovation directly
related to the healthcare provision)

Innovators recognised that diffusion would involve
adaptation to the new context. “Re-invention” [12]
rather than replication was seen as an imperative for
prospective adopters. In the context of complex organi-
sational settings, interviewees stressed that to “survive”
and be successful was only possible if necessary adapta-
tions and adjustments were made.

“And also everybody, to some extent, has to evolve
these things for their own circumstances, don’t
they?….It’s not a one-size fits all.” (Innovation
directly related to the healthcare provision)

D2. Extra-organisational influences
Three main external influences were identified in inno-
vators’ accounts: (a) economic, (b) political, and (c)
ideological.
a. Current economic climate The constrained eco-
nomic climate was often cited as inhibiting initiatives
which were expensive and did not save costs directly.
Such innovations were unlikely to attract funding and
sustained financial support, since they were seen to
oppose the measures needed in challenging economic
conditions.

“I think we did it at exactly the right time, because I
think, now, it’s quite a difficult time in terms of
obviously, the economic climate. It’s really difficult
in terms of funding that’s available, and actually,
there needed to have been some pump-priming
upfront to be able to deliver this.” (Innovation
directly related to the healthcare provision)

b. Political influences Politics was constructed both as
a positive and a negative force in diffusion efforts. In
terms of benefit, the presence of regulatory bodies that

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 9 of 12

would shape practices around specific innovations was
considered to facilitate certain initiatives. Conversely, it
was more problematic where there was a more fragmen-
ted landscape of accountability and no clear responsible
body to align stakeholders’ activities.

“The problem is, there is actually no national body,
as I said again, responsible for food. If you ring the
Department of Health and say “Who’s responsible
for food in the NHS?” you’ll get a complete blank.”
(Innovation less related to the healthcare provision)

Additionally, when the innovation was believed to
entail political risks or ran against the dominant political
forces within the local context, the diffusion met severe
challenges.
c. Ideological influences The last critical factor for suc-
cess was the perceived fit of the innovation with the
broader ideological context, both within and outside the
healthcare sector. When an innovation was viewed to
reflect dominant ideological beliefs and to be consistent
with the “spirit of the times”, initiatives were more likely
to become established. This was especially so for those
innovations which were peripherally linked to the core
function of the NHS. In this case, innovators had to
resort to ideological resources external to the domain of
healthcare, such as environmentalism, in order to
endorse the value of their initiatives.

“I was going to say this is a really good period of
time to be doing these things because there is a gen-
eral awareness, there are all sorts of things that the
Trust has to do in saving energy. People think it’s a
good idea. The community thinks it’s a good idea. It
costs you if you don’t.” (Innovation less related to
the healthcare provision)

Discussion
This study sought to examine the subjective experiences
and interpretations of factors facilitating or blocking the
implementation and diffusion of process-based health-
care innovations. It did this by exploring innovators’
own accounts of these processes. Overall, our results eli-
cited themes commonly found in the literature of inno-
vation diffusion, echoing previous studies [33,34].
Significantly, the notion of evidence consistently
emerged as the key leitmotif in narratives of the innova-
tion journey. The development of social networks, both
inter-personal, expressed through champions and advo-
cates, and inter-organisational, was an additional critical
theme, while both the immediate organisational context
and the wider socio-political and economic environment
were recurrently articulated as major influencing factors.

Evidence was constructed as a powerful parameter
that provided innovators with a sound base for their
own assessment in turn allowing the initiation and diffu-
sion of innovation. Evidential knowledge constituted a
transparent, unbiased and credible source, from which
innovators could extract their arguments and could
structure their persuasive efforts in terms of innovation
utility and effectiveness. As May et al. [35] suggested
“evidential knowledge serves a stabilising purpose for
ideational claims” (p. 703). The desire for “hard” or
numerical evidence dominated. This was evident, not
simply around those innovations which were more clini-
cally-oriented but was also considered vital for innova-
tions less centrally-related to healthcare provision or led
by non-medical staff. Here too numerical or financial
descriptors were aspired to; they provided ideal metrics
for indicating the impact of the innovation. Importantly,
a polarisation was observed with innovators often con-
trasting numerical evidence with experiential testimonies
or anecdotal evidence, while the potential for a rigorous
qualitative assessment of the value of the initiative was
absent from peoples’ accounts. The strong reliance on
quantification and the accompanying disregard of
experiential knowledge is reflective of the pursuit of
objectivity. Quantitative data and the highly structured
rules for producing this have been rendered as a power-
ful tool for conferring trustworthiness to knowledge
claims, appearing exempt from subjective judgments
and local singularities [36].
Although evidence was depicted as a powerful tool

from the innovators’ point of view, they did not claim
that the evidence they had assembled were necessarily
the most accurate or reliable metrics of innovation
effectiveness or consider that it would be uncontested
and readily acceptable among a broader range of stake-
holders. Indeed, the innovators were often aware of the
weaknesses or deficiencies of their approach and
expressed a desire for more or stronger evidence [Vasi-
leiou, Barnett, Young, unpublished data]. The evidence
they were able to collect and produce seemed enough to
convince immediate stakeholders but how compelling it
was considered to be for a broader audience was much
more questionable.
Arguably, the ubiquitous preoccupation with evidence

reflects the strong profile of the “evidence-based prac-
tice” movement within healthcare sector since the early
1990s [37]. Though deriving from the medical commu-
nity, our findings suggest that representations of evi-
dence-based practice and of its value have been
assimilated by other professional fields within health-
care. The sort of evidence perceived as adequate and
thus persuasive varied considerably across the profes-
sional memberships of innovators, with medical staff
espousing almost exclusively scientifically derived

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 10 of 12

evidence, while other professionals contented themselves
with statistics and financial figures. However, conviction
concerning the necessity, transparency and objectivity of
empirical data for the audit of innovations was common
to all. Ultimately, evidence was constructed as the tool
which would legitimate an unproblematic and direct dif-
fusion of innovations within a sector that traditionally
relied on scientific knowledge. Other sources of knowl-
edge, such as experience, would fail to do so especially
under the burden of uncertainty and risks that any orga-
nisational change embodies. Consequently, evidence
constituted a stable and substantial reference point from
which arguments of innovation utility could be justified
and practices of persuasion could be initiated.
Several types of inter-organisational links (e.g. struc-

tural, administrative, institutional, or resource links)
have been conceptualised as antecedents of organisa-
tional innovativeness [14]. In this study partnerships
were not only seen as a prerequisite of innovation, but
also as a result of innovation; inter-organisational links
were part of the essence of the innovation itself. This
was highly valued and constructed as an important
“legacy” for the local community.
Our findings suggest that inter-organisational links

served two important and complementary purposes:
material and symbolic. Materially-based partnerships
provided the innovative organisation with the necessary
resources, required for the implementation and diffusion
of initiatives. Symbolically, inter-organisational
exchanges allowed organisations to gain local consensus
and therefore to bolster the new service with legitimacy.
Particularly when innovations were perceived to be radi-
cal, proactive engagement with various stakeholders was
common. Consequently, inter-organisational collabora-
tion was not only seen as vital to securing resources but
also as an important social exchange that assisted with
powering innovation through gaining a broader consen-
sual base.
Reinvention [12] on the part of prospective adopters

was a common theme in that innovators expected
others to adapt and modify the innovation in the new
context thus increasing the likelihood of sustainable dif-
fusion. For those leading the development of new ser-
vices in healthcare settings, having an identity of
“successful innovator” was both feasible and desirable. It
was thus vital to the maintenance of that identity that
potential adopters in other healthcare organisations
should understand that the initiatives were highly con-
text-specific (see also [38]) and thus their active adop-
tion in new contexts all but constituted another
innovation.
Finally, our findings indicate the importance of a sup-

portive environment for the establishment and diffusion
of service innovation along with the technological

enablers, as this is proposed in the theory of disruptive
innovation [22,23]. The example of the multilateral
innovation around sustainability is informative: most of
the single initiatives within this innovation-green IT
developments, sustainable transportation scheme-reso-
nated with the broader spirit of environmentalism and
energy-saving policies and were implemented success-
fully. However there was a particular case-the construc-
tion of a sustainable food procurement unit-which, even
though it could bring significant cost-savings and was
consonant with the sustainability agenda adopted by the
Trust, was nevertheless severely impeded due to the
absence of a regulatory body within healthcare that
would align, support and coordinate the relevant
activities.

Limitations and strengths of the present study
One limitation of this research is its cross-sectional
design which precludes an examination of the underly-
ing processes of innovation initiation, implementation
and diffusion, as a longitudinal study would have done.
However, this study provided the unique insights and

experiences of healthcare innovators who had conceived
and led process-oriented innovations. Innovators can
contribute significantly in the diffusion of new initia-
tives, as they often appear willing to lead constructive
efforts of dissemination, operating as powerful cham-
pions. They are able to advise and indicate the most
promising ways of adoption and implementation, since
they carry valuable experience of their own efforts to
implement the initiative in their organisations. However
attention should be paid as to what stages of diffusion
innovators are more likely to contribute positively, since
high levels of champions’ identification with their role
or organisation may actually impede further innovation
diffusion [39]. On-going support of healthcare innova-
tors, especially in their attempts to promulgate and pub-
licise the novelty, is crucial for the dissemination of new
initiatives.

Conclusions
By interviewing key organisational representatives who
had developed and established a range of healthcare ser-
vice innovations from a variety of healthcare sectors we
attempted to understand the factors obstructing or facil-
itating the innovation implementation and diffusion. A
set of common determinants was identified across inter-
views pertaining to the availability of quantitative evi-
dence, the building of trustworthy partnerships, the
support from human resources, and the existence of a
favourable inner and outer context. Innovators repeat-
edly stressed the necessity of innovation adaptation if it
were to be implemented in a different context, suggest-
ing innovators’ awareness of the context-specific

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 11 of 12

character of the innovations, and their desire to defend
an identity of successful innovator. Finally, the contribu-
tion of innovators to the promulgation and dissemina-
tion of the novel message beyond the boundaries of
their organisation may be beneficial in guiding and
advising prospective adopters in their own effort to
introduce change.

Acknowledgements
We would like to thank all participants who willingly took part in our study.
This research was supported financially by the Multidisciplinary Assessment
of Technology Centre for Healthcare (MATCH). The funding body did not
have any role in the study design, the collection, analysis and interpretation
of the data, in the writing of the paper, and in the decision to submit the
manuscript for publication.

Author details
1Department of Information Systems and Computing, Brunel University
London, Uxbridge, Middlesex, UB8 3PH, UK. 2Systems Engineering and
Human Factors Department, Cranfield University, Cranfield, Bedfordshire,
MK43 0AL, UK.

Authors’ contributions
JB conceived of, designed and carried out the study. JB also analysed and
interpreted the data and contributed to the writing of the manuscript. KV
analysed and interpreted the data and contributed to the writing of the
manuscript. FD and LB carried out the study. TY conceived of and designed
the study. All authors read, critically assessed and approved the final
manuscript.

Competing interests
The authors declare that they have no competing interests.

Received: 25 July 2011 Accepted: 16 December 2011
Published: 16 December 2011

References
1. Tidd J, Bessant J: Managing Innovation: Integrating Technological Market and

Organizational Change. 4 edition. Chichester: John Wiley & Sons, Ltd.; 2009.
2. Miles I: Services in the new industrial economy. Futures 1994, 25:653-672.
3. Howells J: The nature of innovation in services. In Innovation and

Productivity in Services. Edited by: Pilat D. Paris: OECD; 2001:55-79.
4. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of

innovations in service organisations: Systematic review and
recommendations. The Milbank Q 2004, 82(4):581-629.

5. Länsisalmi H, Kivimäki M, Aalto P, Ruoranen R: Innovation in healthcare: A
systematic review of recent research. Nurs Sci Q 2006, 19(1):66-72.

6. Hunter DJ: Promoting innovation in the NHS. Br Med J 1983, 286:736-738.
7. Berwick DM: Disseminating innovations in health care. J Am Med Assoc

2003, 289(15):1969-1975.
8. Department of Health: Equity and Excellence: Liberating the NHS London:

Stationary Office; 2009 [http://www.dh.gov.uk/prod_consum_dh/groups/
dh_digitalassets/@dh/@en/@ps/documents/digitalasset/dh_117794.pdf].

9. West MA: The social psychology of innovation in groups. In Innovation
and Creativity at Work: Psychological and Organizational Strategies. Edited by:
West MA, Farr JL. Chichester: Wiley; 1990:309-334.

10. Anderson N, De Dreu CK, Nijstad BA: The routinization of innovation
research: A constructively critical review of the state-of-the-science. J
Organ Behav 2004, 25:147-173.

11. Bower DJ: Innovation in healthcare delivery. In Service Innovation:
Organizational Responses to Technological Opportunities & Market Imperatives.
Edited by: Tidd J, Hull FM. London: Imperial College Press; 2003:211-230.

12. Rogers EM: Diffusion of Innovation. 3 edition. New York: The Free Press;
1983.

13. Cain M, Mittman R: Diffusion of innovation in healthcare. iHealth Reports
Oakland, California: California Healthcare Foundation; 2002.

14. Goes JB, Park SH: Interorganizational links and innovation: The case of
hospital services. Acad Manage J 1997, 40(3):673-696.

15. Sibthorpe BM, Glasgow NJ, Wells RW: Emergent themes in the
sustainability of primary health care innovations. Med J Aust 2005,
183(10):S77-S80.

16. Van de Ven AH, Polley DE, Garud R, Venkataraman S: The Innovation Journey
Oxford: Oxford University Press; 1999.

17. Denis JL, Hebert Y, Langley A, Lozeau D, Trottier LH: Explaining diffusion
patterns for complex health care innovations. Health Care Manage R
2002, 27(3):60-73.

18. Fitzgerald L, Ferlie E, Wood M, Hawkins C: Interlocking interactions, the
diffusion of innovations in healthcare. Hum Relat 2002, 55(12):1429-1449.

19. Fitzgerald L, Ferlie E, Hawkins C: Innovation in healthcare: How does
credible evidence influence professionals? Health Soc Care Comm 2003,
11(3):219-228.

20. Ferlie E, Fitzgerald L, Wood M, Hawkins S: The non-spread of innovations:
The mediating role of professionals. Acad Manage J 2005, 48(1):117-134.

21. Dopson S, Fitzgerald L: Knowledge to Action: Evidence-Based Healthcare in
Context Oxford: Oxford University Press; 2005.

22. Christensen CM, Grossman JH, Hwang MD: The Innovator’s Prescription: A
Disruptive Solution for Health Care New York: McGraw Hill; 2009.

23. Hwang MD, Christensen CM: Disruptive innovation in health care
delivery: A framework for business-model innovation. Health Affair 2008,
27(5):1329-1335.

24. Bhaskar R: Reclaiming Reality: A Critical Introduction in Contemporary
Philosophy London: Verso; 1989.

25. Health Service Journal. [http://www.hsj.co.uk/].
26. EMAP, Health Service Journal. [http://www.emap.com/platforms/awards/

health-service-journal-hsj].
27. Stiles WB: Quality control in qualitative research. Clin Psychol Rev 1993,

13(6):593-618.
28. Sims-Schouten W, Riley SCE, Willig C: Critical realism in discourse analysis:

A presentation of a systematic method of analysis using women’s talk
of motherhood, childcare and female employment as an example. Theor
Psychol 2007, 17(1):101-124.

29. Boyatzis RE: Transforming Qualitative Information: Thematic Analysis and
Code Development Thousand Oaks, CA: Sage; 1998.

30. Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol
2006, 3:77-101.

31. NVivo: Qualitative Data Analysis Software (Computer Programme) QSR
International Pty Ltd; 2008, Version 8.

32. Patton MQ: Qualitative Evaluation and Research Methods. 2 edition. Newbury
Park, CA: Sage; 1990.

33. Barlow J, Bayer S, Curry R: Implementing complex innovations in fluid
multi-stakeholder environments: Experiences of ‘telecare’. Technovation
2006, 26:396-406.

34. Finch TL, Mair FS, May CR: Teledermatology in the UK: Lessons in service
innovation. Brit J Dermatol 2007, 156:521-527.

35. May C, Mort M, Williams T, Mair F, Gask L: Health technology assessment
in its local context: Studies in telehealthcare. Soc Sci Med 2003,
57:697-710.

36. Porter TM: Trust in Numbers: The Pursuit of Objectivity in Science and Public
Life Princeton, NJ: Princeton University Press; 1995.

37. Trinder L: Introduction: The context of evidence-based practice. In
Evidence-Based Practice: A Critical Appraisal. Edited by: Trinder L, Reynolds S.
Oxford: Blackwell Science Ltd; 2000:1-16.

38. Krein SL, Damschroder LJ, Kowalski CP, Forman J, Hofer TP, Saint S: The
influence of organizational context on quality improvement and patient
safety efforts in infection prevention: A multi-center qualitative study.
Soc Sci Med 2010, 71:1692-1701.

39. Hendy J, Barlow J: The role of the organizational champion in achieving
health system change. Soc Sci Med .

Pre-publication history
The pre-publication history for this paper can be accessed here:
http://www.biomedcentral.com/1472-6963/11/342/prepub

doi:10.1186/1472-6963-11-342
Cite this article as: Barnett et al.: Understanding innovators’ experiences
of barriers and facilitators in implementation and diffusion of healthcare
service innovations: a qualitative study. BMC Health Services Research 2011
11:342.

Barnett et al. BMC Health Services Research 2011, 11:342
http://www.biomedcentral.com/1472-6963/11/342

Page 12 of 12

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under the CCAL, authors

retain copyright to the article but users are allowed to download, reprint, distribute and /or copy articles in

BioMed Central journals, as long as the original work is properly cited.

Summative Evaluation

In: The SAGE Encyclopedia of Educational Research,

Measurement, and Evaluation

By: Anthony Jason Plotner

Edited by: Bruce B. Frey

Book Title: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation

Chapter Title: “Summative Evaluation”

Pub. Date: 2018

Access Date: June 19, 2022

Publishing Company: SAGE Publications, Inc.

City: Thousand Oaks,

Print ISBN: 9781506326153

Online ISBN: 9781506326139

DOI: https://dx.doi.org/10.4135/9781506326139

Print pages: 1636-1637

© 2018 SAGE Publications, Inc. All Rights Reserved.

This PDF has been generated from SAGE Research Methods. Please note that the pagination of the

online version will vary from the pagination of the print book.

An evaluation is a systematic and purposeful collection and analysis of data used to document the

effectiveness of programs or interventions. Rigorous evaluation can determine if programs or interventions

should be maintained, improved, or eliminated. The term summative evaluation (sometimes referred to as ex-

post evaluation or outcome evaluation) was first introduced in the mid-1960s by Lee Cronbach and Michael

Scriven and refers to a process of evaluating a program’s or intervention’s impact or efficacy through careful

examination of program design and management. It is often used to assess the accountability of a program

or intervention. As such, summative evaluation is outcome focused more than process focused and most

often undertaken at the end of the project, when the program or intervention is stable and/or when program

services are implemented with consistency (otherwise known as fidelity). Furthermore, there are some types

of summative evaluation that require the collection of baseline data in order to provide a before and after

understanding; thus, it is important to factor this into the evaluation. Summative evaluation is undertaken to

determine whether the program or intervention achieved its goals, objectives, or outcomes; how the program’s

impact compares to different programs; and to better understand the process of change, what works, what

doesn’t, and why.

Understanding Summative Evaluation

Summative evaluation is also often conducted or undertaken by people considered independent or external

of the responsible project. The methods used to gather the data used in a summative evaluation should

incorporate a detailed step-by-step procedure that is carefully designed and executed to ensure the data are

accurate and valid. A balance of both quantitative and qualitative methods can help researchers obtain a

better understanding of project achievements and information that led to these achievements. The various

instruments or tools used to collect data when conducting a summative evaluation include interviews,

questionnaires, surveys, observations, and testing.

Summative evaluations are conducted to determine the value of a program or intervention—its merit or worth,

often in comparison with other programs or interventions. Summative evaluation can enable stakeholders

to make decisions regarding specific services and the future direction of the program that cannot be made

during the beginning or middle of program or intervention implementation. By contrast, formative evaluation

(also known as process or implementation evaluation) is designed to form or improve the program or

intervention being evaluated by examining aspects of an ongoing program in order to make improvements

as the program is being implemented. Most evaluations can be summative (i.e., have the potential to serve

a summative function), but only some have the additional capability to serving formative functions. One way

to truly understand summative evaluation is to differentiate between formative and summative evaluation. It

is considered good evaluation practice to include both formative and summative evaluation. Table 1 shows

some fundamental differences between formative and summative evaluation.

Common Types of Summative Evaluation

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 2 of 4 Summative Evaluation

There are a variety of types of summative evaluations. Some of these types include cost-benefit/cost-

effectiveness analysis, goal-based evaluation, outcome evaluation, secondary analysis, meta-analysis, and

impact evaluations. Cost-effectiveness and cost-benefit analysis address questions of efficiency by

standardizing outcomes in terms of their dollar costs and values. Goal-based evaluation determines if the

intended goals of a program or intervention were achieved. Outcome evaluation investigates whether the

program caused demonstrable effects on specifically defined target outcomes. Secondary analysis examines

existing data to address new questions or use methods not previously employed. Meta-analysis integrates

the outcome estimates from multiple studies to arrive at an overall or summary judgment on an evaluation

question. Impact evaluation is broader and assesses the overall or net effects—intended or unintended—of

the program or intervention.

Table 1 Fundamental Differences Between Formative and Summative Evaluation

Formative Evaluation Summative Evaluation

Why? Purpose

Analyze strengths and weaknesses

Shape direction

Feedback

Improve a program or intervention

Goal achievement

Unintended consequences

How to improve

Evidence

Determine value or quality

When? Context

Project implementation

Primarily prospective

Project implementation

Postproject

Primarily retrospective

What? Information

Needs assessment

Process

Implementation

Acceptability

Efficacy

Impact

Outcomes

Results

Who? Evaluators Primary internal supported by external evaluators Primary external supported by internal evaluators

See also Evaluation; Formative Evaluation; Program Evaluation; Summative Assessment

Website

Web Center for Social Research Methods: www.socialresearchmethods.net

Anthony Jason Plotner

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 3 of 4 Summative Evaluation

http://dx.doi.org/10.4135/9781506326139.n676

10.4135/9781506326139.n676

Further Readings

Coryn, C. L. S., & Scriven, M. (Eds.). (2008). Reforming the Evaluation of Research: New Directions for

Evaluation, Number 118. San Francisco, CA: Jossey-Bass.

Coryn, C. L. S., & Westine, C. D. (Eds.). (2015). Contemporary trends in evaluation research. Sage

Benchmarks in Social Research Methods (Vols. 1–4). London, UK: Sage.

Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagne, & M. Scriven (Eds.),

Perspectives of curriculum evaluation (pp. 39–83). Chicago, IL: Rand McNally.

Scriven, M. (1991). Beyond formative and summative evaluation. In M. W. McLaughlin & D. D. Phillips

(Eds.), Evaluation and education: At quarter century (pp. 19–64). Chicago, IL: University of Chicago Press.

Wholey, J. S. (1994). Assessing the feasibility and likely usefulness of evaluation. In J. S. Wholey, H. P.

Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 15–39). San Francisco, CA:

Jossey-Bass.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 4 of 4 Summative Evaluation

Evaluation

In: The SAGE Encyclopedia of Educational Research,

Measurement, and Evaluation

By: Dominica McBride

Edited by: Bruce B. Frey

Book Title: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation

Chapter Title: “Evaluation”

Pub. Date: 2018

Access Date: June 19, 2022

Publishing Company: SAGE Publications, Inc.

City: Thousand Oaks,

Print ISBN: 9781506326153

Online ISBN: 9781506326139

DOI: https://dx.doi.org/10.4135/9781506326139

Print page: 624

© 2018 SAGE Publications, Inc. All Rights Reserved.

This PDF has been generated from SAGE Research Methods. Please note that the pagination of the

online version will vary from the pagination of the print book.

Evaluation is a process, discipline, and, in some cases, an intervention in and of itself. It entails the systematic

application of social science research to plan for and learn about the impact of policy, performance, programs,

or initiatives in order to create, further, or sustain social change. The policies, performances, and initiatives

being evaluated are called evaluands. Evaluation is performed in sociopolitical environments and political

influences, and their implications must be considered throughout the process.

From struggles to provide quality education and public health, to environmental dilemmas, societies across

the world face issues that often require planning, policy, and subsequent action to address. Unfortunately,

strategies often do not obtain the desired effect because projects are not implemented as planned, policies

are disconnected from the communities they are supposed to benefit, or programs are not well planned.

According to evaluation expert Michael Scriven, evaluation examines the merit and worth of the evaluand.

However, the examination is often not the end but the means to making change through contributing to a

decision or using the results for advocacy purposes, as in the transformative paradigm, a framework for

evaluation that places importance on groups that have been marginalized. For example, the purpose of an

evaluation of a school district’s new program for reading by third grade would be to assess how effective the

program is for all students, especially disadvantaged students in the district. Using this example, the process

of evaluation would include:

• a.

identifying and engaging stakeholders, or people who have different stakes in the process, such as

students, teachers, parents, school administrators, and communities.

• b.

constructing relevant and answerable questions, such as “To what extent did students enhance their

reading skills?” “What worked and did not work? For whom? Why?” “What are the most pressing

needs for low-income K–3rd grade students in the district?” and “What community assets can be

used to address those needs? How?”

• c.

choosing data collection methods, such as tracking reading grades, interviewing students, surveying

teachers, and holding focus groups with parents.

• d.

collecting and analyzing the data. The data and subsequent analysis used to answer questions such

as these are, ideally, used to make changes necessary to effectively address the problem of focus,

such as reading by third grade.

• e.

synthesizing and disseminating the results.

• f.

taking appropriate action to help those results make a difference in the evaluand and ultimately for

the intended community.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 2 of 3 Evaluation

Given the complexity, human dynamics, and sociopolitical nature of evaluation, there are various skills

needed to successfully complete an evaluation. Skills in areas such as analysis, social skills, project

management, critical self-reflection, negotiation, and advocacy allow evaluators to execute the technical

components and navigate the social world of evaluation to make social change.

See also American Evaluation Association; Culturally Responsive Evaluation; Data-Driven Decision Making;

Goals and Objectives; Outcomes; Program Evaluation; Transformative Paradigm

Dominica McBride

http://dx.doi.org/10.4135/9781506326139.n240

10.4135/9781506326139.n240

Further Readings

Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand

Oaks, CA: Sage.

Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A comprehensive guide.

New York, NY: Guilford Press.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A systematic approach (7th ed.). Thousand

Oaks, CA: Sage.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 3 of 3 Evaluation

Formative Evaluation

In: The SAGE Encyclopedia of Educational Research,

Measurement, and Evaluation

By: Theodore J. Christ & Jessie Kember

Edited by: Bruce B. Frey

Book Title: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation

Chapter Title: “Formative Evaluation”

Pub. Date: 2018

Access Date: June 19, 2022

Publishing Company: SAGE Publications, Inc.

City: Thousand Oaks,

Print ISBN: 9781506326153

Online ISBN: 9781506326139

DOI: https://dx.doi.org/10.4135/9781506326139

Print pages: 697-699

© 2018 SAGE Publications, Inc. All Rights Reserved.

This PDF has been generated from SAGE Research Methods. Please note that the pagination of the

online version will vary from the pagination of the print book.

Evaluation is the process of examining a program, procedure, or product to estimate its function, effect,

and worth. There are two main functions of program evaluation in education. The first is to inform the

development and implementation of the program. The second is to estimate the outcomes and program

effects. Formative evaluation is the use of data before and/or during instruction or the implementation of

an intervention. These data are specifically used to improve and inform curriculum planning, instructional

design, and learning. The goal of formative evaluation is to meet the specific needs of students by identifying

those objectives that have and have not been mastered by the student and determining what needs to

be taught, individualizing educational programs for all students. Most importantly, formative evaluation is a

cyclical process that includes planning, managing, delivering, and evaluating instruction, learning, programs,

and interventions.

Formative evaluation allows for ongoing, real-time adaptations and modifications to aid in the development

of empirically developed and empirically informed instruction or intervention practices. While formative

evaluation aims to ensure that specific goals and objectives are being met, it also allows for improvements

to be made. Formative evaluation can involve the use of both quantitative and qualitative data. For example,

formative evaluation can rely on student performance scores on assessments or tests, and it can also rely

on students’ perceptions about an intervention that has been implemented. This entry further expands on

the definition of formative evaluation before detailing its use in education. Methods of formative evaluation

are then reviewed, followed by an examination of effective strategies and themes, and advantages and

disadvantages of formative evaluation. The entry concludes with an example of instructional design.

Defining Formative Evaluation

There has been a clear and fundamental distinction between formative and summative evaluation since

the 1960s. Summative evaluation specifically refers to evaluation completed at the end or summation of

instruction, intervention, or program activities. In contrast, formative evaluation is intended to develop and

improve a process, activity, or product in an ongoing manner, while the process, activity, or product is

active. Formative evaluation and summative evaluation differ with regard to the goals and intended use of

information. For example, summative evaluation provides information about the degree to which terminal

outcomes have been successfully attained over the course of a class, activity, program, or intervention. In

contrast, formative evaluation provides information about needs and progress during the time a program

is implemented. The underlying purpose and expected uses of the information differ. Formative evaluation

answers whether it is working; summative evaluation answers whether it worked. Finally, formative evaluation

is not the same as formative assessment. Although all formative assessment is formative evaluation, not all

formative evaluation is formative assessment. Assessment refers to the process of measuring information

about a student or program to yield a source of information. Evaluation is the process of using the information

that has been collected to make informed decisions. Put simply, assessment is the collection of information,

while evaluation is the use of that information.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 2 of 6 Formative Evaluation

Uses of Formative Evaluation in Education

There are many potential uses of formative evaluation in an educational setting. For example, formative

evaluation can serve as a needs assessment, examining whether a program or intervention is addressing

a specific goal or objective. Formative evaluation specifically involves the use of data to identify individual

student needs. These data are then used to plan, inform, and improve academic instruction. With increased

attention toward accountability of improving outcomes for all students, the need for linking assessment to

intervention practices is irrefutable. Formative evaluation can also be used to modify instruction, a program,

or an intervention. Applying appropriate modifications during these processes allows instructors to increase

the likelihood of success. Even further, formative evaluation may be used to determine the extent to which an

intervention or program is implemented with fidelity and whether it has been implemented with consistency

and quality. Thus, in some instances, formative evaluation can serve as a means for quality control. Finally,

formative evaluation may be used to document progress on an ongoing basis in a standardized fashion,

complementing summative evaluation methods. Although formative evaluation and summative evaluation can

be successful independently, each is supplementary to the other and is more successful when used alongside

the other. A comprehensive evaluation likely includes both summative and formative practices. When used

in conjunction, these evaluation components can examine how an intervention or program was implemented,

factors that both constrained and facilitated success and effectiveness.

Methods of Formative Evaluation

Formative evaluation may include either qualitative or quantitative data. Because formative evaluation may

refer to the evaluation of activities, programs, curricula, or interventions, methods of formative evaluation are

wide and varied and may include various assessment techniques (e.g., midsemester evaluations, curriculum-

based measurement in reading), self-evaluation or self-assessments, surveys (i.e., open and close-ended

questions), focus groups or expert review, or observation techniques. Methods employed will largely be

determined by the purpose of the evaluation and the questions of interest.

Effective Strategies and Themes

For formative evaluation to be effective, there are several recommended strategies and underlying themes.

The first of these strategies is defining a specific purpose of the evaluation. The purpose should be relevant.

Even further, decisions about how the data will be used should be specified before the data are collected.

In the realm of school psychology, there are four general types of decisions that can be made regarding

individual student performance: screening, progress monitoring, analytic, and outcome. Formative evaluation

is used primarily for progress-monitoring decisions. Progress-monitoring decisions refer to those decisions

made to determine whether or not a student’s rate of progress is adequate. While this particular application

is narrow, it can be generalized to the evaluation of activities, programs, curricula, and interventions.

Regardless, there needs to be a clear and explicitly defined purpose for the program, intervention, or

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 3 of 6 Formative Evaluation

curriculum.

The second of these strategies is visual analysis. By evaluating the implementation of an intervention, data

can be visually analyzed at multiple time points throughout the intervention to determine the effectiveness.

There are four visual analysis criteria: change in mean, change in level, change in trend, and latency

of change. Change in mean refers to the extent to which the average rate of performance during the

intervention differs from the average rate of performance before the intervention. Change in level refers to

the extent to which there is discontinuity of performance when comparing baseline data to data collected

during the intervention. Change in trend refers to whether performance is increasing or decreasing throughout

the intervention. Finally, latency of change refers to the amount of time that occurs before a change in

performance is observed after implementing the intervention.

Visual analysis of data can be used to formatively evaluate student progress and to determine whether the

predefined goals will be met. For example, if the student’s performance trend is flatter than the goal line,

a decision might be made to change the implemented intervention. If the student’s performance trend is

equivalent to the goal line, a decision might be made to continue the intervention. Finally, if the student’s

performance trend is greater than the goal line, a decision might be made to increase the goal.

Third, effective formative evaluation rests on our ability to test hypotheses about instruction, learning,

programs, or interventions. This iterative process involves examining student performance frequently,

routinely, and in an ongoing manner. Evaluation is essentially founded on accurate inferences or logical

conclusions derived from a given body of evidence.

Finally, formative evaluation is a dynamic process. Generally, the more frequently educators collect data,

the better. While summative evaluation provides a static determination, decision, or diagnosis, formative

evaluation provides a responsive, data-based problem-solving strategy. Information is specifically collected

for the purpose of making decisions about instruction, learning, programs, or interventions. Formative

evaluation allows for individualized educational programs based on student performance. This inductive

and systematic approach to developing instruction and intervention allows educators to adjust the intensity,

frequency, and content. Formative evaluation relies on follow through. A process or practice is formative to

the extent that evidence about student performance is elicited, documented, interpreted, and used whether it

is used by the teacher, peers, or the learner.

Advantages and Disadvantages

Formative evaluation encourages ongoing, data-based decision making for the purpose of improving

practices, processes, plans, and programs. The information obtained throughout this dynamic process

increases the likelihood of success and allows for efficient resource allocation. In addition, formative

evaluation provides educators with a strategy for refining practices and programs that takes a preventive

approach because it takes place during the formation stage.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 4 of 6 Formative Evaluation

However, formative evaluation does come at a cost. Formative evaluation requires time and resources.

Also, although formative evaluation increases the likelihood of intervention or program success, it also has

the ability to distort our impressions of intervention or program effectiveness. More specifically, formative

evaluation itself may be considered an intervention. Thus, it can be difficult to evaluate the independent

impact of instruction, learning, a program, or an intervention. Finally, in some instances, formative evaluation

may require making decisions and modifications with seemingly little evidence.

An Instructional Design Example

In regard to instruction, formative evaluation requires planning, managing, delivering, and evaluating. In

planning instruction, one needs to assess the baseline skill level of students before instruction occurs or

after preliminary instruction. Screening to collect student performance data can provide educators with an

assessment of the instructional environment. Next, managing instruction involves adjusting the instructional

level for individual students based on one’s assessment of the classroom environment. This includes

identifying concepts and skills that need to be taught to certain groups of students. Effectively delivering

instruction relies on continuous assessment of student mastery of the material and immediate and explicit

feedback. Finally, in evaluating instruction, instructors assess student learning and set goals for future

instruction.

See also Curriculum-Based Assessment; Curriculum-Based Measurement; Evaluation, History of;

Summative Assessment

Theodore J. Christ & Jessie Kember

http://dx.doi.org/10.4135/9781506326139.n272

10.4135/9781506326139.n272

Further Readings

Bloom, B. S. (1971). Handbook on formative and summative evaluation of student learning. New York, NY:

McGraw-Hill.

Deno, S. L. (1986). Formative evaluation of individual student programs: A new role for school psychologists.

School Psychology Review, 15, 358–374.

Fuchs, L. S., & Fuchs, D. (1986). Effects of systematic formative evaluation: A meta-analysis. Exceptional

Children, 53(3), 199–208.

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic

progress: How much growth can we expect? School Psychology Review, 22, 27–27.

Scriven, M. (1967). The methodology of evaluation. In Perspectives of curriculum evaluation (AERA

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 5 of 6 Formative Evaluation

Monograph series on curriculum evaluation 1). New York, NY: Rand McNally.

Stiggins, R., & Chappuis, S. (2005). Putting testing in perspective: It’s for learning. Principal Leadership,

6(2), 16–20.

SAGE

2018 SAGE Publications, Ltd. All Rights Reserved.

SAGE Research Methods

Page 6 of 6 Formative Evaluation

Evaluation of People, Social, and

Organizational Issues – Sociotechnical

Ethnographic Evaluation

Bonnie KAPLAN
a,1

a
Yale Center for Medical Informatics, Yale University, New Haven, CT, USA

Abstract. Sociotechnical approaches are grounded in theory and evidence-based.

They are useful for evaluations involving health information technologies. This

contribution begins with an overview of sociotechnical theory and ethnography.

These theories concern interactions between technology, its use, people who use or

are affected by it, and their organizational and societal situations. Then the

contribution discusses planning and designing evaluations, including frameworks

and models to focus an evaluation, and methodological considerations for

conducting it. Next, ethical issues and further challenges and opportunities are

taken up. Concluding case examples, referenced throughout, illustrate how good

evaluations provide useful results to help design, implement, and use health

information technologies effectively.

Keywords. Evaluation studies, organizational studies, ethnography, medical

informatics, qualitative research, qualitative evaluation, organizational culture,

organizational case studies.

1 Introduction

Successful implementation involves interactions and mutual adjustments among an

information technology application and the organization, people, and practices where it

is used. Sociotechnical evaluation analyzes this interplay between technologies and

social and technical systems. It emphasizes how people, organizations, professions,

culture, work practice, ethical issues, social and political environment, and the like, all

interact and change each other over time. Sociotechnical analyses assess how

information technology and workflow influence each other; how clinical and patient

roles relate to technological use; how useful and usable health information technologies

are; and what consequences, patient safety issues, or user responses might occur. They

involve considering these interdependent elements as a holistic dynamic network rather

than as fixed pre-defined separate domains [1,2,3,4].

For example, Example 1 indicates that using images, and incorporating clinical

images into on-line electronic patient records, depends not only on the computer

system, but also on interwoven issues of expertise, trust and relationships among

colleagues, clinical knowledge of individual patients, institutional priorities, how

1
Corresponding author: Prof. Dr. Bonnie Kaplan, Yale Interdisciplinary Center for Bioethics, Box

208293, 238 Prospect Street, New Haven, CT 06511, United States, [email protected]

Evidence-Based Health Informatics
E. Ammenwerth and M. Rigby (Eds.)
© 2016 The authors and IOS Press.

This article is published online with Open Access by IOS Press and distributed under the terms
of the Creative Commons Attribution Non-Commercial License.

doi:10.3233/978-1-61499-635-4-114

114

conveniently system access fits into a busy and frequently interrupted day full of ad

hoc conversations, ways images are interpreted and their clinical meanings negotiated,

and other socio parts of sociotechnic. That is how sociotechnical systems work, and

how sociotechnical analyses can be helpful.

2 Sociotechnical Theory

Sociotechnical approaches incorporate theories and evidence from multiple disciplines.

Key theoretical features include examining technologies as they actually are used in

natural settings to investigate how technical and physical work settings affect their use;

how users negotiate, re-negotiate, interpret, and re-interpret features of the technology;

and relationships among the social and technical components of these emergent

processes as they unfold over time. The approaches are based on an understanding that

a new information technology and the social system where it is introduced change each

other as different parties pursue different goals [4,5]. These approaches are not

deterministic, nor do they understand technological development in terms of a rational,

linear sequence. Instead, they emphasize evolving processes and interactions so that no

factor acts in isolation from others, or has a uni-directional impact. They see processes

and causes interacting in multiple causal directions and relationships.

Sociotechnical principles developed as part of the Tavistock Institute’s post-WW

II analysis of British industries. They emphasized designing work for workers’ interests

and quality of working life [4]. By the 1990s, sociotechnical ideas had been introduced

into health informatics, as were social interactionist approaches – approaches that

consider relationships between system, individual, and organizational characteristics

and effects among them – which now would be labeled “sociotechnical” [1,2,3,6,7,8,9].

Sociotechnical theory in health informatics, then, has roots in traditional sociotechnical

research, ergonomics, social construction of technology, technology-in-practice, and

social informatics [5]. To these antecedents, I would add theories of change.

Informatics systems introduce change which may be welcome, or disruptive, to

the individual and the organization. Sociotechnical theory conceptualizes

organizational change as interacting components – for example, Leavitt’s well-known

diamond model of people, task, technology, and structure [10] – each responding to a

+change in any other so as to maintain organizational homeostasis, with the

interactions themselves being most important. Other theories of change based on the

foundational work of Rogers [11] and Lewin [12] characterize it as a dynamic process

that proceeds through stages involving multiple actors with different concerns and

perceptions of benefit. These actors include experts, sponsors, and people adopting (or

not adopting) the change. These actors are connected and communicate through various

social, organizational, social, and cultural channels. The change occurs, then, at

individual, group, organizational, and cultural levels. Any of the stages, actors, system

components, and units of analysis could be the focus of evaluation.

3 Ethnography

Ethnographic approaches explore how users experience health information technology

and why they interact with it as they do. They involve getting to know and

documenting the people and culture by spending time and participating in the setting

B. Kaplan / Evaluation of People, Social, and Organizational Issues 115

under study [13]. Ethnography expresses findings in terms meaningful to the people

involved. This enables people to recognize themselves and thereby makes those

findings more convincing and relevant. Ethnographic sociotechnical evaluation can

help prevent difficulties through better needs analysis, system design and

implementation practices, understanding what people do when working with the

technologies, and identifying why they view and use the technologies in those ways.

Ethnography involves starting with a sense of what to investigate and

progressively sharpening the investigation as more is learned. This is different from

beginning with immutable testable hypotheses, a priori research questions or

categories, and a pre-set research design. Instead, the study evolves and changes

according to what is learned as it proceeds. Because sociotechnical systems are

dynamic, freezing a research design before beginning may turn out to poorly match the

situation at hand as it develops. Ethnographic methods are particularly valuable in

natural, uncontrolled settings. They allow for adjusting a study in a fluid environment

where unanticipated findings emerge and situations change.

Methodologically, Examples 1 and 2 are ethnographic. Ethnographies tend to

emphasize the people involved and explore their situations. The main general

investigative questions are:

(1) What is happening here?
(2) Why is it happening?
(3) How has it come to happen in this way?
(4) What do the people involved think is happening?
(5) How are they responding to what is happening?
(6) Why are they responding that way? [14]

The key question is “Why?”: Why are the people who are involved actually

involved; why do they think and react as they do; why do they use the technology as

they do; why are they interacting as they are; what meanings do they attribute to the

technology, health and disease, their roles, and what they do; and why those meanings?

To answer, ethnographic work uses open-ended evaluation questions, qualitative

data collection and analysis, interpretive and multi-level data analysis, a focus on the

lived experience and its meaning to those involved, emergent findings, and making

tacit knowledge and practice manifest. Because it enables a deep understanding of what

is going on, wiser decisions and actions may be based on those findings, and theoretical

insights may be developed [14].

In Example 2, sociotechnical approaches revealed emergent, unexpected findings

involving more general interrelationships between work and technology use. The

analysis reinforces the sociotechnical stance that the technology does not stand alone,

the social system (in this case, laboratory management, laboratory work, and hiring

practices) does not stand alone, but the two mutually affect and change each other. The

ethnographic approach enabled better understanding of how laboratory work was

understood. This resulted from resolving seemingly divergent findings from multiple

sources of data through an interpretation that accounted for all data, in this case, the

job-orientation model that relates how people think of their job to how they think about

computer systems introduced into their work. This rich result contributed to theory.

B. Kaplan / Evaluation of People, Social, and Organizational Issues116

4 Theory Development

Example 2 also exemplifies other theoretical points. The evaluation contributed to the

idea that “the same” system is not the same for all concerned, which also was found in

an evaluation of an automated telephone counseling system [15]. Similarly, “success”

may be defined and experienced differently among different groups and individuals at

different times [16]. Further, as also evident in studies that contributed to the idea of

the importance of fit between a technology and an organization, “fit” has to be

produced actively and changes over time [3,17].

The two examples contributed to another theoretical insight as well. The findings

inspired a framework helpful in future studies: the 4Cs of communication, care (or, if

outside of clinical institutions, whatever else the mission of the organization is),

control, and context [2,8,18]. In Example 1, on-line images improved communication

and care, raised control issues, and occurred in the different contexts of a government

and academic medical center. The laboratory information system in Example 2 also

improved communication and, therefore, care; highlighted control issues; and took

account of the context of different laboratories and technologists in the job-orientation

model. Frameworks like 4Cs can be useful for evaluation planning and design.

5 Planning and Designing Ethnographic Sociotechnical Evaluation

The multiplicity of interacting systems and sub-systems presents a wide range of

choice for how to design an evaluation. To choose among the possibilities, decisions

are needed concerning how to focus an evaluation, when to evaluate, and how to

evaluate.

5.1 What to Evaluate

To answer the key evaluation question of what is happening and why, it is hard to

know at the outset what of all the activity and who of all those involved will be

important. Theories, models, and frameworks can help to target what is most relevant

for the situation at hand. They provide a lens through which situations can be analyzed

and understood; highlight what is important; explain how various factors, influences,

and considerations interrelate; help organize and explain findings; and lead to

predictions for further investigation and planning. Their power comes from

emphasizing only some aspects of the area under study. Because each necessarily

leaves out aspects that may turn out to be important, it can be helpful to use more than

one theory, model, or framework. Sociotechnical evaluation lends itself to just that.

4Cs, discussed in Section 4, brings attention to issues of communication, care, control,

and context. Sitting and Singh’s model focuses on hardware and software; clinical

content; human-computer interface; people; workflow and communication;

organizational policies, procedures, and culture; external rules, regulations, and

pressures; and system measurement and monitoring [19]. An additional set of

evaluation questions, based on those of Anderson and Aydin [7], could be:

(1) Does the system work as designed?
(2) Is it used as anticipated?
(3) Does it produce the desired results?

B. Kaplan / Evaluation of People, Social, and Organizational Issues 117

(4) Does it work better than what it replaced?
(5) Is it cost-effective?
(6) How well have individuals been trained to use it?
(7) What are changes in departmental interaction, delivery of care, patient safety,

control and power in the organization, or the healthcare system at large?

(8) How do the system and these changes relate to the practice setting?

Combining theories, models, or frameworks can help an evaluator choose potential

evaluation questions. What purpose the evaluation serves also is important when

choosing a focus. Table 1 gives some examples.

Table 1. How evaluation purpose can affect evaluation focus.

If the purpose of the evaluation is The evaluation could focus on

• Technical • System requirements
• Economic • Cost/benefit
• Clinical • Patient Care
• Education • Students’ grades, learning outcomes
• Research • Access to literature, data
• Policy • Cost, utilization
• Usefulness • User satisfaction, degree of use

Just as the system, the users, and the context interact and shape each other, the

evaluation context and environment affect how the study is conducted over time. These

include:

(1) purpose of the system, which may be for research and development, a
demonstration project, or a commercial product;

(2) organizational commitment, which might be to continue, maintain, or quash
the system, or to evaluate it;

(3) who the client is;
(4) how evaluation results will be used;
(5) budget and time frame;
(6) evaluator skills and expertise;
(7) who the research subjects are; and
(8) the people who are involved.

Considerations about these people include:

(7) how the need for the system and for the evaluation was determined, and by
whom;

(8) what needs the system and the evaluation meet, and whose needs they are;
(9) who will be using the system, doing data entry, or receiving outputs;
(10) what users’ attitudes towards the system, and towards the evaluation, are;
(11) who was involved in needs assessment, design, and testing, and why those

where the people involved;

(12) whether potential users perceive a need for the system;
(13) whose interests the system or the evaluation serves, or appears to others to

serve; and

(14) what different parties want to know.

B. Kaplan / Evaluation of People, Social, and Organizational Issues118

Knowing the environment and people involved can alert the evaluator to

considerations that should be examined further.

5.2 When to Evaluate

Sociotechnical ethnographic evaluations can be done at any stage, or multiple stages, of

system development or implementation. When to evaluate depends on the purpose of

the evaluation, as in the two examples. There is no need for concern that study results

or even conducting the study will affect the object of study. It will. A moving target is

assumed. Evaluation, then, can be used to influence needs assessment, analysis, design,

implementation, and how a system is used without “tainting” either the process or the

rigor of the study. In fact, it is wise to feed what is learned back into the process so that

it proceeds more smoothly.

5.3 How to Evaluate

Choosing methods depends on evaluation questions, evaluator skills and expertise, and

budget and time table. The theoretical underpinnings of sociotechnical approaches

suggest methods and research designs that are flexible and encourage emergent,

unexpected findings. Rather than the usual impact studies that characterize much

medical research – randomized controlled trials and experimental designs to test

hypotheses – interactionist (i.e. where subsystems and system components interact over

time) sociotechnical study designs are preferred. Table 2 indicates some ways impact

and interactionist studies differ.

Table 2. Differences between impact studies and interactionist studies.

Impact Interactionist

Epistemology Objectivist Objectivist or Subjectivist

Purpose Factors Process

Variance

Methods Quantitative Qualitative

Causality Uni-directional Multi-directional

Question What Why

Sociotechnic approaches examine how peoples’ practices are situated in their

environments and how the actors and technological change interact. These studies are

best done in situ using methods appropriate to naturalistic settings and changing

circumstances. Ethnographic sociotechnical evaluation is interactive not only in

examining interactions among the social and technical components of the system under

study, but also among components of research design. What should be studied and

what the research questions are depends on the purposes, methods, conceptual

concepts, and validity issues involved, and each of these shapes the others [20]. Study

design, then, should be longitudinal, modifiable, and flexible over time. Because

evaluation can help direct a project, it can be both formative and summative, and

should focus on a variety of concerns reflecting the various actors involved. Employing

multiple methods is beneficial because different data sources provide different data

[18]. Different informants may have different focuses, report processes that are

different from what the evaluator observes, and behave differently from the way they

indicate on surveys or in laboratory settings. The challenge is to make sense of these

B. Kaplan / Evaluation of People, Social, and Organizational Issues 119

differences. If data do not converge, a richer understanding develops through

accounting for apparent contradictions, as in the laboratory information system study

(Example 2). Multiple methods and data sources lead to robust results.

Qualitative methods were used in the two examples. They are especially valuable

for sociotechnical ethnographic evaluation. Data collection methods include participant

observation; observation; unstructured or semi-structured interviews; focus groups;

surveys with open-ended questions; analysis of artifacts like documents, images, texts,

or drawings; and the researcher’s own impressions and reactions. Analysis methods

include coding, contextual or narrative analysis, analytic memos, and displays. Data

analysis involves constantly integrating and analyzing voluminous, mostly textual, data

from multiple sources. Interpretations and hypotheses continually are formulated,

tested, and verified or discarded through a process of on-going data analysis and

writing that assesses whether they make sense in light of existing and future data. What

seems most interesting, relevant, or important progressively becomes clearer [14].

Qualitative data analysis software is a boon to managing and analyzing the

volumes of data an evaluation study produces, but it does not do the analysis per se.

The evaluator still needs to figure out how to interpret data. It helps in this process to

focus on:

(1) how people use words and what they mean by them—what is meant by
“work” in the laboratory (Example 2) or “see[ing] what’s really going on” in

an image (Example 1);

(2) what people say and do, and under which circumstances they say and do it—
how the clinicians in the second imaging study (Example 2) negotiated what

images meant;

(3) how people justify or give reasons for what they say, do, believe, etc.—
comments laboratory technologists wrote about why the new system was a

“hassle” or improved reporting (Example 2);

(4) what does not seem to make sense (the puzzles)—how a laboratory
technologist’s job does not change when the technologists’ tasks change

(Example 2); and

(5) how to make sense of all the data.

Focusing this way helps produce evaluations that get at what it means to the people

involved to use health information technologies. Paying close attention to who the

people are, what they think, what they do in real-life settings, and how they differ,

helps explain how all that interacts with health information technology development,

implementation, adoption, and use – in other words, how the social and technical

subsystems interact. The end result, then, goes well beyond simply reporting data. It

requires solving puzzles by accounting for all data in a way that focuses on what the

technology means to the participants, why it means that, and what the implications are.

Explaining the data in this way helps make tacit knowledge, assumptions, meanings,

and values explicit, so they can be taken into account. It tells a coherent, compelling

story that is useful, and makes theoretical contributions by both drawing on theory to

produce an interpretation and also, as in the examples, possibly develop new theory.

5.4 How to Validate Evaluation Results

Qualitative researchers collect rich data and produce intepretations that account for it

all through a process known as triangulation. Particular attempts are made to collect

B. Kaplan / Evaluation of People, Social, and Organizational Issues120

data that may contradict the developing interpretation. Data is continually collected

until no new information seems to be possible, which is known as reaching saturation.

The people involved in the study are asked for feedback and for their responses to the

developing interpretation in a process known as member-checking, and what they say

becomes new data [14]. A neutral partner can review data and how it is interpreted.

Similarly, research team members can test each other’s ideas, methods, and

interpretations. Eventually, reviewers and other researchers judge the work, just as in

any other form of research. Reproducability is impossible; every situation, evaluator,

and study is different. The goal is transferability, so that significant insights can be

developed, theoretical contributions can be made, and the knowledge gained can be

applied elsewhere.

6 Sociotechnical Ethnographic Evaluation Research Ethics

Evaluators face ethical decisions even before beginning an evaluation and thereafter. In

addition to usual research ethics issues, additional concerns arise in sociotechnical

ethnographic evaluation. A few of them are mentioned here. Special considerations

involve informed consent, privacy and confidentiality, social justice, practitioner

research, power, reciprocity, relevance, and how the research is used [21].

As in other fieldwork, interpersonal relationships develop between evaluators and

participants, raising questions of just what those relationships should be. The evaluator

may be privy to material that those involved did not give consent for or see people who

were not asked for permission. When a new technology is introduced it is hard to

anticipate how people will react, making consent even more problematic [22]. The

evaluator may observe what could be unethical behavior, or be asked to engage in

behavior that some may consider unethical. A sociotechnical viewpoint involves

sensitivity to ethical questions like who defines, and should define, the evaluation

questions, interpretation, and use of results, and whose interests are served by the

evaluation. The evaluation, too, likely will involve the goals, values, and assumptions

incorporated into the technology, how it is implemented, how people are expected to

use it, and effects expected from it, also raising ethical concerns.

7 Future Challenges and Opportunities

To date, evaluation mostly concerns visible, tangible health information technologies in

physical settings. Newer developments—virtual health care delivery, distributed

integrated health care organizations, virtual workers, fluid organizational boundaries,

social networks, telehealth and mobile health applications, avatars and artificial

intelligence—make in situ studies more difficult, especially if health care delivery

crosses jurisdictional boundaries. Adding to the complexity is the need for multi-site

studies that include community, home, or other non-academic locations with

geographic or national variation. There is room for sociotechnical evaluation study

designs and methods that address these challenges while also contributing to much-

needed methods to assess patient outcomes better [3].

B. Kaplan / Evaluation of People, Social, and Organizational Issues 121

8 Conclusion

Sociotechnical ethnographic evaluation focuses primarily on the people in addition to

the technology. Contributors to system “success” are sociotechnical. By focusing on

technologies as they actually are used, in the settings in which they are used, and seeing

how people negotiate and reinterpret the technologies as the social and technical

systems interact with each other, sociotechnical ethnographic evaluation can contribute

to theory and practice while improving health information technologies and patient

care.

Example 1 – Clinical Imaging Systems

Administrators and clinicians differed about the value of a new system that integrated

patient record textual, numeric, and image data [23]. This raised an administrative

control issue concerning decisions about continued development. Also, previously the

department where an image was produced kept the image, but now images were

available to all, which potentially created another control issue.

In a week of interviews and observations, we investigated what clinicians thought

about the benefits of the system. Clinicians told us that having the images available as

part of the on-line patient record improved communication and consultation, so

improved clinical decisions, and hence, patient care. Because “a written report won’t

convey everything,” and “you don’t know [if the report] is an accurate description,”

now, clinicians said, they “know what’s there,” they could “look through a patient and

see,” “see what’s really going on.” That way, they did not need to repeat procedures.

They could plan treatment better and give students “real” experience through these

images.

Elsewhere, I spent a week shadowing a physician as he performed his daily

activities. The purpose was to identify how clinical images are used in an academic

medical center planning to develop a stand-alone imaging system [24]. The physician

was interrupted constantly, frequently telephoned for information, talked about patients

with other doctors he met fortuitously on the stairway, and consulted with Pathology

and Cytology after receiving reports that slides were “not diagnostic,” or “inadequate

for evaluation.” The person reading the slides told the physician that he had a “gut

feeling” that the cells indicated cancer, though “quantitatively it was a little short” and

showed him why. At his weekly radiology conference, they discussed each patient’s

images, asking each other about the patient and what they saw, or thought they saw, on

the image. For the physician I shadowed, mutually viewing images was improving

communication and clinical decision making, and seeing the images was better than

reading a report. However, reading an image was not a matter of “see [ing]what’s really

going on,” but of interpreting the image in light of expertise and experience, clinical

knowledge of that particular patient, and discussing all that.

In these studies, clinicians thought of the benefits of viewing images as a whole,

not as a separate part of patient care. They thought having those images improved care

and decision making. They considered the images objective, talking of them as

showing, all by themselves, what was “really going on.” Yet, the studies indicate that

what an image means and what clinical decisions should be based on it depends on far

more than simply having the image. In these evaluations, the meanings of those images

were being negotiated through collegial interchanges, though neither clinicians nor

B. Kaplan / Evaluation of People, Social, and Organizational Issues122

system developers acknowledged it. Even though the same could be said about paper

and film-based images, health information technologies often are premised on a belief

that providing information alone is enough because it speaks for itself. This belief

affects design, implementation, and use.

Example 2 – Clinical Laboratory System

We investigated the impact of a new system on laboratory work in a longitudinal study

ranging from pre- to post-implementation. More in line with a sociotechnical

ethnographic approach, we also sought to identify what happens when an academic

medical center converts from a manual to automated system for ordering clinical

laboratory tests and reporting test results [25]. The study included interviews,

observations, participant observation, and surveys.

Technologists’ responses to scaled-response survey questions indicated no change

in laboratory work. Nevertheless, it was clear from their comments in open-ended

questions that work was changing. Some technologists reported being “happy” because

of fewer abusive telephone calls. They also liked the more legible, timely, and

complete laboratory reports. Others, instead, reported on the “hassle” of having to

interrupt their work to enter test results into the computer. We realized that the first

group of technologists thought of their job as providing laboratory test results, an

outcome- or product-oriented view of laboratory work. The other group of

technologists thought of their work as doing laboratory tests, a more process-oriented

view in which they saw the new computer system as a “hassle” that took them away

from the laboratory bench. This job-orientation model applied not only to individual

technologists, but also to the fit between system and different laboratories. The same

laboratory information system used in all the laboratories was not “the same” for

everyone, nor even every laboratory. Instead, it was viewed differently in ways that

related to job orientation. Moreover, it was apparent that being able to work with the

computer system was a new criterion for being a laboratory technologist.

The findings can be reported in terms of improving communication between the

laboratories and the clinicians by producing better and more timely laboratory reports,

thereby improving care. Laboratory technologists fielded fewer telephone calls asking

for laboratory results. Control issues arose over laboratory work, and the different

context of each laboratory was related to how technologists viewed the new system. In

particular, how the laboratories, as well as individual technologists and laboratory

directors, saw the nature of laboratory work was key to understanding their reactions.

In interviews, directors had told us that the new system would not change

technologists’ jobs. If they had realized that there were different views of laboratory

work, that laboratory work was now different, and that these differences would matter

in how technologists and laboratories related to the new system, they could have

prepared staff better.

B. Kaplan / Evaluation of People, Social, and Organizational Issues 123

Recommended further readings

1. M. Berg, Patient care information systems and health care work: a sociotechnical
approach, International Journal of Medical Informatics 55 (1999), 87-101.

2. B. Kaplan, N. T. Shaw, Future directions in evaluation research: people,
organizational, and social issues, Methods of Information in Medicine 43 (2004),

215-231.

3. B. Kaplan, J.A. Maxwell, Qualitative research methods for evaluating computer
information systems, in: Evaluating the Organizational Impact of Healthcare

Information Systems, 2nd ed., J.G. Anderson, C.E. Aydin, eds., Springer, New

York, 2005. pp. 30-55.

4. M. H. Harrison, R. Koppel, S. Bar-Lev, Unintended consequences of information
technologies in health care – an interactive sociotechnical analysis, Journal of the

American Medical Informatics Association 14 (2007), 542-549.

5. S. Sawyer, M. Jarrahi, Sociotechnical approaches to the study of information
systems, in: Computing Handbook: Information Systems and Information

Technology, 3
rd

ed., v. 2, H. Topi, A. Tucker, eds., Chapman and Hall/CRC, Boca

Raton, FL, 2014. pp. 5-1 – 5-27.

Food for thought

1. What are the distinguishing features of sociotechnical theory? What advantages
and disadvantages would each feature bring to an evaluation?

2. How might ethnography influence evaluation? What are the pros and cons?
3. What are the benefits and pitfalls of using models, theories, or frameworks to focus

an evaluation?

4. How would you address the challenges you would expect to face in qualitative data
collection and analysis?

5. How would you design a sociotechnical ethnographic evaluation outside an
institutional setting, for example, of a smartphone application for managing a

diabetic teenager’s diet or an elderly person’s depression? What evaluation

questions would you investigate? How would you go about investigating them?

What ethical challenges might arise?

References

[1] M. Berg, Patient care information systems and health care work: a sociotechnical approach,

International Journal of Medical Informatics 55 (1999), 87-101.

[2] B. Kaplan, Evaluating informatics applications – some alternative approaches: theory, social
interactionism, and call for methodological pluralism, International Journal of Medical Informatics 64

(2001), 39-56.

[3] B. Kaplan, N. T. Shaw, Future directions in evaluation research: people, organizational, and social
issues, Methods of Information in Medicine 43 (2004), 215-231.

[4] S. Sawyer, M. Jarrahi, Sociotechnical approaches to the study of information systems, in: Computing
Handbook: Information Systems and Information Technology, 3

rd
ed., v. 2, H. Topi, A. Tucker, eds.,

Chapman and Hall/CRC, Boca Raton, FL, 2014. pp. 5-1 – 5-27.

B. Kaplan / Evaluation of People, Social, and Organizational Issues124

[5] M. H. Harrison, R. Koppel, S. Bar-Lev, Unintended consequences of information technologies in

health care – an interactive sociotechnical analysis, Journal of the American Medical Informatics

Association 14 (2007), 542-549.

[6] M. Berg, C. Langenberg, I.V.D. Berg, J. Kwakkernaat, Considerations for sociotechnical design:
experiences with an electronic patient record in a clinical context, International Journal of Medical

Informatic 52 (1998), 243–251.

[7] J.G. Anderson, C.E. Aydin, Overview. Theoretical perspectives and methodologies for the evaluation
of health care information systems, in: Evaluating the Organizational Impact of Healthcare

Information Systems: 2nd ed., J.G. Anderson, C.E. Aydin, eds., Springer, New York, 2005. pp. 5-29.

[8] B. Kaplan, Social interactionist framework for information systems studies: the 4Cs, in: IFIP WG8.2
and WG 8.6 Joint Working Conference on Information Systems: Current Issues and Future Changes,

T.J. Larson, L. Levine, J.I. DeGross, eds., International Federation for Information Processing,

Helsinki, 1998. pp. 327-339, http://ifipwg82.org/node/26, last access 11 February 2016.

[9] J.G. Anderson, C.E. Aydin, B. Kaplan, An analytical framework for measuring the
effectiveness/impacts of computer-based patient record systems, in: Proceedings: Twenty-Eighth

Hawaii International Conference on Systems Science, J.F. Nunamaker, R.H. Sprague, eds., IEEE

Computer Society Press, Los Alamitos, Cal, 1995. pp. 767-776.

[10] H. J. Leavitt, Applying organizational change in industry: structural, technological and humanistic
approaches, in: Handbook of Organizations, J. G. March, ed., Rand McNally, Chicago, 1965. pp.

1144-1170.

[11] E.M. Rogers, Diffusion of Innovations, 5th ed., Free Press, New York, 2003.
[12] K. Lewin, Group decision and social change, in: Readings in Social Psychology, 3rd ed., E. Maccoby,

T. M. Newcomb, E. L. Hartley, eds., Henry Holt, New York, 1958. pp. 197-211.

[13] S. Reeves, A. Kuper, B.D. Hodges, Qualitative research methodologies: ethnography, BMJ 337
(2008), a1020, 512-514.

[14] B. Kaplan, J.A. Maxwell, Qualitative research methods for evaluating computer information systems,
in: Evaluating the Organizational Impact of Healthcare Information Systems, 2nd ed., J.G. Anderson,

C.E. Aydin, eds., Springer, New York, 2005. pp. 30-55.

[15] B. Kaplan, R. Farzanfar, R.H. Friedman, Personal relationships with an intelligent interactive
telephone health behavior advisor system: a multimethod study using surveys and ethnographic

interviews, International Journal of Medical Informatics 71(1) (2003), 33-41.

[16] M. Berg, Implementing information systems in health care organizations: myths and challenges,
International Journal of Medical Informatics 64 (2001), 143-156.

[17] J. Aarts, H. Doorewaard, M. Berg, Understanding implementation: the case of a computerized
physician order entry system in a large Dutch university medical center, Journal of the American

Medical Informatics Association 11(3) (2004), 207-216.

[18] B. Kaplan, Addressing organizational issues into the evaluation of medical systems, Journal of the
American Medical Informatics Association 4 (1997), 94-101.

[19] D.F. Sittig, H. Singh, A new sociotechnical model for studying health information technology in
complex adaptive healthcare systems, Quality and Safety in Health Care 19 (2010), i68-i74.

[20] J.A. Maxwell, Qualitative Research Design: An Interactive Approach, Sage, Thousand Oaks, CA,
1996.

[21] I.F. Shaw, Ethics in qualitative research and evaluation, Journal of Social Work 3 (2003), 9-29.
[22] B. Kaplan, S. Litewka, Ethical challenges of telemedicine and telehealth, Cambridge Quarterly of

Healthcare Ethics 17 (2008), 401-416.

[23] B. Kaplan, H.P. Lundsgaarde, Toward an evaluation of a clinical imaging system: identifying benefits,
Methods of Information in Medicine 35 (1996), 221-229.

[24] B. Kaplan, Objectification and negotiation in interpreting clinical images: implications for computer-
based patient records, Artificial Intelligence in Medicine 280 (1995), 439-454.

[25] B. Kaplan, D. Duchon, Combining qualitative and quantitative approaches in information systems
research: a case study, Management Information Systems Quarterly 12 (1988), 571-586.

B. Kaplan / Evaluation of People, Social, and Organizational Issues 125

Copyright of Studies in Health Technology & Informatics is the property of IOS Press and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder’s express written permission. However, users may print, download, or email
articles for individual use.

The Triangle Model for evaluating the effect of health
information technology on healthcare quality
and safety

Jessica S Ancker,1,2,3 Lisa M Kern,2,3,4 Erika Abramson,1,2,3,5 Rainu Kaushal1,2,3,4,5

ABSTRACT
With the proliferation of relatively mature health
information technology (IT) systems with large numbers
of users, it becomes increasingly important to evaluate
the effect of these systems on the quality and safety of
healthcare. Previous research on the effectiveness of
health IT has had mixed results, which may be in part
attributable to the evaluation frameworks used. The
authors propose a model for evaluation, the Triangle
Model, developed for designing studies of quality and
safety outcomes of health IT. This model identifies
structure-level predictors, including characteristics of: (1)
the technology itself; (2) the provider using the
technology; (3) the organizational setting; and (4) the
patient population. In addition, the model outlines
process predictors, including (1) usage of the
technology, (2) organizational support for and
customization of the technology, and (3) organizational
policies and procedures about quality and safety. The
Triangle Model specifies the variables to be measured,
but is flexible enough to accommodate both qualitative
and quantitative approaches to capturing them. The
authors illustrate this model, which integrates
perspectives from both health services research and
biomedical informatics, with examples from evaluations
of electronic prescribing, but it is also applicable to
a variety of types of health IT systems.

INTRODUCTION
The potential for health information technology
(health IT) to improve the quality and safety of
healthcare is the primary impetus behind the
federal electronic health record (EHR) incentive
program.1 2 However, previous research on the
effects of health IT on healthcare delivery has had
mixed results, with some studies finding improve-
ments and others showing no effect or adverse
effects on quality or safety.3e8

Mixed findings such as these may be in part due
to the evaluation frameworks that have been used
to assess associations between the quality and
safety outcomes and the predictor variabledthat is,
the health IT itself. For example, several of these
studies were beforeeafter studies, which examined
the outcomes of interest before and after the
introduction of a technology. However, it may not
be sufficient simply to categorize a study period by
whether or not a specific technology was present.
For example, two similar healthcare delivery
settings with EHRs or computerized provider order
entry (CPOE) systems may be very different from
each other, because even the same product will be

customized with site-specific configuration of
features such as order sets and interfaces with other
clinical systems. Training and implementation
procedures also differ between institutions and
time periods. Furthermore, the technology is not
only a predictor variable but also a confounder that
can interact with other variables. Technology alters
clinical workflow, staffing levels, and user percep-
tions and attitudes; conversely, organizations can
customize technologies to support specific organi-
zational priorities, such as quality measurement or
patient safety.
Many of these factors may be potential explana-

tions for the observed differences in quality and
safety outcomes for health IT. However, unfortu-
nately, we cannot necessarily be sure of the role of
any of these factors unless they are measured reli-
ably and validly. We suggest that research on the
impact of health ITon the delivery of healthcare will
be stronger if potential predictor variables such as
these are captured systematically and prospectively
during the evaluation process.
In this paper, we outline the Triangle Evaluation

Model, an evaluation model designed to capture the
dimensions of assessment necessary to explain the
quality and safety effects of health IT, and describe
examples of how this model has informed our
evaluation work.

MODEL FORMULATION AND THEORETICAL
GROUNDING
The rapid acceleration in use of health IT nation-
wide, fueled by the federal ‘meaningful use’ policy,1

has resulted in an increased desire to understand
how these systems are affecting the quality, safety,
and efficiency of healthcare across a variety of
healthcare delivery settings. In our view, a joint
evaluation approach combining informatics and
health services research is the most effective way to
answer these research questions.
In developing an evaluation model, we reviewed

the literature on published studies evaluating the
effects of health ITon quality and safety as well as
both evaluation and implementation models
specific to health IT. We identified excellent guid-
ance from previous evaluation models and imple-
mentation researchers about evaluating a number
of aspects of health IT, including technical opera-
tions,9 diffusion, adoption, and fit,9e13 cognitive
effects,14 15 social, organizational, and workflow
impacts,9 16e20 and the general concept of ‘infor-
mation systems success’.21 In addition, we drew
from our own experience conducting quality and
safety research in the field of health IT, and

1Department of Pediatrics, Weill
Cornell Medical College, New
York, New York, USA
2Department of Public Health,
Weill Cornell Medical College,
New York, New York, USA
3Health Information Technology
Evaluation Collaborative (HITEC),
New York, New York, USA
4Department of Medicine, Weill
Cornell Medical College, New
York, New York, USA
5New York-Presbyterian
Hospital, New York, New York,
USA

Correspondence to
Dr Jessica S Ancker, Weill
Cornell Medical College, 402 E
67th St, LA-251, New York, NY
10065, USA;
[email protected]

Received 20 May 2011
Accepted 26 July 2011
Published Online First
20 August 2011

J Am Med Inform Assoc 2012;19:61e65. doi:10.1136/amiajnl-2011-000385 61

Research and applications
D

ow
nloaded from

https://academ
ic.oup.com

/jam
ia/article/19/1/61/735342 by W

alden U
niversity user on 19 June 2022

conducted iterative discussions within our research team (which
contains both health services researchers and informatics
researchers) about constructs to be measured and potential
operationalization of those measurements in the context of our
ongoing and planned research studies.

We accomplished this by mapping elements and processes
from health IT models on to the dominant theoretical model in
health services research, the Donabedian Model, which empha-
sizes a systems-level perspective on the determinants of
healthcare quality.22 23 According to this model, the quality of
a system of healthcare can be defined along three dimensions.
‘Structure’ is the system’s material, organizational, and human
resources. ‘Processes’ are the activities performed by the system
and its people, such as healthcare delivery methods. ‘Outcomes’
are the measurable end results, such as mortality, patient health
status, and medical error rates. When applying these concepts to
health IT, we were influenced not only by the evaluation liter-
ature cited above but also by a second theoretical source, soci-
otechnical theory, which describes how technology is
interconnected with social structure.16 17 20 24 Introducing
technology into an organization changes both the organization
and the technology; there is ‘a process of mutual transformation;
the organization and the technology transform each other’.24

We adapted the Donabedian Model by identifying structure
and process factors with the potential to affect quality and safety
outcomes of health IT. Four structural variables are depicted
in figure 1 and described in additional detail in the next section:
(1) the technology; (2) the provider using it; (3) the organiza-
tional setting; and (4) the involved patient population. We also
identified three categories of processes that connect pairs of
structural variables: (1) the use of the technology by the provider;
(2) the organizational implementation of the technology; and
(3) organizational policies affecting providers (table 1).

MODEL DESCRIPTION
In developing the model, we identified elements of healthcare
structure and processes that should be assessed concurrently
with the outcome variables of quality and safety. In addition, we
incorporated the sociotechnical perspective that the organiza-
tion, technology, and users would influence and change each
other, especially through the processes. In this section, we
describe the constructs that constitute the model, without
specifying how they should be assessed. Assessment methods
can be selected according to the resources of the researcher and
to the research question at hand.

Structure
In the Triangle Model, the relevant elements of structure are: (A)
the technology; (B) the healthcare organization; (C) the
healthcare provider user; and (D) the patients receiving care. In
Figure 1, these elements are represented by the three points of
the triangle and the central circle.

The technology
In order to assess impact, it is first necessary to inventory the
functional capabilities that could affect quality or safety. These
would include issues such as the usability of the user interface
and the availability of clinical decision support, electronic
(e)-prescribing, or interfaces with other systems. Hardware
issues and system reliability are also relevant to technology
performance.

The provider
The healthcare provider who uses the system has attributes that
may affect quality and safety outcomes, such as years in

Table 1 Dimensions of evaluation in the Triangle Model

Donabedian Model
categories

Triangle Model
variable types Sample quantitative variables Sample qualitative variables

Structure Organization Size; type of healthcare organization Group-level workflow and communication

Provider Specialty; computer skills; hours spent in EHR training Attitudes toward health IT or quality improvement

Technology Inventory of features; hardware and software performance Usability

Patients Demographics; insurance status; severity of illness Attitudes toward health, healthcare, or health IT

Process Organization-technology Time and resources spent on implementation, training,
and support

Institutional procedures for implementation, training, and
support; user perceptions of implementation, training,
and support

Provider-technology Individuals’ usage of system and of specific features Tasketechnology fit; perceived workflow integration;
user satisfaction

Organization-provider Time and resources directed to quality or safety initiatives Perceptions of organizational quality and safety initiatives

Outcomes Patient safety Prescribing errors; adverse drug events Perceived patient safety culture

Healthcare quality Performance on nationally recognized quality metrics Patient and provider perceptions of quality

EHR, electronic health record; IT, information technology.

Figure 1 The Triangle Evaluation Model proposes simultaneous
measurement of structure, process, and outcome variables in all
evaluations of the impact of health information technology on healthcare
quality and safety.

62 J Am Med Inform Assoc 2012;19:61e65. doi:10.1136/amiajnl-2011-000385

Research and applications
D

ow
nloaded from

https://academ
ic.oup.com

/jam
ia/article/19/1/61/735342 by W

alden U
niversity user on 19 June 2022

practice, training, and attitude toward quality improvement. In
addition, some provider attributes such as specialty, typing
skills, EHR training and experience, and age may influence how
much they use the technology.

The organization
Organizational mission, resources, and policies affect quality
outcomes directly and also influence how well a technology is
used to pursue these outcomes. For example, organizations may
or may not create usable EHR configurations in patient exami-
nation rooms, devote sufficient resources to EHR training, or
make good choices about system configuration. Small medical
practices are likely to have different resources and needs for
health IT than are large medical centers.

The patients
An organization that treats sicker patients will perform poorly
on patient outcome measures unless comparisons are adjusted
for the population’s burden of illness. A variety of methods for
risk adjustment, such as case mix adjustment and comorbidity
indices, have been developed and validated to more appropriately
compare quality outcomes across physicians or healthcare
organizations.25e27 Other patient-specific characteristics such as
health literacy, patient engagement, and attitudes toward health
IT may also be relevant.

Processes
In the Triangle Model, processes with the potential to affect
quality and safety outcomes of health IT connect the points of
the triangle.

Provideretechnology processes
Only when a technology is used as intended can relevant quality
outcomes be expected. It is thus important to assess the actual
usage of the relevant features, which is likely to vary at the level
of the individual physician according to usability and perceived
usefulness,28e30 integration into clinical workflow and
tasketechnology fit,12 31 and training on the system.

Organizationetechnology processes
Organizational decisions affect which technologies are imple-
mented, system configuration, implementation procedures, and
resources allocated to hardware and technical infrastructure,
technical support, and training. As recognized by the DeLone
and McLean model of information system ‘success’, these
organization-level factors affect the quality of the IT system as
implemented in a specific setting, which has a strong impact on
use as well as user satisfaction.21

Organizationeprovider processes
Finally, organizational policies, culture, and workflow all have
a direct effect on provider activities and the quality-related
outcomes of these activities. For example, an organization may
opt into a voluntary quality improvement initiative or pursue
a care model transformation such as patient-centered medical
home accreditation.32

MODEL APPLICATION AND VALIDATION BY EXAMPLE
The Triangle Model specifies the predictor variables that should
be captured in order to explain quality or safety outcomes of
health IT, but it does not specify how these variables should be
measured. In different situations, provider usage of a particular
EHR feature might be captured by usage logs, by a researcher

making field observations, or by self-reported survey. These
approaches each have strengths and weaknesses. In some situ-
ations, researchers may prefer intensive qualitative studies to
produce a rich and in-depth understanding of a particular situ-
ation, whereas in others, researchers may exploit data available
from the electronic system itself. The study sample size may
limit the number of quantitative predictors that can be included
in a regression model, whereas resources may limit the amount
of qualitative research that can be performed, creating a need to
balance the number of quantitative predictors and qualitative
ones included in any particular study.
Two examples presented here illustrate how we have used the

Triangle Model to inform our research. Although these both
pertain to e-prescribing, the model can be applied to a variety of
health IT evaluations.

Electronic prescribing improves medication safety in community-
based office practices
Our prospective, controlled study of a stand-alone e-prescribing
technology was one of the first to demonstrate that
e-prescribing was highly effective at reducing prescribing error
rates in community-based office practices.33 The primary
comparison in the study was users and non-users of this
e-prescribing system.
In terms of structural variables from the Triangle Model, we

first inventoried the features available in the technology. The
inventory suggested that this technology did have the potential
to reduce prescribing error rates, as it provided clinical decision
support with a wide variety of alert types as well as additional
reference resources. At the provider level, we controlled for
variables that may have affected prescribing error rates,
including years in practice, training, and specialty. Among the
patient population, we limited inclusion to adults and collected
age, gender, and medications. We studied a single independent
practice association (organization), all of whom had access to
the same e-prescribing technology and received relatively
intensive implementation and technical support (organ-
izationetechnology processes). For provideretechnology
processes, we did not quantify usage frequency as a continuous
variable because all providers were incentivized to use the
system for 100% of prescriptions and thus had very high usage
rates. Instead, we minimized variability in our dataset by
limiting the study to providers who had used the e-prescribing
system to write a minimum of 75 prescriptions.
The outcome variable, prescribing errors, was assessed using

a rigorously controlled and previously validated manual
review process in which research nurses used a standardized
methodology to evaluate paper and electronic prescriptions.
The results of this study were striking. Among providers who

adopted e-prescribing, error rates decreased from 42.5 to 6.6 per
100 prescriptions; among non-adopters, error rates remained
nearly unchanged (37.3 to 38.4 per 100 prescriptions).33

Capturing the structural elements associated with technology,
provider, and patient population allowed us to perform appro-
priate adjustment in the statistical model, and designing the
study to control the variability in the remaining structural and
process elements simplified the analyses.

Ambulatory prescribing safety in two EHRs
In this preepost study, the outcome of interest was also
prescribing errors, but the primary comparison was between use
of two different EHR systems, an in-house system that was
replaced at the institutional level by a commercial EHR system
(Abramson EL, Patel V, Malhotra S, et al; unpublished data).34

J Am Med Inform Assoc 2012;19:61e65. doi:10.1136/amiajnl-2011-000385 63

Research and applications
D

ow
nloaded from

https://academ
ic.oup.com

/jam
ia/article/19/1/61/735342 by W

alden U
niversity user on 19 June 2022

We applied the Triangle Model by inventorying the features
available in each technology. The locally developed e-prescribing
system provided very little clinical decision support, whereas the
commercial system provided a wide variety of clinical decision-
support alerts and default dosing with the potential to reduce
prescribing error rates. At the provider level, we adjusted for
demographics and years in practice, and, among the patients, we
restricted eligibility to adults and adjusted for age, sex, and
insurance status. This study was conducted in a single organi-
zation, where the locally developed system was replaced
by the commercial system institution-wide, all physicians
underwent the required training, and use of the new system
was mandatory (organizationetechnology and organizatione
provider processes). The study showed that implementation of
the commercial system was associated with a marked fall in the
rate of prescribing errors in the short term, with a further
decrease at 1 year. However, when inappropriate abbreviations
were excluded from the analysis, the rate of errors increased
immediately after the transition to the new system, and at
1 year returned to baseline.

Concurrently, we sought additional insight into the providere
technology processes through a survey, semistructured inter-
views, and field observations; this qualitative data collection was
performed concurrently with our quantitative data collection.
Among other findings, the results suggested that physicians
perceived the locally developed system as faster and easier to use,
that the clinical decision-support alerts in the new system led to
‘alert fatigue’ and were often over-ridden, and that few users
knew how to use system shortcuts to increase efficiency. These
findings provided additional insight into the potential reasons
behind the observed spike in certain types of prescribing errors
during the transition from one system to the other (Abramson
EL, Patel V, Malhotra S, et al; unpublished data). By considering
these factors in the design of the research, and conducting
qualitative and quantitative evaluation simultaneously, we
increased the explanatory power of our study.

DISCUSSION AND IMPLICATIONS
Research on the effects of health IT may oversimplify complex
issues if health IT is treated as a simple categorical variable
(present or absent, or before or after). Capturing more detailed
predictor variables about the technology, users, and the
surrounding context increases the ability to interpret findings
and compare studies, while minimizing the need to cite
unmeasured variables as potential explanations of results. In this
paper, we have proposed a more comprehensive evaluation
model specifically designed for studies of the quality or safety
effects of health IT. The Triangle Model specifies that research
studies should assess structural elements (the technology,
the provider using it, and the organizational setting) and
process variables (provideretechnology processes such as usage,
organizationetechnology processes such as infrastructure
support, and organizationeprovider processes such as quality
improvement initiative), and that evaluations should adjust for
characteristics of the patient population.

The Triangle Model carries the implication that unmeasured
structure and process variables may account for why field
studies of the effects of health IT on quality, safety, and effi-
ciency have had mixed results, with some showing the expected
improvements, others failing to find any effect, and others
revealing adverse effects.3e7

As an example, we can apply the Triangle Model to under-
standing two high-profile studies of a commercial CPOE system.
Although this system had been shown to reduce prescribing

errors and adverse drug events,7 Han et al found that the system
was associated with a mortality increase in a pediatric intensive
care unit,5 whereas Del Beccaro and colleagues found no such
association in a similar setting.8 A critical review of these two
papers shows that both sets of researchers came up with plau-
sible potential explanations for their results, but since data were
not collected in a systematic fashion as part of the evaluation,
no definitive conclusions can be drawn.
Han et al, as well as other commentators,35 attribute their

findings to a number of variables they did not measure. They
describe some aspects of the system that they felt may have
presented usability barriers (interrupting provideretechnology
processes), as well as problems with the hospital’s technical
infrastructure such as lack of order sets and an overloaded
wireless network (organizationetechnology processes), and
perceived negative effects on workflow and communication
(organization factors).5 Similarly, Del Beccaro et al list several
unmeasured factors that may have played a role at the level of
the technology and the organization, as well as in the interac-
tions between them. These included the organization’s
construction of order sets and perceived emphasis on encour-
aging good communication processes among healthcare
providers.8 Han et al suggested that the effect of the unmeasured
factors was to render the CPOE system slower and less reliable
than the old paper-based system and thus less safe in critical
care; Del Beccaro et al dispute this interpretation and suggest it
was based on an underestimate of the true time needed to place
paper orders. However, neither study actually measured the
speed of ordering or the other factors cited as potential expla-
nations for the differences between the results. We propose that
an evaluation of the system conducted under the Triangle Model
might have more systematically captured factors such as these
that may have contributed to the healthcare quality outcomes,
reducing the need for speculation about the causes of the
differences.

Comparisons
In the Triangle Model, we have attempted to summarize and
categorize elements from other evaluation models while
emphasizing the relationship between technology and health-
care quality and safety outcomes. This creates some similarities
to some other evaluation frameworks. Like the SEIPS Model of
Carayon et al,19 the Triangle Model adapts the Donabedian
Model for use in health ITevaluations. However, SEIPS considers
primarily healthcare delivery processes, whereas in the Triangle
Model, additional processes of interest include the interactions
between the individual user and the technology, and between
the organization and the technology. This reflects our emphasis
on capturing technology usage patterns as potential predictors of
quality outcomes. Our focus on individual usage of technology
also distinguishes the Triangle Model from another evaluation
model, Westbrook’s multi-method evaluation model,20 which
encourages the study of organizational-level factors.

Limitations
The Triangle Model is not intended to be a model of diffusion,
adoption, or implementation nor a framework to study
outcomes such as successful technology adoption, satisfaction,
or workflow. Rather, it is designed to guide evaluations that seek
to assess the effect of health IT systems on healthcare delivery,
specifically the quality and safety of healthcare. It is thus most
appropriate for summative evaluations of relatively mature
health IT systems with good adoption rates. As others have
noted, summative evaluations are less appropriate for systems in

64 J Am Med Inform Assoc 2012;19:61e65. doi:10.1136/amiajnl-2011-000385

Research and applications
D

ow
nloaded from

https://academ
ic.oup.com

/jam
ia/article/19/1/61/735342 by W

alden U
niversity user on 19 June 2022

development or in the process of implementation.36 An addi-
tional limitation of this model is that we have not specified the
measurement instruments or the level of measurement for the
various predictor variables we have identified. However, we
believe that the resulting flexibility may make this model more
generalizable and widely applicable than it otherwise would be.
Finally, this model has not been formally validated, and it is
possible that additional dimensions could be determined to be
useful.

Conclusions
This paper proposes a general model for conducting evaluations
of the impact of health IT systems on the outcomes of health-
care quality and safety. This model outlines the domains and
constructs that should be assessed, but does not specify whether
the methods should be quantitative, qualitative, or hybrid. In
our experience, we have found value in applying a variety of
different methods, sometimes with the purpose of producing
rich qualitative data to explain results, and other times taking
advantage of the capabilities of electronic systems to obtain
quantitative datasets that allow statistical modeling. We have
provided illustrative examples from the domain of medication
safety in the ambulatory setting, but the model is broadly
applicable to a variety of health IT applications. An evaluation
approach that integrates perspectives from health services
research and biomedical informatics has the potential to capture
the quality and safety effects of the health IT systems that are
currently transforming the US healthcare system.

Funding The investigators are supported by the New York State Department of Health
(NYS contract number C023699).

Competing interests None.

Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES
1. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health

records. N Engl J Med 2010;363:501e4.
2. Centers for Medicare & Medicaid Services (CMS), HHS. Medicare and

Medicaid programs; electronic health record incentive program. Final rule. Fed Regist
2010;75:44313e588.

3. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry
and clinical decision support systems on medication safety: a systematic review.
Arch Intern Med 2003;163:1409e16.

4. Chaudhry B, Jerome W, Shinyi W, et al. Systematic review: impact of health
information technology on quality, efficiency, and costs of medical care. Ann Intern
Med 2006;144:742e52.

5. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after
implementation of a commercially sold computerized physician order entry system.
Pediatrics 2005;116:1506e12.

6. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry
systems in facilitating medication errors. JAMA 2005;293:1197e203.

7. Upperman JS, Staley P, Friend K, et al. The impact of hospitalwide computerized
physician order entry on medical errors in a pediatric hospital. J Pediatr Surg
2005;40:57e9.

8. Del Beccaro MA, Jeffries HE, Eisenberg MA, et al. Computerized provider order
entry implementation: no association with increased mortality rates in an intensive
care unit. Pediatrics 2006;118:290e5.

9. Anderson JG, Aydin CE, eds. Evaluating the Organizational Impact of Healthcare
Information Systems. 2nd edn. New York: Springer Science, 2005. Health Informatics
Series.

10. Greenhalgh T, Stramer K, Bratan T, et al. Introduction of shared electronic records:
multi-site case study using diffusion of innovation theory. BMJ 2008;337:a1786.

11. Rogers E. Diffusion of Innovations. 4th edn. New York: Free Press, 1995.
12. Ammenwerth E, Iller C, Mahler C. IT adoption and the interaction of task,

technology and individuals: a fit framework and a case study. BMC Med Inform Decis
Mak 2006;6:3.

13. Kaplan B. Evaluating informatics applicationsdsome alternative approaches:
theory, social interactionism, and call for methodological pluralism. Int J Med Inform
2001;64:39e56.

14. Patel VL, Kushniruk AW, Yang S, et al. Impact of a computer-based patient record
system on data collection, knowledge organization, and reasoning. J Am Med Inform
Assoc 2000;7:569e85.

15. Patel VL, Arocha JF, Kaufman DR. A primer on aspects of cognition for medical
informatics. J Am Med Inform Assoc 2001;8:324e43.

16. Aarts J, Ash J, Berg M. Extending the understanding of computerized provider order
entry: implications for professional collaboration, workflow and quality of care. Int J
Med Inform 2007;76:S4e13.

17. Berg M, Aarts J, van der Lei J. ICT in health care: sociotechnical approaches.
Methods Inf Med 2003;42:297e301.

18. Ash JS, Bates DW. Factors and forces affecting EHR system adoption: report of
a 2004 ACMI discussion. J Am Med Inform Assoc 2005;12:8e12.

19. Carayon P, Schoofs Hundt S, Karsh BT, et al. Work system design for patient safety:
the SEIPS model. Qual Saf Health Care 2006;15(Suppl 1):i50e8.

20. Westbrook JI, Braithwaite J, Georgiou A, et al. Multimethod evaluation
of information and communication technologies in health in the context of
wicked problems and sociotechnical theory. J Am Med Inform Assoc
2007;14:746e55.

21. DeLone WH, McLean ER. The Delone and McLean model of information systems
success: a 10-year update. J Manag Inform Syst 2003;19:9e30.

22. Donabedian A. The quality of care. How can it be assessed? JAMA
1988;260:1743e8.

23. Mitchell PH, Ferketich S, Jennings BM. Quality health outcomes model. Health
Policy 1998;30:43e6.

24. Berg M. Implementing information systems in health care organizations: myths and
challenges. Int J Med Inform 2001;64:143e56.

25. Charlson ME, Pompei P, Ales K, et al. A new method of classifying prognostic
comorbidity in longitudinal studies: development and validation. J Chronic Dis
1987;40:373e83.

26. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with
ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613e19.

27. Weiner J, Starfield B, Steinwachs D, et al. Development and application of
a population-oriented measure of ambulatory care case-mix. Med Care
1991;29:452e72.

28. Rogers EM. Diffusion of Innovations. New York: Free Press, 1962.
29. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of

information technology. MIS Quarterly 1989;13:319e40.
30. Davis F. User acceptance of information technology: system characteristics,

user perceptions and behavioral impacts. Int J Man Mach Stud 1993;38:475e87.
31. Goodhue D, Thompson R. Task-technology fit and individual performance. MIS

Quarterly 1995;19:213e36.
32. National Committee for Quality Assurance. Physician Practice

ConnectionsdPatient-Centered Medical HomeTM. Washington DC: National
Committee for Quality Assurance. http://www.ncqa.org/tabid/631/Default.aspx
(accessed 21 Jul 2011).

33. Kaushal R, Kern LM, Barron Y, et al. Electronic prescribing improves medication
safety in community-based office practices. J Gen Intern Med 2010;25:530e6.

34. Abramson EL, Malhotra S, Fischer K, et al. Transitioning between electronic health
records: effects on ambulatory prescribing safety. J Gen Intern Med
2011;26:868e74.

35. Sittig DF, Ash JS, Zhang J, et al. Lessons from “unexpected increased mortality
after implementation of a commercially sold computerized physician order entry
system”. Pediatrics 2006;118:797e801.

36. Friedman CP. “Smallball” evaluation: a prescription for studying community-based
information interventions. J Med Libr Assoc 2005;93(4 Suppl):S43e8.

PAGE fraction trail=5

J Am Med Inform Assoc 2012;19:61e65. doi:10.1136/amiajnl-2011-000385 65

Research and applications
D

ow
nloaded from

https://academ
ic.oup.com

/jam
ia/article/19/1/61/735342 by W

alden U
niversity user on 19 June 2022

CONTINUING EDUCATION
1.0ANCCCONTACT HOUR

Developing and Implementing a Simulated Electronic
Medication Administration Record for Undergraduate
Nursing Education
Using Sociotechnical Systems Theory to Inform Practice and Curricula
Richard G. Booth, PhD, RN, Barbara Sinclair, MScN, RN, Laura Brennan, BScN, RN, Gillian Strudwick, MN, RN

Co

Au
an
To

Th
W
Ca

Th
an

Co
He
Ca

Vo

Knowledge and skills related to medication administration
are a fundamental element of nursing education. With the
increased use of electronic medication administration tech-
nology in practice settings where nurses work, nursing edu-
cators need to consider how best to implement these forms
of technology into clinical simulation. This article describes the
development of a simulated electronic medication administra-
tion system, including the use of sociotechnical systems the-
ory to inform elements of the design, implementation, and
testing of the system. Given the differences in the medication
administration process and workflow generated by electronic
medication administration technology, nursing educators
should explore sociotechnical theory as a potentially informa-
tive lens from which to plan and build curricula related to sim-
ulation activities involving clinical technology.

KEY WORDS: Barcode medication administration,
Computerized provider order entry, Electronic medication
administration, Informatics, Nursing, Nursing education,
Simulation, Sociotechnical

significant element of any nursing curriculum is
the instruction of safe and effective medication

A administration practices drawing upon nursingknowledge.1 During basic undergraduate nursingeducation, students generally receive both theoret-

ical courses related to pharmacology and clinically focused ex-
periences in the pragmatics of medication administration.
Prior to students administering medications to real patients,
learners are also typically provided simulated learning oppor-
tunities within classroom and clinical simulation settings.2–4 In

pyright © 2017 Wolters Kluwer Health, Inc. All rights reserved.

thor Affiliations: Arthur Labatt Family School of Nursing, Western University London (Dr Booth
d Mss Sinclair and Brennan); and Lawrence S. Bloomberg Faculty of Nursing University of
ronto, Toronto, Ontario, Canada (Ms Strudwick).

is study was funded in part through a Fellowship in Teaching Innovation Award (to R.B., B.S.),
estern University Teaching Support Centre, and a Canadian Association of Schools of Nursing/
nada Health Infoway Digital Health Nursing Faculty Peer Network (R.B., B.S., G.S.) grant.

e authors have disclosed that they have no significant relationship with, or financial interest in,
y commercial companies pertaining to this article.

rresponding author: Richard G. Booth, PhD, RN, Arthur Labatt Family School of Nursing,
alth Sciences Addition H035, 1151 Richmond Street, Western University London, Ontario,
nada N6A 3K7 ([email protected]).

lume 35 | Number 3

Copyright © 2017 Wolters Kluwer H

a clinical simulation environment, students are able to prac-
tice skills and further develop their theoretical knowledge
of pharmacology with professional practice skills related to
medication administration.

The recent introduction of electronic medication adminis-
tration (eMAR) and computerized provider order entry (CPOE)
systems in many healthcare and hospital environments has fun-
damentally changed how nurses administer medications.5–7

Electronic medication administration systems are electronic
health (eHealth) technologies that clinicians use to record and
validate the administration of medications, whereby a nurse is
required to scan a patient identification bracelet barcode and
prescribed medications in order to confirm that the medications
being provided to the patient are correct in terms of timing, dose,
and route. Given the high frequency and rate of adverse medica-
tion events in healthcare,8 some have proposed that eMAR/
CPOE technology is essential to reduce medication pre-
scribing and administration errors.9–11

Although eMAR systems appear simple to operate, med-
ication administration using these systems is a subtle, yet sig-
nificant progression from the processes found in traditional
paper-based medication administration activities (eg, record
keeping, workflow, and methods of instruction).12 Within
nursing education, there has been little development or re-
search toward evidence-based approaches to implement
eMAR technology into curricula or education practices re-
lated to teaching eMAR. Currently, the majority of the nurs-
ing research literature exploring the integration or teaching of
these forms of clinical technology has focused on electronic
medical records (EMRs) and other related technology.13–15

Medical education literature has reported more experiences
in relation to the integration and teaching of eMAR/CPOE
technology into curricula, including exploring medical stu-
dents’ abilities to write orders,16 learning and supervision
implications of using CPOE in clinical education,17 and stu-
dent preferences and attitudes toward using CPOE technology.18

In light of the lack of nursing education literature regarding
eMAR technology, it is important for nursing educators to
explore and develop teaching-learning practices that are

CIN: Computers, Informatics, Nursing 131

ealth, Inc. All rights reserved.

CONTINUING EDUCATION

sensitive to the dynamic human-technical interface require-
ments of an eMAR system. A long sequence of research in
the health informatics literature has demonstrated that eMAR
and related systems have the potential to facilitate unintended
consequences throughout the medication administration pro-
cess, including the generation of new types of medication er-
rors, human-technical interface difficulties, and a redefining
of workflow and administration processes.19–21

Therefore, in an effort to generate usable insights for
nursing educators, the purpose of this article is twofold: (1)
to report details regarding the development and implemen-
tation of a simulated eMAR platform, designed and built
to operate within the school’s clinical simulation suite (CSS);
and (2) to offer recommendations related to the development
and implementation of an eMAR system in clinical simula-
tion, informed through the lens of a sociotechnical systems
theoretical perspective.

THE DEVELOPMENT AND THEORETICAL
CONSIDERATIONS OF IMPLEMENTING A SIMULATED
eMAR SYSTEM IN NURSING EDUCATION
The addition of an eMAR system into nursing education
represented a significant change in how medication ad-
ministration was both taught to and conceptualized for stu-
dent populations. To undertake this curriculum evolution,
the development team conducted a review of existing prac-
tices related to eMAR medication administration. Included
in this review were training materials developed for clinical
staff at the university affiliated hospital system,22 academic
literature related to CPOE and eMAR technology,14,23–27

and insights from other clinically active faculty. In order to
draw together these recommendations and research, the the-
oretical lens of sociotechnical systems theory28–30 was se-
lected to both inform the development of the simulated
eMAR system and its subsequent implementation into the
university CSS and nursing education.

SOCIOTECHNICAL SYSTEMS THEORY
Sociotechnical systems theory is a body of literature that ex-
plores the dynamic relationship between humans and mate-
rial (ie, technology) objects in the generation of action. In the
case of healthcare, sociotechnical perspectives are generally
used to explore how humans and health technology interact
and operate within health environments, since all work prac-
tices are conceptualized as larger, interconnected networks
of peoples, tools, objects, and routines.29 A sociotechnical
lens enables an educator to question the role, importance,
and influence of material objects such as technology in the
learning or operation of specific tasks, including within clini-
cian simulated education.31–34 From a clinical simulation
perspective, appreciating the various human and technical
forces present in a learning experience involving eHealth

132 CIN: Computers, Informatics, Nursing

Copyright © 2017 Wolters Kluwer H

technology such as eMAR is argued to be an important,
yet sometimes overlooked, consideration. For instance, the
transposition of teaching methods related to medication ad-
ministration using a paper-based eight medication rights35

(right patient, medication, dose, route, time, documentation,
reason, and response) process to a computerized eMAR
methodology must appreciate the new workflows and rela-
tionships generated between both human and technologi-
cal entities. As outlined by Novak et al,36 CPOE and
eMAR technology commonly forced “rigid interpretation”
36(pe333) of medication rights processes. In their study, they
found that the rigid, stepwise interpretation mandated by
eMAR technology forced nurses to generate adaptations
and workarounds to complete the medication administration
process. Furthermore, eMAR technology and its prescribed
processes failed to appreciate the workflow flexibility commonly
required during nursing care. Subsequently, implementation of
eMAR technology in nursing education requires educators to
understand the various sociotechnical implications generated
by this new form of clinical technology and ensure this knowl-
edge is meaningfully translated into teaching methods. Al-
though other informative theoretical models exist, sociotechnical
systems theory was deemed by the development team to be
a conceptual perspective that adequately addressed the com-
plex relationships among several elements of technology use
in nursing practice, including the environmental and social
context, and related technological infrastructure. Models
such as the Technology Acceptance Model,37 Unified Theory
of Acceptance and Use of Technology,38 and DeLone and
McLean Information Systems Success Model39 were not seen
by the development team to be reactive enough to capture
the multifaceted simulation context; furthermore, these other
models have also been critiqued within the nursing literature
for their lack of specificity and sensitivity toward elements of
the nursing role.40–42

In the following sections, the development of the Simulated
Medication Administration Record Technology (SMART)
eMAR and the way sociotechnical systems theory was used to
guide all elements of the eMAR creation and implementation
are described. The Sittig and Singh43 Eight Dimensional
Model was selected as the guiding sociotechnical framework
to assist in all elements of the conceptualization and development
of the SMART eMAR system. Further discussion and descrip-
tion of the Sittig and Singh model and its related dimensions
are provided in the following sections (Table 1).

DESIGN AND DEVELOPMENT OF THE eMAR SYSTEM
As part of ongoing curriculum and innovation development,
nursing faculty and researchers developed a simulated eMAR
system for learning purposes. Known as the SMART eMAR
system, the technology was developed to operate within a
modern university CCS and be used by nursing students

March 2017

ealth, Inc. All rights reserved.

Table 1. Sittig and Singh43 Model Dimensions and Descriptions

Sittig and Singh43 Model Dimension Dimension description

Primarily technical dimensions Hardware and software Focused on the technical elements of a health technology,
including hardware, software, and other related
technological peripherals of the related system

System measurement and monitoring Related to the measurement and monitoring of a health
technology, in terms of a number of variables, including
system uptime, response time, click rates, and other
measureable clinical or benchmark outcomes

Primarily
social dimensions

People This dimension represents all human users of the health
technology, throughout development and implementation of
the system. The dimension also captures how the system
“help users think and make them feel”43(pi70)

Workflow and communication The process or steps that people take to complete action
with a health technology system, including various
communications between providers and also
technological systems

Clinical content Represents all the information and data related to the
clinical presentation or episode, including laboratory
information, clinical records, and other various structured
and unstructured clinical data

Internal organizational policies and procedures This dimension represents the various policy and
procedural structures within an organization that shape
action and behavior

External rules, regulations, and pressures The dimension encompasses the various forces externally
that act upon various health technologies and its users
in the delivery of care. These forces include various
macropolicy data exchange agreements, data standards,
and other health-human resource shortages

Sociotechnically blurred dimension Human-computer interface The human-computer interface dimension accounts
for how users operationalize, interact, and “see, touch
or hear”43(pi70) with the system and its related interface(s)

during their simulated clinical education. The physical layout
of the CSS included a nine-bed hospital ward, with hospital
beds, infusion pumps, simulated oxygen outlets, a medication
room, nursing station, and storage area with supplies for pa-
tient care. The CSS also featured a host of medium- and
high-fidelity mannequins that could be programmed to simu-
late a variety of patient conditions. A network of cameras and
microphones was connected to each mannequin, allowing ob-
servers (eg, clinical instructors, teaching assistants) to act as the
voice of the mannequins and interact verbally with students.
Standardized patients (ie, actors trained by the local medical
school to portray a variety of patient conditions) were also
used during simulated practice by nursing students to provide
realism in the simulation experience. Simulated medication
administration experiences were initiated at the beginning of
students’ second year in the baccalaureate nursing program.
The development of the SMART eMAR was directly in-
spired by the recognition of the gap in nursing curricula, re-
garding the increasing need to educate students about
eMAR practices. Second, there was a desire by nursing faculty
to obtain a platform that was customizable and owned by the

Volume 35 | Number 3

Copyright © 2017 Wolters Kluwer H

university to afford students an opportunity to learn and use a
computerized administration technology in a cost-effective
fashion. Finally, the developers hoped to create a system that
would replicate the workflow and processes of a real eMAR
system as closely as possible, without burdening students or in-
structors with excessive functionality or technical complexity
that would not be of value.

The SMART eMAR was developed over a 4-month pe-
riod utilizing a custom-built user interface, thin-client personal
computers, flat screen LCD monitors, barcode generation
software, and retail-grade barcode scanners. The aforemen-
tioned components were mounted on six hospital-grade
mobile clinical carts, similar to those used in the university-
affiliated hospital system in order to develop a cost-effective,
mobile, eMAR system workstation for educational purposes.
The SMART eMAR system supported similar workflows
and processes of a real clinical system (ie, prompts for incor-
rect medication scans, basic decision support, personalized
demographic information for each patient) and also mim-
icked the physical usability of other eMAR workstation
found within acute care environments (Figure 1).

CIN: Computers, Informatics, Nursing 133

ealth, Inc. All rights reserved.