Measurement error in performance studies of health information technology

Lessons from the management literature

Adam Seth Litwin, Ariel Chanan Avgar, P. J. Pronovost

Research output: Contribution to journalReview article

Abstract

Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature's more conceptual examination of health IT's limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mismeasured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.

Original languageEnglish (US)
Pages (from-to)210-220
Number of pages11
JournalApplied Clinical Informatics
Volume3
Issue number2
DOIs
StatePublished - Dec 1 2012

Fingerprint

Medical Informatics
Measurement errors
Information technology
Health
Health Information Management
Research Personnel
Organizational Innovation
Decision Making
Research Design
Technology
Innovation
Decision making

Keywords

  • Electronic health records
  • Health information technology
  • Organizational behavior
  • Research methods

ASJC Scopus subject areas

  • Health Informatics
  • Computer Science Applications
  • Health Information Management

Cite this

Measurement error in performance studies of health information technology : Lessons from the management literature. / Litwin, Adam Seth; Avgar, Ariel Chanan; Pronovost, P. J.

In: Applied Clinical Informatics, Vol. 3, No. 2, 01.12.2012, p. 210-220.

Research output: Contribution to journalReview article

Litwin, Adam Seth ; Avgar, Ariel Chanan ; Pronovost, P. J. / Measurement error in performance studies of health information technology : Lessons from the management literature. In: Applied Clinical Informatics. 2012 ; Vol. 3, No. 2. pp. 210-220.
@article{8a104e603d4943098fd4091314f9f742,
title = "Measurement error in performance studies of health information technology: Lessons from the management literature",
abstract = "Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature's more conceptual examination of health IT's limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mismeasured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.",
keywords = "Electronic health records, Health information technology, Organizational behavior, Research methods",
author = "Litwin, {Adam Seth} and Avgar, {Ariel Chanan} and Pronovost, {P. J.}",
year = "2012",
month = "12",
day = "1",
doi = "10.4338/ACI-2012-02-R-0004",
language = "English (US)",
volume = "3",
pages = "210--220",
journal = "Applied Clinical Informatics",
issn = "1869-0327",
publisher = "Schattauer GmbH",
number = "2",

}

TY - JOUR

T1 - Measurement error in performance studies of health information technology

T2 - Lessons from the management literature

AU - Litwin, Adam Seth

AU - Avgar, Ariel Chanan

AU - Pronovost, P. J.

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature's more conceptual examination of health IT's limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mismeasured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.

AB - Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature's more conceptual examination of health IT's limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mismeasured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.

KW - Electronic health records

KW - Health information technology

KW - Organizational behavior

KW - Research methods

UR - http://www.scopus.com/inward/record.url?scp=84884487132&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884487132&partnerID=8YFLogxK

U2 - 10.4338/ACI-2012-02-R-0004

DO - 10.4338/ACI-2012-02-R-0004

M3 - Review article

VL - 3

SP - 210

EP - 220

JO - Applied Clinical Informatics

JF - Applied Clinical Informatics

SN - 1869-0327

IS - 2

ER -