Skip to main content

Psychology: Research and Review

Table 1 Systematic review of the scale development process recorded in 105 included studies

From: Scale development: ten main limitations and recommendations to improve future research practices

Study

Scale

Step 1

Step 2

Step 3

N

step 3

Initial item pool

Final item pool

Main limitations reported

Aagja and Garg (2010)

PubHosQual Scale

LR/ES/I

EJ

EFA/CFA/NV/CV/DV/ICR

401

59

24

LG

Ahmad et al. (2009)

Service Quality Scale

LR/FC

EJ

CFA/CV/DV/S-RR/ICR

413

31

10

LG

Akter et al. (2013)

MHealth Service Quality Scale

LR/ES/FC/I

EJ

EFA/CFA/NV/PV/CV/DV/I-JR/I-T-CR/ICR

305

29

22

LG/CSM

Alvarado-Herrera et al. (2015)

CSRConsPerScale

LR/ES

EJ

CFA/CV/DV/NV/ICR

1087

73

18

LG/Lack of the PV

Armfield (2010)

IDAF-4C+

LR/ES

EJ

EFA/CtV/PV/T-RR/ICR

1083

29

8

LG/Lack of the CV/SRM

Atkins and Kim (2012)

Smart Shopping Scale

LR/FC/I

EJ/TPJ

EFA/CFA/NV/CV/DV/ICR

1.474

62

15

LG

Bagdare and Jain (2013)

Retail Customer Experience Scale

LR/EP

EJ/TPJ

EFA/CFA/CV/ICR

676

45

12

LG/This study has not established DV and NV

Bakar and Mustaffa (2013)

Organizational Communication Scale

LR/FC

EJ

EFA/CFA/CtV/CV/DV/T-RR/ICR

596

386

38

LG/Inadequate choose variables to be correlated

Beaudreuil et al. (2011)

URAM

EP/I

EJ/TPJ

EFA/CV/DiV/T-RR/ICR

138

52

9

LG/SSS

Bhattacherjee (2002)

Individual trust in online firms scale

LR/ES

TPJ

CFA/CV/DV/NV/ICR

269

18

7

WBS

Blankson et al. (2007)

International consumers’selection of banks scale

LR/FC

EJ/TPJ

EFA/CFA/CV/PV/NV/ICR/I-T-CR

773

60

18

LG

Blankson et al. (2012)

Scale measuring college students’ choice criteria of credit cards

FC/ES

EJ

EFA/CFA/CV/DV/S-RR/ICR

405

59

19

LG/CSM

Bolton and Lane (2012)

IEO

LR/ES

TPJ

EFA/IV/EV/CV/DV/I-T-CR/ICR

1162

NCR

10

LG/Lack of the CFA

Bova et al. (2006)

HCR

I/FC.

EJ

EFA/T-RR/ICR

99

58

15

LG/Scale was administered in a face-to-face interview/SSS.

Boyar et al. (2014)

CESS

LR

EJ

CFA/DV/ICR

446

140

52

CSM

Brock and Zhou (2005)

OIU

LR/I

EJ

DV/PV/NV/ICR

112

NCR

7

LG

Brun et al. (2014)

Online relationship quality scale

LR, and ES

EJ/TPJ

EFA/CFA/CV/DV/PV/ICR

476

33

21

LG

Butt and Run (2010)

SERVQUAL model scale

LR/EP

EJ

EFA/CFA/CV/DV/ICR

340

17

17

LG

Caro and García (2007)

Perceived service quality in urgent transport service scale

LR/I/ES

EJ/TPJ

EFA/CFA/DV/CV/NV/I-T-CR/ICR

375

68

38

LG/Lack of the CV or DV

Chahal and Kumari (2012)

CPV Scale

LR/ES

EJ/TPJ

EFA/CFA/CV/I-T-CR/ICR

515

32

27

LG

Chen et al. (2009)

Process Orientation Scale

LR/I

EJ

EFA/CFA/CV/DV/I-I-CR/ICR

356

NCR

6

LG/SSS/Lack of the NV

Choi et al. (2011)

Measure of dyspnea severity and related functional limitations

LR/I/EP

EJ/TPJ

EFA/CFA/CV/DiV/T-RR/ICR

608

364

33

CSM

Christophersen and Konradt (2012)

Reflective and formative usability scales

LR/EP/ES

EJ

CFA/CtV/PV/EV/ICR

378

80

54

CSM/SRM

Cicero et al. (2010)

ASI

LR/EP

EJ/TPJ

EFA/CFA/CV/DV/ICR

1281

NCR

29

Items are not reverse-scored

Coker et al. (2011)

IPPR

LR/I

EJ/TPJ

EFA/CFA/CtV/NV/DV/ICR

1200

65

11

LG

Coleman et al. (2011)

B2B service brand identity scale

LR

EJ/TPJ

EFA/CFA/DV/I-T-CR/ICR

210

119

15

LG/Deductive approach to scale development

Colwell et al. (2008)

Measure of service convenience

LR/I

EJ/TPJ

EFA/CFA/CV/DV/NV/ICR

201

30

17

LG/CSM

Cossette et al. (2005)

Caring Nurse–Patient Interactions Scale

LR/ES

EJ

EFA/CV/CtV/ICR

332

121

70

CSM

Dennis and Bocarnea (2005)

Servant leadership assessment instrument

LR/EP

EJ

EFA/CtV/ICR

293

71

42

MD

Devlin et al. (2014)

Fairness measurement scale

LR, and ES

EJ/TPJ

EFA/CFA/CV/DV/NV/ICR

3100

9

8

LG

Dunham and Burt (2014)

Organizational Memory Scale

LR/ES

NCR

EFA/CFA/CV/T-RR/ICR

431

72

21

SRM

Edwards et al. (2010)

STL

FC

TPJ

EFA/CV/CtV/ICR

270

NCR

84

LG

Feuerstein et al. (2005)

Response to work in those with upper extremity pain scale

LR/FC/ES

TPJ

EFA/T-RR/ICR

282

136

136

LG/SSS

Fisher et al. (2014)

Entrepreneurial Success Scale

LR/I

EJ

EFA/CFA/ICR

213

9

4

SSS/Subjective Analysis/SRM

Flight et al. (2011)

Characteristics-based innovation adoption scale

LR

EJ/TPJ

EFA/CFA/ICR/EV/CV/DV/NV/ICR

430

122

43

LG

Forbush et al. (2013)

EPSI

LR

NCR

EFA/CFA/CV/CtV/DV/T-RR/ICR

1528

160

45

LG

Foster et al. (2015)

GNS

LR/ES

NCR

EFA/CFA/ICR

2259

35

33

Lack of the validity

Franche et al. (2007)

RRTW

LR/EP

EJ

EFA/CFA/CrV/PV/IV/EV/ICR

632

NCR

22

SSS/CSM

Gesten (1976)

HRI

LR/EP/ES

EJ

EFA/T-RR/ICR

592

79

54

LG

Gibbons et al. (2013)

MULTIPleS

LR/ES/QER

TPJ

EFA/T-RR/ICR

490

53

22

LG

Gligor and Holcomb (2014)

SCA

LR/ES/I

EJ/TPJ

EFA/CFA/CV/DV/EV/ICR

151

NCR

21

CSM

Glynn et al. (2015)

PFS

ES/QER

NCR

EFA/CV/T-RR/ICR

1496

26

10

LG/MD

Gottlieb et al. (2014)

Consumer perceptions of trade show effectiveness scale

LR/I

NCR

EFA/CFA/CV/DV/NV/I-T-CR/ICR

739

13

11

LG/Items ambiguous/Difficult to control variables

Hall et al. (2002)

General Trust in physicians scale

LR/FC/EP

EJ/TPJ

EFA/CV/CtV/ICR

502

25

11

LG/CSM

Han et al. (2011)

Scale of switching barriers in full-service restaurants

LR/FC

EJ/TPJ

EFA/CFA/CV/NV/I-JR/ICR

401

NCR

17

LG

Henderson-King and Henderson-King (2005)

ACSS

LR

TPJ

EFA/DV/CV/T-RR/ICR

1288

26

15

LG

Hernandez and Santos (2010)

Development-based trust

LR/I

TPJ

EFA/CFA/CV/DV/NV/ICR

238

30

27

CSM

Hildebrandt et al. (2004)

MDDI

LR/ES

NCR

EFA/CV/DiV/T-RR/ICR

245

21

20

LG/Lack of the DV

Ho and Lin (2010)

Scale for measuring internet banking service quality

LR/I/ES

TPJ

EFA/DV/CV/ICR

130

30

17

SSS

Jong et al. (2014)

CRIQ

LR/ES

EJ

EFA/CFA/T-RR/ICR

310

120

120

Lack of the CFA - the CFI fit is below the 0.90

Kim et al. (2011)

CEI

LR

TPJ

EFA/CFA/CV/DV/ICR

397

134

26

LG/Lack of the validity/WBS

Kim et al. (2014)

SAPS

LR

EJ

CFA/CtV/CV/ICR

795

29

15

Lack of the DV

Kwon and Lennon (2011)

Brand Association Scale

LR

EJ

EFA/CFA/CV/DV/I-JR/ICR

671

28

14

LG

Lin and Hsieh (2011)

SSTQUAL Scale

LR/FC/I

EJ

EFA/CFA/CV/DV/NV/I-T-CR/ICR

1238

75

20

LG/subjectivity in the EFA and CFA

Lombaerts et al. (2009)

SRLTB

LR

EJ

EFA/CFA/ICR

952

39

10

Initial unsatisfactory factor analysis output

Lucas-Carrasco et al. (2011)

QOCS

LR/FC

TPJ

EFA/CFA/CV/DV/ICR

3772

44

17

Recruitment of a larger number of interviewers

Mahudin et al. (2012)

Measuring rail passenger crowding

LR/ES

EJ/TPJ

EFA/CFA/CV/DV/ICR

525

9

20

Lack of the CtV/SRM

Medina-Pradas et al. (2011)

BDSEE

ES/EP

EJ

EFA/CV/CtV/ICR

77

14

14

SSS/CSM

Morean et al. (2012)

AEAS

LR/ES

EJ/TPJ

EFA/CFA/CV/CtV/DV/T-RR/ICR

546

40

22

LG/SRM/CSM

Morgado et al. (2014)

SAS-EB

LR/FC

EJ/TPJ

CFA/CV/DV/ICR

318

33

18

Lack of the use of a validated scale in the CV

Nagy et al. (2014)

Scale to measure liabilities and assets of newness after start-up

LR/I

EJ

EFA/CFA/DV/ICR

260

235

19

LG/SSS

Napoli et al. (2014)

Consumer-based brand authenticity scale

LR

EJ/TPJ

EFA/CFA/CV/DV/PV/ICR

762

157

14

Lack of a more robust LR

Negra and Mzoughi (2012)

OCPS

LR/I

EJ

EFA/CFA/CV/DV/NV/I-T-CR/ICR

512

77

5

Widely heterogeneous sample/Brevity of the scale.

Ngorsuraches et al. (2007)

TRUST-Ph

LR/FC/EP/ES

EJ

EFA/CV/CtV/ICR

400

40

30

LG/SSS/MD/social desirability bias

Oh (2005)

Affective reactions to print apparel advertisements scale

LR/FC/ES

TPJ

EFA/CFA/CV/DV/CtV/ICR

128

66

54

LG

Olaya et al. (2012)

ISAD

EP/ES

EJ

CV/DiV/T-RR/ICR

76

20

17

LG

Omar and Musa (2011)

LPSQual

LR/FC

EJ

EFA/CFA/CV/DV/NV/ICR

655

57

26

LG/Lack of the NV/CSM

Pan et al. (2013)

PMGS

I

EJ/TPJ

EFA/CFA/CV/S-RR/I-I-CR/I-T-CR/ICR

554

71

14

LG/SRM/Lack of the T-RR

Patwardhan and Balasubramanian (2011)

Measurement scale for brand romance

LR/ES/QER

TPJ

EFA/CFA/CV/DV/CtV/NV/ICR

711

70

12

LG

Pimentel et al. (2007)

EPM

NCR

EJ/TPJ

EFA/CFA/ICR

480

13

13

LG/Lack of the CV and DV

Pommer et al. (2013)

PCQ

EP/FC

TPJ

EFA/CFA/CV/ICR

953

391

18

CSM

Reed et al. (2011)

ESLS

ES

EJ

EFA/CFA/CV/DV/ICR

218

55

25

LG/SRM/WBS

Rice et al. (2013)

MDRS-22

LR

EJ

EFA/CFA/ICR

1176

82

22

LG/Lack of the T-RR/Lack of the CV

Riedel et al. (2011)

RSM-scale

LR/ES

EJ/TPJ

DV/T-RR/ICR

136

43

36

LG

Rodrigues and Bastos (2012)

Organizational Entrenchment Scale

EP/ES

EJ

EFA/CFA/I-T-CR/I-I-CR/ICR

721

31

22

LG

Rodríguez et al. (2013)

VEDAS

ES

NCR

EFA/CFA/CV/CtV/T-RR/ICR

1034

40

20

Long time between the test and retest/Lower Cronbach’s alpha

Rosenthal (2011)

IEK

EP

EJ

EFA/CV/CrV/I-T-CR/I-I-/T-RR/ICR

292

54

21

LG/SSS

Saxena et al. (2015)

UCLA Hoarding Severity Scale

LR/EP

EJ

EFA/CV/DV/I-I-CR/ICR

127

NCR

10

Lack of the T-RR/Lack of the instructions for raters in the initial version of the scale

Schlosser and McNaughton (2009) -

I-MARKOR scale

LR/FC/I

EJ/TPJ

EFA/CFA/CV/DV/NV/ICR

138

71

20

SSS/CSM.

Sewitch et al. (2003)

PPDS

LR

EJ

EFA/CrV/CV/ICR

200

10

10

LG/CrV was limited/content validity was not formally assessed

Sharma (2010)

Personal Cultural Orientations Scale

LR/I

EJ

EFA/CFA/NV/CV/PV/DV/ICR

2332

96

40

LG/Lack of the PV

Sharma and Gassenheimer (2009)

SPC Scale

LR/EP/I

EJ

EFA/CFA/CV/DV/ICR

511

8

17

Lack of the EV

Shawyer et al. (2007)

VAAS

LR

EJ

CV/T-RR/ICR

41

61

31

Lack of a more robust demonstration of the validity/SSS

Sin et al. (2005)

CRM

LR

EJ

EFA/CFA/CV/DV/NV/ICR

641

78

18

LG/CSM

Sohn and Choi (2014)

Expected Interactivity Scale

LR/EP/I

EJ

EFA/CFA/CV/DV/CtV/T-RR/ICR

378

50

12

Lack of the empirical test

Song et al. (2011)

SECI

LR

EJ

EFA/CFA/CV/I-I-CR/ICR

469

26

17

LG/deductive approach

Staines (2013)

Investigative Thinking Styles Scale

LR

TPJ

EFA/CV/CtV/ICR

545

68

16

LG

Sultan and Wong (2010)

Performance-based servisse quality model scale

LR/FC/ES

EJ

EFA/CFA/ICR

362

67

67

The study uses three sources to collect data

Swaid and Wigand (2009)

E-Service Quality Scale

LR

TPJ

EFA/CFA/CV/DV/ICR.

557

NCR

28

Online survey

Tanimura et al. (2011)

DDLKOS

LR/I

EJ

EFA/CFA/CtV/ICR

362

48

14

Inadequate choose variables to be correlated with that of the study

Taute and Sierra (2014)

Brand Tribalism Scale

LR/ES

NCR

EFA/CFA/CV/DV/ICR

616

35

16

LG

Tombaugh et al. (2011)

SEW

LR/EP

NCR

EFA/CFA/CV/DV/PV/ICR

348

5

5

CSM/Brevity of the scale

Turker (2009)

CSR

LR/FC/ES

TPJ

EFA/I-I-CR/I-T-CR/ICR

269

55

18

LG

Uzunboylu and Ozdamli (2011)

MLPS

LR/I/EP

EJ

EFA/S-RR/ICR

467

31

26

LG

Van der Gaag et al. (2013)

DACOBS

EP

EJ

EFA/CV/ICR/S-RR/T-RR

257

70

42

SSS/Validation performed with patients/Inappropriate choice of the instruments for validation

Von Steinbüchel et al. (2010)

QOLIBRI

LR/ES

EJ

EFA/CFA/T-RR/ICR

2449

148

37

SSS

Voon et al. (2014)

HospiSE

LR/FC

EJ/TPJ

EFA/CFA/CV/DV/CtV/ICR

1558

NCR

21

LG/CSM

Walshe et al. (2009)

DIP

LR/I/ES

TPJ

Ecological validity/ICR

31

48

48

SSS/Lack of the DV, CV and T-RR

Wang and Mowen (1997)

SC

LR

EJ

EFA/CFA/CV/DV/PV/I-T-CR/ICR

140

60

9

SSS

Wepener and Boshoff (2015)

The customer-based corporate reputation of large service organizations scale

LR/ES/FC

EJ

EFA/CFA/NV/CV/DV/ICR

2551

78

19

LG

Williams et al. (2009)

SCSC

LR/I

EJ/TPJ

EFA/CFA/CV/DV/PV/I-T-CR/ICR

162

5

5

LG; b) WBS.

Wilson and Holmvall (2013)

Incivility from customers scale

LR/FC

EJ

EFA/CFA/CV/DV/CtV/ICR

439

27

10

LG/CSM/SRM

Yang et al. (2014)

BLOG-S-INNO Scale

EP

TPJ

EFA/CFA/CV/DV/ICR

498

517

18

LG

Zhang and Hu (2011)

Farmer-buyer relationships in China Scale

LR/ES

EJ/TPJ

EFA/CFA/CV/I-I-CR/ICR

210

39

22

LG

Zheng et al. (2010)

DPEBBS

LR/FC

EJ

EFA/CFA/CtV/T-RR/I-T-CR/ICR

269

51

24

LG/SSS/EFA and CFA - same sample/Reliability coefficients - unsatisfactory.

  1. N sample size, EFA exploratory factor analysis, CFA confirmatory factor analysis, NV nomological validity, CV convergent validity, CrV concurrent validity, CtV criterion validity, DV discriminant validity, DiV divergent validiy, PV predictive validity, IV internal validity, EV external validity, ICR internal consistency reliability, S-RR split-half reliability, I-JR inter-judge reliability, I-T-CR item–total correlation reliability, I-I-CR inter-item correlation reliability, T-RR test-retest reliability, LR literature review, ES existing scales, I interview, FC Focus group, EP expert panel, QER qualitative exploratory research, NCR not clearly reported, EJ expert judges, TPJ target population judges, LG limitations of generalization, SSS small sample size, CSM cross-sectional methodology, SEM self-reporting methodology, WBS web-based survey, MD Missing data