Skip to main content

Advertisement

Psicologia: Reflexão e Crítica
Psicologia: Reflexão e Crítica Cover Image
We’d like to understand how you use our websites in order to improve them. Register your interest.
Download PDF

University student’s engagement: development of the University Student Engagement Inventory (USEI)

Psicologia: Reflexão e Crítica volume 29, Article number: 21 (2016) Cite this article

Abstract

Student engagement is a key factor in academic achievement and degree completion, though there is much debate about the operationalization and dimensionality of this construct. The goal of this paper is to describe the development of an psycho-educational oriented measure – the University Student Engagement Inventory (USEI). This measure draws on the conceptualization of engagement as a multidimensional construct, including cognitive, behavioural and emotional engagement. Participants were 609 Portuguese University students (67 % female) majoring in Social Sciences, Biological Sciences or Engineering and Exact Sciences. The content, construct and predictive validity, and reliability of the USEI were tested. The validated USEI was composed of 15 items, and supported the tri-factorial structure of student engagement. We documented evidence of adequate reliability, factorial, convergent and discriminant validities. USEI’s concurrent validity, with the Utrecht Work Engagement Scale-Student Survey, and the predictive validity for self-reported academic achievement and intention to dropout from school were also observed.

Background

In recent years, student engagement, broadly defined as the “students’ willingness to participate in routine school activities, such as attending classes, submitting required work, and following teachers’ directions in class” (Nystrand & Gamoran 1992, p. 14) has received increasing attention by researchers, practitioners, and policy makers (Christenson et al. 2012; Fredricks et al. 2004; Reschly & Christenson 2012; Salanova et al. 2010). The growing body of research, primarily conducted in the United States and Australia, shows that student engagement can act as an antidote to low academic achievement, student burnout, student lack of resilience, dissatisfaction, and school dropout (Christenson & Reschly 2010; Elmore & Huebner 2010; Finn & Zimmer 2012; Wang & Eccles 2012; Krause & Coates 2008). It can also act as a strong mediator between out-of-school variables (e.g., home and family, friends and class-inmates,) (Chen & Astor 2011), teacher-student interactions, academic achievement, school success and life-long learning (Christenson et al. 2012; Gilardi & Guglielmetti 2011; Reschly & Christenson 2012). Recent studies have found a correlation between different engagement profiles and students’ learning (Wang & Eccles 2012) and physical and psychological health and well-being (Li & Lerner 2011; Wang & Eccles 2012).

In Europe, the engagement construct has emerged mainly in professional and occupational activities, and has been researched primarily in relation to business organizations (Bakker et al. 2008; Hirschi 2012; Schaufeli & Bakker 2010). From the business organizations' perspective, work engagement has been defined as a positive, fulfilling, affective-motivational psychological state that is characterized by vigour, dedication and absorption associated with work-related well-being (Bakker et al. 2008). This conceptualization is supported by Maslach and Leiter’s (1997) initial definition of work engagement, which is the opposite of work burnout. Several different conceptualizations of burnout and its relations to work engagement coexist (Hirschi 2012). However, most professionals would agree that engagement is an independent and distinct construct that is negatively correlated with burnout, and that engaged employees have high levels of energy, low burnout and are strongly identified with their organization (González-Romá, Schaufeli, Bakker & Lloret 2006; Bakker et al. 2008; Demerouti et al. 2010). In this organizational perspective, work engagement is operationalized by three correlated constructs: vigour, dedication and absorption. According to Bakker et al. (2008), vigour is characterized by high levels of physical energy, ability to deal with the job’s challenges, and a willingness to invest the required effort to overcome difficulties. Dedication is characterized by a strong involvement in one’s work with a sense of personal realization and pride. Finally, absorption is defined as concentration and happiness in performing the job related tasks (Bakker et al. 2008; Hirschi 2012). Grounded in these definitions, Schaufeli and co-workers have developed the Utrecht Work Engagement Scale (UWES) to measure these three dimensions of work engagement (Schaufeli & Bakker 2003). Nowadays, the UWES is the most used instrument to measure engagement in the work place (Bakker et al. 2008). There have been attempts to expand this construct to non-workplace settings. For example, the UWES was adapted to measure school engagement in university settings, around its three dimensions: vigour, dedication and absorption (Salanova et al. 2010). The UWES-Student Survey has been used in a few studies (Bresó et al. 2011; Gilardi & Guglielmetti 2011). However, concerns have been raised about simply rephrasing items for the work-place to the university setting (Mills et al. 2012; Schaufeli & Bakker 2010), as well as its dimensionality in different age-groups of students (Upadaya & Salmela-Aro 2012).

Indeed, there is a growing consensus among researchers that engagement is a multidimensional construct with both behavioural, emotional and psychological components (see Finn & Zimmer 2012 for a review). However, there is still no clear consensus on the construct precise definition and its dimensionality (Christenson et al. 2012; Fredricks & McColskey, 2012; Reschly & Christenson 2012; Wolf-Wendel et al. 2009; Kahu 2013). For example, proposals for the construct dimensionality have ranged from two- (emotional and behavioural) (Skinner et al. 2009) to eight- (Learning strategies, Academic integration, Institutional emphasis, Co-curricular activity, Diverse interactions, Effort, Overall relationships and Workload) first order factors (LaNasa et al. 2009). Higher dimensional construct conceptualizations have been also proposed. For example, Martin (2007) documented 11 first-order factors with a second order 4-factor model of student engagement and motivation with Australian high school students. There is also large variation in how engagement has been measured (e.g. questionnaires, self-report measures, teacher ratings, interviews and observations) (Fredricks et al. 2011). In another study with first year Australian Universities’ undergraduates, seven ‘calibrated’ measures of engagement were described (Krause & Coates 2008). This variation in both the operationalization and measurement of the engagement construct has made it difficult to make comparisons between studies’ findings (Fredricks & McColskey 2012; Kahu 2013). Therefore, consensus on the different measuring instruments, their dimensionality and concurrent validity is an important area of future research (Reschly & Christenson 2012; Wolf-Wendel et al. 2009; Krause & Coates 2008; Kahu, 2013).

In this paper, we expand Nystrand and Gamoran (1992) broad definition of students’ engagement, by adding the Fredricks et al. (2004) conceptualization of engagement as a three factor construct including behavioural, emotional and cognitive dimensions. This is one approach, to measure students’ engagement, that tries to integrate the four dominant research perspectives on this important construct, namely, the behavioural perspective, the psychological perspective, the socio-cultural perspective and the holistic perspective (for a review and criticism of the different perspectives on student engagement see Kahu 2013). Cognitive engagement is defined as the students’ investment and willingness to exert the necessary efforts for the comprehension and mastering of complex ideas and difficult skills. The emotional engagement dimension reflects both the positive and negative reactions to teachers’ instructions, classmates and school, perceptions of school belonging, and beliefs about the value of schooling. Finally, behavioural engagement is defined in terms of student’s participation in classroom tasks, conduct, and participation in school-related extracurricular activities (see also Carter et al. 2012; Sheppard, 2011).

The majority of studies of student engagement have been conducted in the USA and Australia, with elementary, middle and high school students. One exception is the USA’s National Survey of Student Engagement (NSSE) (NSSE 2012) and the Beginning College Survey of Student Engagement (BCSSE) (BCSSE 2013), which measures engagement in first-year US college students (Chambers & Chiang 2011; Kuh 2009; McCormick & McClenney 2012). However, the NSSE has been strongly criticized. Some scholars emphasize that the NSSE lacks good psychometric properties (namely construct and predictive validities and overall reliability) (Campbell & Cabrera 2011; LaNasa et al. 2009), and that it does not directly measure the “student engagement” psychological construct, but rather students’ studying habits, gains from their college experiences, and other aspects of student life (Wefald & Downey 2009). Although most of these criticisms on the NSSE have been addressed (see McCormick & McClenney 2012), there are still unanswered questions regarding the psychological “student engagement” construct definition and its dimensionality in different schooling contexts, student minority groups and family and social contexts where schools are immersed. Another example, is the seven ‘calibrated’ scales of first-year undergraduates’ engagement in Australian Universities (Krause & Coates 2008). However, this Australian study is not conclusive regarding the construct dimensionality. On the contrary, it points out to the imperative necessity for developing a broader understanding of engagement as a construct with several dimensions (Krause & Coates 2008). Accordingly to these scholars, the multidimensional feature of students’ engagement must be acknowledged in any measurement and monitoring of this construct in higher education.

Interventions aimed at improving university students’ academic achievement, reduce burnout improve student well-being, engagement and graduation success have been pursued in several high schools and universities all over the world (see e.g., Bresó et al. 2011; Chen & Astor 2011; Christenson & Reschly 2010; Elmore & Huebner 2010; Fredricks et al. 2004; Harlow et al. 2011; Kuh 2009; Li & Lerner 2011). Increasing engagement among university students has become crucial to improving their learning experiences, well-being and return in the investment in higher education (Bresó et al. 2011; Christenson & Reschly 2010; Kuh 2009; Salmela-Aro et al. 2009). Therefore, data with good psychometric properties on university students’ engagement and its correlates with academic achievement, course-work fulfilment, graduation and school integration are fundamental for educational psychologists, school counsellors as well as education policy-makers.

To the best of our knowledge, with the exception of the UWES-SS developed in Europe, the Beginning College Survey of Student Engagement (BCSSE) (BCSSE 2013) in the USA and seven ‘calibrated’ scales of first year undergraduate students in Australian Universities (Krause & Coates 2008) both of which have suffered from several criticisms, there are no psychometrically valid and publically available measures of student engagement in university settings. The UWES-SS mimics the professional work-place related engagement and only some of its items comply with studying and university activities (e.g., vigour). Thus, its use in the university context has suffered some criticisms, either due to methodological limitations in items’ construction or non-adaptation to the university context (Mills et al. 2012). On the other hand, surveys developed for high school students (see e.g. Fredricks & McColskey 2012) leave out some key facets of students’ engagement that are important at the university level (e.g., attendance at conferences and seminars). Also, the engagement construct’s dimensionality may vary with different students’ ages (Upadaya & Salmela-Aro 2012). Therefore, surveys developed for younger students may lack the appropriate construct definition and dimensionality, as well as adequate psychometric properties for evaluating student engagement in university students.

The present study draws from the earlier work of Schaufeli et al. (2002) on the UWES-SS, the work by Fredricks et al. (2004) with upper elementary school students in an effort to integrate the 4 perspectives on student engagement described by Kahu (2013) into a single measure instrument. In this study, we developed a new measure, – the University Student Engagement Inventory (USEI) and describe its psychometric properties in a sample of Portuguese college students. We examine content, construct and criterion related validities, and measurement invariance in two independent samples of students, from public and private universities and several study areas. Concurrent validity with the UWES-SS was assessed, as well as predictive validity for self-reported academic achievement and intention to dropout. We hypothesize that ‘university student engagement’, conceptualized as the students’ involvement and motivation towards his or her course work and academic activities within the university, is a second-order construct that is reflected in 3 first-order emotional, behavioural and cognitive dimensions expanding the Nystrand and Gamoran (1992); Fredricks et al. (2004) and Kahu (2013) conceptualizations. We anticipate that this construct is predictive of students’ academic achievement and intention to drop out of school. This new measure, focused on behaviors, cognitions, emotions and actions, that is, on the behavioral, psychological and holistic aspects of engagement can be used both for student counselling as well as research aimed at educational policy and good practices development to improve academic success and school retention.

Methods

Participants

A total of 609 university students volunteered to participate in this study. Sixty-seven percent of participants were female. Students’ mean age was 23.6 years (SD = 7.5) the 1st quartile was19 years; The median was 21 years and the 3rd quartile was 24 years. Students were enrolled in 6 or 10-semester degree courses (BSc or MSc, respectively). Forty one percent of students were in their 2nd semester, 23 % were enrolled in the 6th semester and 17 % were enrolled in the 4th semester. Enrolment in the MSc 8th and 10th semesters was 8 % and 7 % respectively (the % of students enrolled in odd numbered semesters was marginal, since the data collection occurred in the Spring Semester). Seventy percent of the students were enrolled in public universities; the remaining 30 % were attending private universities. Participating students were enrolled in human and social sciences courses (51 %) health and biological sciences (26 %), and engineering and exact sciences (23 %). The large majority of students lived with their parents (75 %); the remaining lived with friends and colleagues (17 %) or by themselves (8 %). Furthermore, the majority of students had their studies sponsored by their families (65 %) with approximate distributions for scholarships (15 %) and own funds (20 %).

Questionnaire development

A questionnaire composed of socio-demographic questions (age, gender, semester, type of school, area of studies, financing and housing), the number of courses taken, self-perceived academic achievement (on a 5-point scale ranging from ‘1-not good at all’ to ‘5-very good’) and intention to dropout (on 3-point rating scale from ‘1-never’ to ‘3-always’) was constructed.

The initial USEI form was composed of 32 items (see Table 1 for the items’ description) rated on a ‘1-never’ to ‘5-always’ response scale. The initial USEI form included the 15-item School Engagement questionnaire, developed by Fredricks and her colleagues (2004) for upper elementary school students, which was adapted to the university context, after obtaining the authors authorization. Additionally, 17 new items were created from a focus group analysis.

Table 1 Minimum (Min), Maximum (Max), Median (Me), Mean (M), Skewness (Sk), Kurtosis (Ku) and Content Validity Ratio (CVR) for the original 32 items tested (SE Sk  = 0.138; SE Ku  = 0.275). In between parenthesis are the original Portuguese language items
Full size table

The focus group included 10 university students from the social, health and engineering sciences, who volunteered to participate in the focus groups session held at a private university. These students were interviewed in an open discussion forum during a two-hour period. In this forum, students were presented with a brief and broad definition of student engagement as being the personal and emotional involvement of the student with his or her coursework, classroom and academic activities and the willingness to follow teachers’ instructions within the university settings. Thereafter, the students were asked to answer and or discuss three topics/questions posed by the first author of this manuscript. The three questions were “1. How would you define a student engaged in his/her course work and university?”; “2. What daily practices do you use to succeed in your course work requirements?” and “3. What kind of academic activities do you practice or would like to practice outside of the classroom that are related to your university experience?” This group was composed by 6 female and 4 male students, with a mean age of 21 years (in the 5th semester of their courses). Main ideas from the students’ answers and open discussion that followed were recorded. From these notes, 17 new statements/items related to the university context and activities were created and distributed, accordingly to their content, to one of the 3 dimensions: emotional, cognitive and behavioural engagement.

Items with good content related validity and good psychometric properties (see below) in the test sample (n = 313) were retained for the final format of the 15-items USEI. This form was then applied to a second, independent, validation sample (n = 296) simultaneously with the UWES-SS (Schaufeli et al. 2002). The UWES-SS is composed of 15 ordinal rating items scored from ‘0-never’ to ‘6-always’ that define three factors as described in the introduction. A Portuguese, short-form version with 9 items provided and validated by the UWES-SS authors was used for evidence of concurrent validity studies (for details on the UWES-SS see Schaufeli et al. 2002).

Procedure

Participant recruitment was done through students’ syndicates and or academic associations, which invited their members/associates by email. Each participant agreed on an informed consent stating that the participation was voluntary, that no personal information that could be used to identify the participants would be asked, that they could drop the research at any time and that the study was purely academic with no intention to diagnose or to treat. Participants were also assured that the data would be presented only in aggregated statistical analysis. No incentive to participation was given. The data was gathered through an online questionnaire composed of sample characterization questions and the USEI and UWES-SS’ items available online during the 2nd semester of the 2012–2013 school year (February to May 2013). The ISPA/UIPES Ethics committee approved the proposed research study.

Questionnaire evaluation

Psychometric properties

To examine psychometric properties, we gathered two independent samples: an initial test sample (n = 313) where sensitivity and confirmatory factor analysis were performed and a second validation sample (n = 296) to test evidence for sampling measurement invariance and concurrent validity with the UWES-SS. Both samples were equivalent by mean age and semester as well as by gender, areas of study, and public versus private university distributions. Minimum sample size requirements’ was determined for 32 manifest variables and 3 latent factors with an a priori estimated effect size of 0.1, statistical power of 0.8 and type I error rate of 0.05, using the formula provided by Westland (2010).

Items’ distributional properties

Summary measures and skewness (sk) and kurtosis (ku) for each of the 32 original items estimated with IBM SPSS Statistics (v. 20, SPSS An IBM company, Chicago, IL) were used to judge distributional properties and psychometric sensitivity. Absolute values of sk smaller than 3 and ku smaller than 7 were considered indicative of no strong deviations from the normal distribution (Finney & DiStefano 2006; Kline 2005). These items’ distributional properties are indicative of appropriate psychometric sensitivity as it would be expected that these items would follow an approximate normal distribution in the population under study.

Evidence of content related validity

The content related validity of the USEI’s 32 original items was evaluated by a pool of 20 educational psychologists, as proposed by Lawshe (1975). The psychologists rated each item on a 3-point rating scale: “Essential”, “Useful, but not essential” and “Not necessary”. The Lawshe’s content validity ratio (CVR) was calculated for each item and averaged for the total scale. A CVR of 0 or greater indicates that at least half of the judges deemed the item as “Essential” for the construct assessment (Lawshe 1975).

Evidence of construct related validity

We tested the factorial, convergent and discriminant related validities of the cognitive, emotional and behavioural dimensions theorized for university students’ engagement following Fornell & Larcker (1981) theoretical framework (see below). To evaluate the USEI’s 3-factor model evidence for factorial validity, a confirmatory factor analysis (CFA) was performed on the items’ polychoric correlation matrix using the WLMSV estimator as implemented in Mplus (v. 7, Muthén & Muthén, Los Angeles, CA). The goodness-of-fit of the 3-factor engagement structure was assessed with usual CFA indexes: Chi-square over degrees of freedom (χ2/df), Comparative fit index (CFI), Tucker-Lewis fit index (TLI) and Root mean square error of approximation (RMSEA). The fit was considered good for χ2/df around 2, CFI and TLI larger than .95 and RMSEA smaller than .08 (see e.g. Byrne 2012; Maroco 2014). Individual standardized factor loadings (λ) and modification indices (MI) for each of the original items were evaluated for item reduction, supported on theoretical considerations regarding items’ content, CVR and factor manifestations. Items with λ < .5, CVR < .0 and or MI > 11 (Maroco 2014) were removed from the original 32 items pool. Convergent and discriminant related validity for the 3 factors was evaluated as proposed by Fornell and Larcker (1981). Average variance extracted (AVE) by each factor larger than .5 was considered indicative of convergent validity while squared correlations between every two factors smaller than each of the factors’ AVE was indicative of discriminant validity (Fornell & Larcker 1981; Maroco 2014). Considering the theoretical definition of student engagement, supported by moderate correlations between factors, a second-order factor model, with student engagement reflecting on emotional, cognitive and behavioural engagement dimensions was also tested by CFA as described above.

Evidence of strong measurement invariance for the 15-best items retained form the analysis on the test sample was evaluated using multi-group analysis by imposing increasing constrains for the factor weights, mean intercepts and factor covariances for the test and validation samples (Byrne 2012; Maroco 2014). A Δχ2 test, corrected for WLMSV estimation as proposed by Muthén & Muthén (2012) Mplus’ v.7 user guide, was used to evaluate the significance of the differences between the constrained models and the free estimates models in both samples. Strong measurement invariance was assumed when the Δχ2 for the factor weights, mean intercepts and factor covariances was not statistically significant (p > .05).

Reliability

Reliability for the 3 factors of student engagement was evaluated by the factors’ internal consistency evaluated both by the Composite Reliability (CR) (Fornell & Larcker 1981) and Cronbach’s alpha calculated on the items’ Polychoric correlations matrix (αP) (Gadermann et al. 2012). Values of CR and αP larger than .7 were indicative of acceptable internal consistency for exploratory research (Gadermann et al. 2012; Maroco & Garcia-Marques 2006).

Evidence for Criterion related validity

The evidence for concurrent criterion related validity was evaluated by correlational analysis between the mean score estimates for the 3 factors of USEI and the 3 factors of the UWES-SS (Schaufeli et al. 2002) on the validation sample (n = 296). On a previous step, evidence for construct validity was demonstrated for the UWES-SS short form (composed by 9 items) (χ2/df = 4.32; CFI = .97; TLI = .95; GFI = .96; RMSEA = .08). Evidence of predictive criterion validity of the USEI total mean score on self-perceived academic achievement, failing courses and intention to drop out of school was evaluated by ordinal logit regression using Mplus (v. 7; Muthén & Muthén, Los Angeles, CA).

Results

Items distributional properties

Descriptive Statistics and CVR for the original 32 items are given in Table 1. Nineteen of the 32 items had a median value of 4. The overall mean response for the 32 items was 3.5 (SD = 0.75). No item showed Sk and Ku values that were suggestive of a severe deviation from the Normal distribution (all absolute values of Sk and Ku were lower than 2).

Evidence for content related validity

Evaluation of the relevance of the items by a panel of 20 educational psychologists indicated that 27 of the 32 initial items were “essential” to evaluate university’s student engagement by at least half of the specialists (CVR > 0) (see Table 1). These items were then subjected to a confirmatory factor analysis for the assessment of construct related validity.

Evidence for construct related validity

From the CFA of the 27 original items with content related validity, a final set of 15 items, which showed factor loadings greater than .5 in each of the 3 dimensions tested and no factors’ cross-loadings as suggested by Modification Indices analysis (MI >11; p < .001) were retained (See Table 2). Factorial related validity was observed for the 15-item Questionnaire distributed by the theoretical 3-factor model of Student Engagement (χ2/df = 2.26; CFI = .97; TLI = .97; RMSEA = .06). Convergent related validity, as defined by Fornell and Larcker (1981) could not be assumed only for the Behavioural factor. On the other hand, both Emotional Engagement as well as the Cognitive Engagement had AVE and CR larger than .5 and .7 respectively (see Table 2).

Table 2 Standardized factor loadings (λ) for the retained 15 items with CVR > 0 and λ > .5, AVE, CR and CVR for each of the 3 engagement dimensions (Test sample n = 313). In between parenthesis are the original language items
Full size table

Squared correlations between Behavioural vs. Emotional and Cognitive engagements were .39 and .41, respectively. Squared correlation between Emotional vs. Cognitive engagement was .17 (see Table 3 for the correlations between dimensions). Therefore, discriminant validity was observed between Emotional and Cognitive engagement and between Behavioural and Emotional engagement. The retained factors had an average CVR of .62 for the Behavioural dimension, .32 for the Emotional dimension and .24 for the Cognitive dimension.

Table 3 Correlations (r) between first-order behavioural, emotional and cognitive engagement and standardized structural weights (β) from USEI to Behavioural (BE), Emotional (EE) and Cognitive Engagement (CE), AVE, CR and overall CVR on the overall combined test and validation samples (n = 609)
Full size table

Evidence for measurement invariance

The multigroup analysis comparing the tri-factor engagement dimensions in the test sample (n = 313) and in a new, independent, validity sample (n = 296) with similar socio-demographic characteristics revealed both weak (equal factor weights: Δχ2(15) = 13.9; p = .588), strong (equal intercepts: Δχ2(15) = 18.3; p = .247) and structural (equal covariances: Δχ2(3) = 3.7; p = .296) invariance. Thus, the USEI showed strong measurement invariance in two independent samples of university students.

Engagement as a second order factor

Giving the theoretical considerations that student engagement is a second order factor that is reflected in three behavioural, emotional and cognitive first order factors, and given the moderate correlations between these first order factors, we tested a second order factor model where student engagement as a 2nd order factor. Since the 3-factor measurement model of engagement showed strong measurement invariance in the test and validation sample, the second order factor structure was tested on the combined sample. Table 3 gives the standardized factor loadings for the second order (β) student engagement construct.

University student engagement reflects mostly on the behavioural engagement (β = .95; p < .001), but also has strong impact on cognitive (β = .73; p < .001) and emotional engagement (β = .66; p < .001). This second order construct, constituted by 15 items with an average CVR of .39, explained 62 % of variability of the 3 first order engagement dimensions with strong reliability (CR = .83) and good overall goodness-of-fit to the items’ variance-covariance data (χ2/df = 2.78; CFI = .978; TLI = .974; RMSEA = .054).

Evidence for criterion related validity

The concurrent validity of the proposed USEI with the UWES-SS, was evaluated by a correlational analysis done in the validation sample (n = 296). Table 4 shows the correlations between the USEI and the UWES-SS short form.

Table 4 Correlations between the USEI dimensions and its total and the UWES-SS dimensions and its total in the validation sample
Full size table

A strong and positive correlation was found between the USEI total and the UWES-SS short form total (r = .99; p < .001) demonstrating the concurrent validity of the USEI when compared to the UWES-SS. Moderate correlations were found between the 3 dimensions of USEI and the 3 dimensions of UWES.

The ordinal logit regression of self-perceived academic achievement, failing courses and intention to drop out of school on the USEI total score showed positive predictive validity over self-perceived academic achievement (Β = 1.024; SE = 0.125; p < .001, OR = 2.79). Furthermore, it presented a negative predictive ability both on failing courses (Β = -0.649; SE = 0.111; p < .001; OR = 0.52) and intention to drop out of school (Β = -0.543; SE = 0.102; p < .001; OR = 0.58).

Reliability

Ordinal Cronbach’s α was .74 for the Behavioural Engagement, .88 for Emotional engagement and .82 for Cognitive engagement. These values were similar to the ones obtained for the composite reliability (see Table 3). Ordinal Cronbach’s α for the USEI’s 15 items total was .88. Altogether, these values show the overall good reliability of the three USEI dimensions and its total.

Discussion

A growing body of research has shown that student engagement is predictive of students’ well-being, satisfaction with school and peers, academic achievement, and lower burnout and school dropout rates (see e.g. Chen & Astor 2011; Finn & Zimmer 2012; Kuh 2009; Salmela-Aro et al. 2008). In studies with University students, engagement has been measured primarily with the UWES-SS in Europe and the BCSSE in the USA. However, these two instruments have suffered several criticisms ranging from the construct definitions and dimensionality to its application to university students (Campbell & Cabrera 2011; LaNasa et al. 2009; Mills et al. 2012; Wefald & Downey 2009). However, there is still a large debate on the definition (Finn & Zimmer 2012; Fredricks et al. 2011; Schaufeli & Bakker 2010; Schaufeli et al. 2002) and dimensionality (Fredricks & McColskey 2012; LaNasa et al. 2009) of student engagement construct and the need for validated measures is necessary for research in psychology and education in the area of student motivation, burnout and academic success.

In this study, we aimed to evaluate student engagement at the university level, using the expanded three dimensions (behavioural, emotional, and cognitive engagement) previously addressed with students from elementary and high schools in the US. From the different construct dimensionalities observed in the literature (see Fredricks et al. 2011; Kahu 2013; Krause & Coates 2008), our proposed USEI was built upon a scale initially developed by Fredricks and her colleagues for use with elementary students, and which has been also used with older students. Since contextual factors specifically related to the university context have been left out from these previous adaptations, a focus group analysis was conducted to identify specific university activities that could impact on student engagement and a set of new items were developed. Furthermore, there is evidence that the construct dimensionality can be also age-related (Upadaya & Salmela-Aro 2012). Thus, reported differences in the constructs dimensionality may be not only due to school and family related contexts, but also to students’ developmental processes.

Our study adds to the definition of university students’ engagement from an educational perspective in opposition it with the organizational perspective proposed by Schaufeli and colleagues, currently, the most used in Europe (Bresó et al. 2011; Salanova et al. 2010; Schaufeli et al. 2002). The Schaufeli et al. inventory has been criticized for its’ methodological approach (Mills et al. 2012), and difficulties in applying the organizational dimensions to the educational context (e.g., absorption and vigour), (Upadaya & Salmela-Aro 2012; Wefald & Downey 2009). Our proposed inventory addresses these criticisms.

Psychometric data analyses support the behavioural, cognitive and emotional tri-factorial conceptualization of student engagement. Evidence for strong measurement and structural invariance for this 3-factor model was demonstrated in two independent samples of Portuguese university students. Moderate to strong correlations were observed between the three factors, allowing for an empirically supported second-order engagement construct that is reflected on first order behavioural, emotional and cognitive dimensions. This second-order model showed a good fit to university students from both public and private universities majoring in social and human sciences, biological sciences and exact sciences.

Furthermore, the USEI produced reliable estimates of engagement and showed evidence for concurrent validity with the UWES-SS. Although both scales assess overall student engagement, the moderate correlations between their dimensions suggest that different facets of student engagement are being assessed by USEI and UWES-SS as an item’s content analysis would support. The USEI also showed evidence for predictive validity over a set of indicators of student academic achievement, courses failed and intention to dropout. For each extra point in the engagement scale the odds of improving academic achievement, as self-perceived by the student, increases significantly almost 3 times while the chances of failing one or more courses or dropping out of school decreases 2 times.

Our results show that data collected from a new instrument of student engagement has predictive value for important academic variables and can be used to further develop research and psycho-educational interventions on student engagement. Further research should be aimed at developing student engagement as a mediator to improve student achievement, reduce academic failure, promote psychological well-being and reduce university dropout rates.

However, this study has some limitations that need to be pointed out. First, the data of this study is only for a Portuguese population. Therefore, other cross-cultural studies are necessary before the USEI can be used more broadly in research and interventions aimed at improving student engagement. Second, a limitation of this instrument for a U.S. college student population is that it has no items applying to residential life which is a very important aspect of the college experience in the U.S for first year students. Thus the USEI can be used for students living out-of-campus, but may miss an important dimension for the other students’ academic experience. It also did not escape our attention that some items targeted initially to evaluate facets of behavioral engagement (e.g., items encompassing both classroom tasks and school-related extracurricular activities) did not make it to the final 15-item format, either because the experts did not judge these items as fundamental and/or either because they showed weak factor loadings in their proposed factors.

The results observed with the USEI also point to some new research direction to which the USEI maybe useful. For example, when is a student’s engagement at risk during his/her course works? In the earlier semesters or near the end of the academic work? Can the USEI be used as a predictor of student drop-out and or failing courses? Can a relationship between student academic performance, physical and psychological well-being and the students’ engagement in his/her course work be demonstrated? Can psychoeducational interventions aimed at improving students’ engagement have beneficial outcomes for the students’ proficiency and well-being? Can the USEI be used to produce valid and reliable measures of engagement during interventions? These are just a few research questions for an area where much research, supported on valid and reliable data, is needed.

Conclusions

The validated USEI was composed of 15 items, supports the tri-factorial structure of student engagement. We documented evidence of adequate reliability, factorial, convergent and discriminant validities in a sample of Portuguese college students. USEI’s concurrent validity, with the Utrecht Work Engagement Scale-Student Survey, and the predictive validity for self-reported academic achievement and intention to dropout from school were also observed.

References

  1. Bakker AB, Schaufeli WB, Leiter MP, Taris TW. Work engagement: an emerging concept in occupational health psychology. Work Stress. 2008;22(3):187–200. doi:10.1080/02678370802393649.

  2. BCSSE. (2013). Beggining college survey of student engagement. Retrieved 30 Jul 2013, 2013, from http://bcsse.indiana.edu/_/?cid=15.

  3. Bresó E, Schaufeli WB, Salanova M. Can a self-efficacy-based intervention decrease burnout, increase engagement, and enhance performance? A quasi-experimental study. Higher Educ. 2011;61:339–55. doi:10.1007/s10734-010-9334-6.

  4. Byrne BM. Structural equation modeling with Mplus: basic concepts, applications, and programming. New York: Taylor and Francis Group; 2012.

  5. Campbell CM, Cabrera AF. How sound is NSSE?: Investigating the psychometric properties of NSSE at a public, research-extensive institution. Rev Higher Educ. 2011;33(1):77–103.

  6. Carter CP, Reschly AL, Lovelace MD, Appleton JJ, Thompson D. Measuring student engagement among elementary students: pilot of the student engagement instrument—elementary version. School Psychol Quart. 2012;27(2):61–73. doi:10.1037/a0029229.

  7. Chambers T, Chiang C-H. Understanding undergraduate students’ experience: a content analysis using NSSE open-ended comments as an example. Qual Quant. 2011;46(4):1113–23. doi:10.1007/s11135-011-9549-3.

  8. Chen J-K, Astor RA. School engagement, risky peers, and student-teacher relationships as mediators of school violence in Taiwanese vocational versus academically oriented high schools. J Community Psychol. 2011;39(1):10–30. doi:10.1002/jcop.20413.

  9. Christenson SL, Reschly AL. Check & connect: enhancing school completion through student engagement. In: Doll B, Pfohl W, Yoon J, editors. Handbook of youth prevention science. New York: Routledge; 2010. p. 327–34.

  10. Christenson SL, Reschly AL, Wylie C. Handbook of research on student engagement. New York: Springer Science + Business Media; 2012.

  11. Demerouti E, Mostert K, Bakker AB. Burnout and work engagement: a thorough investigation of the independency of both constructs. J Occup Health Psychol. 2010;15(3):209–22. doi:10.1037/a0019408.

  12. Elmore GM, Huebner ES. Adolescents’ satisfaction with school experiences: relationships with demographics, attachment relationships, and school engagement behavior. Psychol Schools. 2010;47(6):525–37. doi:10.1002/pits.20488.

  13. Finn JD, Zimmer KS. Student engagement: What is it? Why does it matter? In: Christenson SL, Reschly AL, Wylie C, editors. Handbook of research on student engagement. New York: Springer Science + Business Media; 2012. p. 97–131.

  14. Finney SJ, DiStefano C. Non-normal and categorical data in structural equation modelling. In: Hancock GR, Mueller RO, editors. Structural equation modeling: a second course. Greenwich: IAP; 2006. p. 269–314.

  15. Fornell C, Larcker DF. Evaluating SEM with unobserved variables and measurment error. J Market Res. 1981;18:39–50.

  16. Fredricks J, McColskey W. The measurement of student engagement: a comparative analysis of various methods and student self-report instruments. In: Christenson SL, Reschly AL, Wylie C, editors. Handbook of research on student engagement. US: Springer; 2012. p. 763–82.

  17. Fredricks JA, Blumenfeld P, Friedel J, Paris A. School engagement. In: Moore KA, Lippman LH, editors. What do children need to flourish: conceptualizing and measuring indicators of positive development. New York: Springer Science + Business Media; 2004. p. 305–21.

  18. Fredricks JA, McColskey W, Meli J, Mordica J, Montrosse B, Mooney K. Measuring student engagement in upper elementary through high school: a description of 21 instruments (Vol. 98): SERVE Center at the University of North Carolina at Greensboro. 2011.

  19. Gadermann AM, Guhn M, Zumbo BD. Estimating ordinal reliability for Likert-type and ordinal item response data: a conceptual, empirical, and practical guide. Prat Assess Res Evaluat. 2012;17(3):1–13.

  20. Gilardi S, Guglielmetti C. University life of Non-traditional students: engagement styles and impact on attrition. J Higher Educ. 2011;82(1):19.

  21. González-Romá V, Schaufeli WB, Bakker AB, Lloret S. Burnout and work engagement: independent factors or opposite poles? J Voc Behav. 2006;68(1):165–74. doi:10.1016/j.jvb.2005.01.003.

  22. Harlow L, DeBacker T, Crowson HM. Need for closure, achievement goals, and cognitive engagement in high school students. J Educ Res. 2011;104(2):110–9. doi:10.1080/00220670903567406.

  23. Hirschi A. Callings and work engagement: moderated mediation model of work meaningfulness, occupational identity, and occupational self-efficacy. J Couns Psychol. 2012;59(3):479–85. doi:10.1037/a0028949.

  24. Kahu ER. Framing student engagement in higher education. Stud Higher Educ. 2013;38(5):758–73. doi:10.1080/03075079.2011.598505.

  25. Kline RB. Principles and practice of structural equation modeling. 2nd ed. New York: Guilford Press; 2005.

  26. Krause KL, Coates H. Students’ engagement in first-year university. Assess Evaluat Higher Educ. 2008;33(5):493–505. http://dx.doi.org/10.1080/02602930701698892.

  27. Kuh GD. What student affairs professionals need to know about student engagement. J College Student Dev. 2009;50:683–706.

  28. LaNasa SM, Cabrera AF, Trangsrud H. The construct validity of student engagement: a confirmatory factor analysis approach. Res Higher Educ. 2009;4:315. doi:10.2307/29782925.

  29. Lawshe CH. A quantitative approach to content validity. Personnel Psychol. 1975;28:563–75.

  30. Li Y, Lerner RM. Developmental trajectories of school engagement across adolescence: implications for academic achievement, substance use, depression, and delinquency. Dev Psychol. 2011;42:233–47. doi:10.1037/a0021307.

  31. Maroco J. Análise de Equações Estruturais: Fundamentos teóricos, software & Aplicações [Structural Equations Analysis: Theorethical foundations, software and software]. 2ªth ed. Pero Pinheiro: ReportNumber; 2014.

  32. Maroco J, Garcia-Marques T. Qual a fiabilidade do alfa de Cronbach? Questões antigas e soluções modernas? Laboratório de Psicologia. 2006;4(1):65–90.

  33. Martin, AJ. Examining a multidimensional model of student motivation and engagement using a construct validation approach. Br J Educ Psychol. 2007;77(2):413–40. doi:10.1348/000709906X118036.

  34. Maslach C, Leiter MP. The truth about burnout. San Francisco: Jossey-Bass; 1997.

  35. McCormick AC, McClenney K. Will these trees “ever” bear fruit? A response to the special issue on student engagement. Rev Higher Educ. 2012;35(2):307–33.

  36. Mills MJ, Culbertson SS, Fullagar CJ. Conceptualizing and measuring engagement: an analysis of the Utrecht work engagement scale. J Happiness Stud. 2012;13(3):519–45. doi:10.1007/s10902-011-9277-3.

  37. Muthén LK, Muthén BO. Mplus user’s guide. 7th ed. Los Angeles: Muthén & Muthén; 2012.

  38. NSSE. National survey of student engagement: promoting student learning and institutional improvement: lessons from NSSE at 13. Bloomington: Indiana University Center for Postsecondary Research; 2012.

  39. Nystrand M, Gamoran A. Instructional discourse and student engagement. In: Schunk DH, Meece J, editors. Student perceptions in the classroom. Hillsdale: Lawrence Erlbaum; 1992. p. 149–79.

  40. Reschly AL, Christenson SL. Jingle, jangle, and conceptual haziness: evolution and future directions of the engagement construct. In: Christenson SL, Reschly AL, Wylie C, editors. Handbook of research on student engagement. New York: Springer Science + Business Media; 2012. p. 3–19.

  41. Salanova M, Schaufeli WB, Martinez I, Breso E. How obstacles and facilitators predict academic performance: the mediating role of study burnout and engagement. Anxiety Stress Coping. 2010;23(1):53–70. doi:10.1080/10615800802609965.

  42. Salmela-Aro K, Kiuru N, Pietikäinen M, Jokela J. Does school matter? The role of school context in adolescents’ school-related burnout. Eur Psychol. 2008;13(1):12–23. doi:10.1027/1016-9040.13.1.12.

  43. Salmela-Aro K, Tolvanen A, Nurmi J-E. Achievement strategies during university studies predict early career burnout and engagement. J Voc Behav. 2009;75(2):162–72. doi:10.1016/j.jvb.2009.03.009.

  44. Schaufeli WB, Bakker AB. Ultrech work engagement scale, [Preliminary Manual]. Utrecht University. 2003.

  45. Schaufeli WB, Bakker AB. The conceptualization and measurement of work engagement: a review. In: Leiter ABBMP, editor. Work engagement: a handbook of essential theory and research. New York: Psychology Press; 2010.

  46. Schaufeli WB, Martínez IM, Marques Pinto A, Salanova M, Bakker AB. Burnout and engagement in university students: a cross-national study. J Cross Cultur Psychol. 2002;33(5):464–81. doi:10.1177/0022022102033005003.

  47. Sheppard SL. School engagement: a ‘Danse Macabre’? J Philo Educ. 2011;45(1), 2011, 45(1):111–23.

  48. Skinner EA, Kindermann TA, Furrer CJ. A motivational perspective on engagement and disaffection: conceptualization and assessment of Children’s behavioral and emotional participation in academic activities in the classroom. Educ Psychol Meas. 2009;69(3):493–525.

  49. Upadaya K, Salmela-Aro K. The schoolwork engagement inventory. Eur J Psychol Assess. 2012;28(1):60–7. doi:10.1027/1015-5759/a000091.

  50. Wang M-T, Eccles JS. Social support matters: longitudinal effects of social support on three dimensions of school engagement from middle to high school. Child Dev. 2012;83(3):877–95.

  51. Wefald AJ, Downey RG. Construct dimensionality of engagement and its relation with satisfaction. J Psychol Interdisciplin Appl. 2009;143(1):91–111. doi:10.3200/JRLP.143.1.91-112.

  52. Westland CJ. Lower bounds on sample size in structural equation modeling. Electron Commerce Res Appl. 2010;9(6):476–87.

  53. Wolf-Wendel L, Ward K, Kinzie J. A tangled web of terms: the overlap and unique contribution of involvement, engagement, and integration to understanding college student success. J College Student Dev. 2009;50(4):407–28. doi:10.1353/csd.0.0077.

Download references

Author information

Affiliations

  1. Psychological Sciences Department and William James Center for Research, ISPA - Instituto Universitário, Rua Jardim do Tabaco, 34, 1149-041, Lisboa, Portugal
    • João Maroco
  2. Statistics Department, Escola Superior de Comunicação Social, IPL-Lisboa, Lisboa, Portugal
    • Ana Lúcia Maroco
  3. Universidade Estadual Paulista – UNESP, Araraquara, SP, Brazil
    • Juliana Alvares Duarte Bonini Campos
  4. Human Development Department, Connecticut College, New London, USA
    • Jennifer A. Fredricks
Authors
  1. João Maroco
    View author publications
    You can also search for this author in
  2. Ana Lúcia Maroco
    View author publications
    You can also search for this author in
  3. Juliana Alvares Duarte Bonini Campos
    View author publications
    You can also search for this author in
  4. Jennifer A. Fredricks
    View author publications
    You can also search for this author in

Corresponding author

Correspondence to João Maroco.

Additional information

Competing interests

The authors declare that they have no competing interests. The “Comissão de Ética da Unidade de Investigação em Psicologia e Saúde, ISPA-IU” ethical committee approved this study.

Authors’ contributions

JM has setup the conceptual research design, collected most of the data, did most of the data analysis and interpretation and wrote the first draft of the manuscript; ALM collected some data, collaborated on the data analysis and in the writing and revision of the manuscript; JADBC contributed to the data analysis and in the writing of the manuscript; JAF collaborated on the conceptual design of the research and in the writing and critical revision of the manuscript for important intellectual content. All authors read and approved the final manuscript.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Maroco, J., Maroco, A.L., Campos, J.A.D.B. et al. University student’s engagement: development of the University Student Engagement Inventory (USEI). Psicol. Refl. Crít. 29, 21 (2016). https://doi.org/10.1186/s41155-016-0042-8

Download citation

Keywords

  • University students’ engagement
  • Scale construction
  • Academic achievement
  • School dropout