University student’s engagement: development of the University Student Engagement Inventory (USEI)

Abstract

Student engagement is a key factor in academic achievement and degree completion, though there is much debate about the operationalization and dimensionality of this construct. The goal of this paper is to describe the development of an psycho-educational oriented measure – the University Student Engagement Inventory (USEI). This measure draws on the conceptualization of engagement as a multidimensional construct, including cognitive, behavioural and emotional engagement. Participants were 609 Portuguese University students (67 % female) majoring in Social Sciences, Biological Sciences or Engineering and Exact Sciences. The content, construct and predictive validity, and reliability of the USEI were tested. The validated USEI was composed of 15 items, and supported the tri-factorial structure of student engagement. We documented evidence of adequate reliability, factorial, convergent and discriminant validities. USEI’s concurrent validity, with the Utrecht Work Engagement Scale-Student Survey, and the predictive validity for self-reported academic achievement and intention to dropout from school were also observed.

Background

In recent years, student engagement, broadly defined as the “students’ willingness to participate in routine school activities, such as attending classes, submitting required work, and following teachers’ directions in class” (Nystrand & Gamoran 1992, p. 14) has received increasing attention by researchers, practitioners, and policy makers (Christenson et al. 2012; Fredricks et al. 2004; Reschly & Christenson 2012; Salanova et al. 2010). The growing body of research, primarily conducted in the United States and Australia, shows that student engagement can act as an antidote to low academic achievement, student burnout, student lack of resilience, dissatisfaction, and school dropout (Christenson & Reschly 2010; Elmore & Huebner 2010; Finn & Zimmer 2012; Wang & Eccles 2012; Krause & Coates 2008). It can also act as a strong mediator between out-of-school variables (e.g., home and family, friends and class-inmates,) (Chen & Astor 2011), teacher-student interactions, academic achievement, school success and life-long learning (Christenson et al. 2012; Gilardi & Guglielmetti 2011; Reschly & Christenson 2012). Recent studies have found a correlation between different engagement profiles and students’ learning (Wang & Eccles 2012) and physical and psychological health and well-being (Li & Lerner 2011; Wang & Eccles 2012).

In Europe, the engagement construct has emerged mainly in professional and occupational activities, and has been researched primarily in relation to business organizations (Bakker et al. 2008; Hirschi 2012; Schaufeli & Bakker 2010). From the business organizations' perspective, work engagement has been defined as a positive, fulfilling, affective-motivational psychological state that is characterized by vigour, dedication and absorption associated with work-related well-being (Bakker et al. 2008). This conceptualization is supported by Maslach and Leiter’s (1997) initial definition of work engagement, which is the opposite of work burnout. Several different conceptualizations of burnout and its relations to work engagement coexist (Hirschi 2012). However, most professionals would agree that engagement is an independent and distinct construct that is negatively correlated with burnout, and that engaged employees have high levels of energy, low burnout and are strongly identified with their organization (González-Romá, Schaufeli, Bakker & Lloret 2006; Bakker et al. 2008; Demerouti et al. 2010). In this organizational perspective, work engagement is operationalized by three correlated constructs: vigour, dedication and absorption. According to Bakker et al. (2008), vigour is characterized by high levels of physical energy, ability to deal with the job’s challenges, and a willingness to invest the required effort to overcome difficulties. Dedication is characterized by a strong involvement in one’s work with a sense of personal realization and pride. Finally, absorption is defined as concentration and happiness in performing the job related tasks (Bakker et al. 2008; Hirschi 2012). Grounded in these definitions, Schaufeli and co-workers have developed the Utrecht Work Engagement Scale (UWES) to measure these three dimensions of work engagement (Schaufeli & Bakker 2003). Nowadays, the UWES is the most used instrument to measure engagement in the work place (Bakker et al. 2008). There have been attempts to expand this construct to non-workplace settings. For example, the UWES was adapted to measure school engagement in university settings, around its three dimensions: vigour, dedication and absorption (Salanova et al. 2010). The UWES-Student Survey has been used in a few studies (Bresó et al. 2011; Gilardi & Guglielmetti 2011). However, concerns have been raised about simply rephrasing items for the work-place to the university setting (Mills et al. 2012; Schaufeli & Bakker 2010), as well as its dimensionality in different age-groups of students (Upadaya & Salmela-Aro 2012).

Indeed, there is a growing consensus among researchers that engagement is a multidimensional construct with both behavioural, emotional and psychological components (see Finn & Zimmer 2012 for a review). However, there is still no clear consensus on the construct precise definition and its dimensionality (Christenson et al. 2012; Fredricks & McColskey, 2012; Reschly & Christenson 2012; Wolf-Wendel et al. 2009; Kahu 2013). For example, proposals for the construct dimensionality have ranged from two- (emotional and behavioural) (Skinner et al. 2009) to eight- (Learning strategies, Academic integration, Institutional emphasis, Co-curricular activity, Diverse interactions, Effort, Overall relationships and Workload) first order factors (LaNasa et al. 2009). Higher dimensional construct conceptualizations have been also proposed. For example, Martin (2007) documented 11 first-order factors with a second order 4-factor model of student engagement and motivation with Australian high school students. There is also large variation in how engagement has been measured (e.g. questionnaires, self-report measures, teacher ratings, interviews and observations) (Fredricks et al. 2011). In another study with first year Australian Universities’ undergraduates, seven ‘calibrated’ measures of engagement were described (Krause & Coates 2008). This variation in both the operationalization and measurement of the engagement construct has made it difficult to make comparisons between studies’ findings (Fredricks & McColskey 2012; Kahu 2013). Therefore, consensus on the different measuring instruments, their dimensionality and concurrent validity is an important area of future research (Reschly & Christenson 2012; Wolf-Wendel et al. 2009; Krause & Coates 2008; Kahu, 2013).

In this paper, we expand Nystrand and Gamoran (1992) broad definition of students’ engagement, by adding the Fredricks et al. (2004) conceptualization of engagement as a three factor construct including behavioural, emotional and cognitive dimensions. This is one approach, to measure students’ engagement, that tries to integrate the four dominant research perspectives on this important construct, namely, the behavioural perspective, the psychological perspective, the socio-cultural perspective and the holistic perspective (for a review and criticism of the different perspectives on student engagement see Kahu 2013). Cognitive engagement is defined as the students’ investment and willingness to exert the necessary efforts for the comprehension and mastering of complex ideas and difficult skills. The emotional engagement dimension reflects both the positive and negative reactions to teachers’ instructions, classmates and school, perceptions of school belonging, and beliefs about the value of schooling. Finally, behavioural engagement is defined in terms of student’s participation in classroom tasks, conduct, and participation in school-related extracurricular activities (see also Carter et al. 2012; Sheppard, 2011).

The majority of studies of student engagement have been conducted in the USA and Australia, with elementary, middle and high school students. One exception is the USA’s National Survey of Student Engagement (NSSE) (NSSE 2012) and the Beginning College Survey of Student Engagement (BCSSE) (BCSSE 2013), which measures engagement in first-year US college students (Chambers & Chiang 2011; Kuh 2009; McCormick & McClenney 2012). However, the NSSE has been strongly criticized. Some scholars emphasize that the NSSE lacks good psychometric properties (namely construct and predictive validities and overall reliability) (Campbell & Cabrera 2011; LaNasa et al. 2009), and that it does not directly measure the “student engagement” psychological construct, but rather students’ studying habits, gains from their college experiences, and other aspects of student life (Wefald & Downey 2009). Although most of these criticisms on the NSSE have been addressed (see McCormick & McClenney 2012), there are still unanswered questions regarding the psychological “student engagement” construct definition and its dimensionality in different schooling contexts, student minority groups and family and social contexts where schools are immersed. Another example, is the seven ‘calibrated’ scales of first-year undergraduates’ engagement in Australian Universities (Krause & Coates 2008). However, this Australian study is not conclusive regarding the construct dimensionality. On the contrary, it points out to the imperative necessity for developing a broader understanding of engagement as a construct with several dimensions (Krause & Coates 2008). Accordingly to these scholars, the multidimensional feature of students’ engagement must be acknowledged in any measurement and monitoring of this construct in higher education.

Interventions aimed at improving university students’ academic achievement, reduce burnout improve student well-being, engagement and graduation success have been pursued in several high schools and universities all over the world (see e.g., Bresó et al. 2011; Chen & Astor 2011; Christenson & Reschly 2010; Elmore & Huebner 2010; Fredricks et al. 2004; Harlow et al. 2011; Kuh 2009; Li & Lerner 2011). Increasing engagement among university students has become crucial to improving their learning experiences, well-being and return in the investment in higher education (Bresó et al. 2011; Christenson & Reschly 2010; Kuh 2009; Salmela-Aro et al. 2009). Therefore, data with good psychometric properties on university students’ engagement and its correlates with academic achievement, course-work fulfilment, graduation and school integration are fundamental for educational psychologists, school counsellors as well as education policy-makers.

To the best of our knowledge, with the exception of the UWES-SS developed in Europe, the Beginning College Survey of Student Engagement (BCSSE) (BCSSE 2013) in the USA and seven ‘calibrated’ scales of first year undergraduate students in Australian Universities (Krause & Coates 2008) both of which have suffered from several criticisms, there are no psychometrically valid and publically available measures of student engagement in university settings. The UWES-SS mimics the professional work-place related engagement and only some of its items comply with studying and university activities (e.g., vigour). Thus, its use in the university context has suffered some criticisms, either due to methodological limitations in items’ construction or non-adaptation to the university context (Mills et al. 2012). On the other hand, surveys developed for high school students (see e.g. Fredricks & McColskey 2012) leave out some key facets of students’ engagement that are important at the university level (e.g., attendance at conferences and seminars). Also, the engagement construct’s dimensionality may vary with different students’ ages (Upadaya & Salmela-Aro 2012). Therefore, surveys developed for younger students may lack the appropriate construct definition and dimensionality, as well as adequate psychometric properties for evaluating student engagement in university students.

The present study draws from the earlier work of Schaufeli et al. (2002) on the UWES-SS, the work by Fredricks et al. (2004) with upper elementary school students in an effort to integrate the 4 perspectives on student engagement described by Kahu (2013) into a single measure instrument. In this study, we developed a new measure, – the University Student Engagement Inventory (USEI) and describe its psychometric properties in a sample of Portuguese college students. We examine content, construct and criterion related validities, and measurement invariance in two independent samples of students, from public and private universities and several study areas. Concurrent validity with the UWES-SS was assessed, as well as predictive validity for self-reported academic achievement and intention to dropout. We hypothesize that ‘university student engagement’, conceptualized as the students’ involvement and motivation towards his or her course work and academic activities within the university, is a second-order construct that is reflected in 3 first-order emotional, behavioural and cognitive dimensions expanding the Nystrand and Gamoran (1992); Fredricks et al. (2004) and Kahu (2013) conceptualizations. We anticipate that this construct is predictive of students’ academic achievement and intention to drop out of school. This new measure, focused on behaviors, cognitions, emotions and actions, that is, on the behavioral, psychological and holistic aspects of engagement can be used both for student counselling as well as research aimed at educational policy and good practices development to improve academic success and school retention.

Methods

Participants

A total of 609 university students volunteered to participate in this study. Sixty-seven percent of participants were female. Students’ mean age was 23.6 years (SD = 7.5) the 1^st quartile was19 years; The median was 21 years and the 3^rd quartile was 24 years. Students were enrolled in 6 or 10-semester degree courses (BSc or MSc, respectively). Forty one percent of students were in their 2^nd semester, 23 % were enrolled in the 6^th semester and 17 % were enrolled in the 4^th semester. Enrolment in the MSc 8^th and 10^th semesters was 8 % and 7 % respectively (the % of students enrolled in odd numbered semesters was marginal, since the data collection occurred in the Spring Semester). Seventy percent of the students were enrolled in public universities; the remaining 30 % were attending private universities. Participating students were enrolled in human and social sciences courses (51 %) health and biological sciences (26 %), and engineering and exact sciences (23 %). The large majority of students lived with their parents (75 %); the remaining lived with friends and colleagues (17 %) or by themselves (8 %). Furthermore, the majority of students had their studies sponsored by their families (65 %) with approximate distributions for scholarships (15 %) and own funds (20 %).

Questionnaire development

A questionnaire composed of socio-demographic questions (age, gender, semester, type of school, area of studies, financing and housing), the number of courses taken, self-perceived academic achievement (on a 5-point scale ranging from ‘1-not good at all’ to ‘5-very good’) and intention to dropout (on 3-point rating scale from ‘1-never’ to ‘3-always’) was constructed.

The initial USEI form was composed of 32 items (see Table 1 for the items’ description) rated on a ‘1-never’ to ‘5-always’ response scale. The initial USEI form included the 15-item School Engagement questionnaire, developed by Fredricks and her colleagues (2004) for upper elementary school students, which was adapted to the university context, after obtaining the authors authorization. Additionally, 17 new items were created from a focus group analysis.

Table 1 Minimum (Min), Maximum (Max), Median (Me), Mean (M), Skewness (Sk), Kurtosis (Ku) and Content Validity Ratio (CVR) for the original 32 items tested (SE _Sk  = 0.138; SE _Ku  = 0.275). In between parenthesis are the original Portuguese language items

The focus group included 10 university students from the social, health and engineering sciences, who volunteered to participate in the focus groups session held at a private university. These students were interviewed in an open discussion forum during a two-hour period. In this forum, students were presented with a brief and broad definition of student engagement as being the personal and emotional involvement of the student with his or her coursework, classroom and academic activities and the willingness to follow teachers’ instructions within the university settings. Thereafter, the students were asked to answer and or discuss three topics/questions posed by the first author of this manuscript. The three questions were “1. How would you define a student engaged in his/her course work and university?”; “2. What daily practices do you use to succeed in your course work requirements?” and “3. What kind of academic activities do you practice or would like to practice outside of the classroom that are related to your university experience?” This group was composed by 6 female and 4 male students, with a mean age of 21 years (in the 5^th semester of their courses). Main ideas from the students’ answers and open discussion that followed were recorded. From these notes, 17 new statements/items related to the university context and activities were created and distributed, accordingly to their content, to one of the 3 dimensions: emotional, cognitive and behavioural engagement.

Items with good content related validity and good psychometric properties (see below) in the test sample (n = 313) were retained for the final format of the 15-items USEI. This form was then applied to a second, independent, validation sample (n = 296) simultaneously with the UWES-SS (Schaufeli et al. 2002). The UWES-SS is composed of 15 ordinal rating items scored from ‘0-never’ to ‘6-always’ that define three factors as described in the introduction. A Portuguese, short-form version with 9 items provided and validated by the UWES-SS authors was used for evidence of concurrent validity studies (for details on the UWES-SS see Schaufeli et al. 2002).

Procedure

Participant recruitment was done through students’ syndicates and or academic associations, which invited their members/associates by email. Each participant agreed on an informed consent stating that the participation was voluntary, that no personal information that could be used to identify the participants would be asked, that they could drop the research at any time and that the study was purely academic with no intention to diagnose or to treat. Participants were also assured that the data would be presented only in aggregated statistical analysis. No incentive to participation was given. The data was gathered through an online questionnaire composed of sample characterization questions and the USEI and UWES-SS’ items available online during the 2^nd semester of the 2012–2013 school year (February to May 2013). The ISPA/UIPES Ethics committee approved the proposed research study.

Questionnaire evaluation

Psychometric properties

To examine psychometric properties, we gathered two independent samples: an initial test sample (n = 313) where sensitivity and confirmatory factor analysis were performed and a second validation sample (n = 296) to test evidence for sampling measurement invariance and concurrent validity with the UWES-SS. Both samples were equivalent by mean age and semester as well as by gender, areas of study, and public versus private university distributions. Minimum sample size requirements’ was determined for 32 manifest variables and 3 latent factors with an a priori estimated effect size of 0.1, statistical power of 0.8 and type I error rate of 0.05, using the formula provided by Westland (2010).

Items’ distributional properties

Summary measures and skewness (sk) and kurtosis (ku) for each of the 32 original items estimated with IBM SPSS Statistics (v. 20, SPSS An IBM company, Chicago, IL) were used to judge distributional properties and psychometric sensitivity. Absolute values of sk smaller than 3 and ku smaller than 7 were considered indicative of no strong deviations from the normal distribution (Finney & DiStefano 2006; Kline 2005). These items’ distributional properties are indicative of appropriate psychometric sensitivity as it would be expected that these items would follow an approximate normal distribution in the population under study.

Evidence of content related validity

The content related validity of the USEI’s 32 original items was evaluated by a pool of 20 educational psychologists, as proposed by Lawshe (1975). The psychologists rated each item on a 3-point rating scale: “Essential”, “Useful, but not essential” and “Not necessary”. The Lawshe’s content validity ratio (CVR) was calculated for each item and averaged for the total scale. A CVR of 0 or greater indicates that at least half of the judges deemed the item as “Essential” for the construct assessment (Lawshe 1975).

Evidence of construct related validity

We tested the factorial, convergent and discriminant related validities of the cognitive, emotional and behavioural dimensions theorized for university students’ engagement following Fornell & Larcker (1981) theoretical framework (see below). To evaluate the USEI’s 3-factor model evidence for factorial validity, a confirmatory factor analysis (CFA) was performed on the items’ polychoric correlation matrix using the WLMSV estimator as implemented in Mplus (v. 7, Muthén & Muthén, Los Angeles, CA). The goodness-of-fit of the 3-factor engagement structure was assessed with usual CFA indexes: Chi-square over degrees of freedom (χ²/df), Comparative fit index (CFI), Tucker-Lewis fit index (TLI) and Root mean square error of approximation (RMSEA). The fit was considered good for χ²/df around 2, CFI and TLI larger than .95 and RMSEA smaller than .08 (see e.g. Byrne 2012; Maroco 2014). Individual standardized factor loadings (λ) and modification indices (MI) for each of the original items were evaluated for item reduction, supported on theoretical considerations regarding items’ content, CVR and factor manifestations. Items with λ < .5, CVR < .0 and or MI > 11 (Maroco 2014) were removed from the original 32 items pool. Convergent and discriminant related validity for the 3 factors was evaluated as proposed by Fornell and Larcker (1981). Average variance extracted (AVE) by each factor larger than .5 was considered indicative of convergent validity while squared correlations between every two factors smaller than each of the factors’ AVE was indicative of discriminant validity (Fornell & Larcker 1981; Maroco 2014). Considering the theoretical definition of student engagement, supported by moderate correlations between factors, a second-order factor model, with student engagement reflecting on emotional, cognitive and behavioural engagement dimensions was also tested by CFA as described above.

Evidence of strong measurement invariance for the 15-best items retained form the analysis on the test sample was evaluated using multi-group analysis by imposing increasing constrains for the factor weights, mean intercepts and factor covariances for the test and validation samples (Byrne 2012; Maroco 2014). A Δχ² test, corrected for WLMSV estimation as proposed by Muthén & Muthén (2012) Mplus’ v.7 user guide, was used to evaluate the significance of the differences between the constrained models and the free estimates models in both samples. Strong measurement invariance was assumed when the Δχ² for the factor weights, mean intercepts and factor covariances was not statistically significant (p > .05).

Reliability

Reliability for the 3 factors of student engagement was evaluated by the factors’ internal consistency evaluated both by the Composite Reliability (CR) (Fornell & Larcker 1981) and Cronbach’s alpha calculated on the items’ Polychoric correlations matrix (α_P) (Gadermann et al. 2012). Values of CR and α_P larger than .7 were indicative of acceptable internal consistency for exploratory research (Gadermann et al. 2012; Maroco & Garcia-Marques 2006).

Evidence for Criterion related validity

The evidence for concurrent criterion related validity was evaluated by correlational analysis between the mean score estimates for the 3 factors of USEI and the 3 factors of the UWES-SS (Schaufeli et al. 2002) on the validation sample (n = 296). On a previous step, evidence for construct validity was demonstrated for the UWES-SS short form (composed by 9 items) (χ²/df = 4.32; CFI = .97; TLI = .95; GFI = .96; RMSEA = .08). Evidence of predictive criterion validity of the USEI total mean score on self-perceived academic achievement, failing courses and intention to drop out of school was evaluated by ordinal logit regression using Mplus (v. 7; Muthén & Muthén, Los Angeles, CA).

Results

Items distributional properties

Descriptive Statistics and CVR for the original 32 items are given in Table 1. Nineteen of the 32 items had a median value of 4. The overall mean response for the 32 items was 3.5 (SD = 0.75). No item showed Sk and Ku values that were suggestive of a severe deviation from the Normal distribution (all absolute values of Sk and Ku were lower than 2).

Evidence for content related validity

Evaluation of the relevance of the items by a panel of 20 educational psychologists indicated that 27 of the 32 initial items were “essential” to evaluate university’s student engagement by at least half of the specialists (CVR > 0) (see Table 1). These items were then subjected to a confirmatory factor analysis for the assessment of construct related validity.

Evidence for construct related validity

From the CFA of the 27 original items with content related validity, a final set of 15 items, which showed factor loadings greater than .5 in each of the 3 dimensions tested and no factors’ cross-loadings as suggested by Modification Indices analysis (MI >11; p < .001) were retained (See Table 2). Factorial related validity was observed for the 15-item Questionnaire distributed by the theoretical 3-factor model of Student Engagement (χ²/df = 2.26; CFI = .97; TLI = .97; RMSEA = .06). Convergent related validity, as defined by Fornell and Larcker (1981) could not be assumed only for the Behavioural factor. On the other hand, both Emotional Engagement as well as the Cognitive Engagement had AVE and CR larger than .5 and .7 respectively (see Table 2).

Table 2 Standardized factor loadings (λ) for the retained 15 items with CVR > 0 and λ > .5, AVE, CR and CVR for each of the 3 engagement dimensions (Test sample n = 313). In between parenthesis are the original language items

Squared correlations between Behavioural vs. Emotional and Cognitive engagements were .39 and .41, respectively. Squared correlation between Emotional vs. Cognitive engagement was .17 (see Table 3 for the correlations between dimensions). Therefore, discriminant validity was observed between Emotional and Cognitive engagement and between Behavioural and Emotional engagement. The retained factors had an average CVR of .62 for the Behavioural dimension, .32 for the Emotional dimension and .24 for the Cognitive dimension.

Table 3 Correlations (r) between first-order behavioural, emotional and cognitive engagement and standardized structural weights (β) from USEI to Behavioural (BE), Emotional (EE) and Cognitive Engagement (CE), AVE, CR and overall CVR on the overall combined test and validation samples (n = 609)

Evidence for measurement invariance

The multigroup analysis comparing the tri-factor engagement dimensions in the test sample (n = 313) and in a new, independent, validity sample (n = 296) with similar socio-demographic characteristics revealed both weak (equal factor weights: Δχ²(15) = 13.9; p = .588), strong (equal intercepts: Δχ²(15) = 18.3; p = .247) and structural (equal covariances: Δχ²(3) = 3.7; p = .296) invariance. Thus, the USEI showed strong measurement invariance in two independent samples of university students.

Engagement as a second order factor

Giving the theoretical considerations that student engagement is a second order factor that is reflected in three behavioural, emotional and cognitive first order factors, and given the moderate correlations between these first order factors, we tested a second order factor model where student engagement as a 2^nd order factor. Since the 3-factor measurement model of engagement showed strong measurement invariance in the test and validation sample, the second order factor structure was tested on the combined sample. Table 3 gives the standardized factor loadings for the second order (β) student engagement construct.

University student engagement reflects mostly on the behavioural engagement (β = .95; p < .001), but also has strong impact on cognitive (β = .73; p < .001) and emotional engagement (β = .66; p < .001). This second order construct, constituted by 15 items with an average CVR of .39, explained 62 % of variability of the 3 first order engagement dimensions with strong reliability (CR = .83) and good overall goodness-of-fit to the items’ variance-covariance data (χ²/df = 2.78; CFI = .978; TLI = .974; RMSEA = .054).

Evidence for criterion related validity

The concurrent validity of the proposed USEI with the UWES-SS, was evaluated by a correlational analysis done in the validation sample (n = 296). Table 4 shows the correlations between the USEI and the UWES-SS short form.

Table 4 Correlations between the USEI dimensions and its total and the UWES-SS dimensions and its total in the validation sample

A strong and positive correlation was found between the USEI total and the UWES-SS short form total (r = .99; p < .001) demonstrating the concurrent validity of the USEI when compared to the UWES-SS. Moderate correlations were found between the 3 dimensions of USEI and the 3 dimensions of UWES.

The ordinal logit regression of self-perceived academic achievement, failing courses and intention to drop out of school on the USEI total score showed positive predictive validity over self-perceived academic achievement (Β = 1.024; SE = 0.125; p < .001, OR = 2.79). Furthermore, it presented a negative predictive ability both on failing courses (Β = -0.649; SE = 0.111; p < .001; OR = 0.52) and intention to drop out of school (Β = -0.543; SE = 0.102; p < .001; OR = 0.58).

Reliability

Ordinal Cronbach’s α was .74 for the Behavioural Engagement, .88 for Emotional engagement and .82 for Cognitive engagement. These values were similar to the ones obtained for the composite reliability (see Table 3). Ordinal Cronbach’s α for the USEI’s 15 items total was .88. Altogether, these values show the overall good reliability of the three USEI dimensions and its total.

Discussion

A growing body of research has shown that student engagement is predictive of students’ well-being, satisfaction with school and peers, academic achievement, and lower burnout and school dropout rates (see e.g. Chen & Astor 2011; Finn & Zimmer 2012; Kuh 2009; Salmela-Aro et al. 2008). In studies with University students, engagement has been measured primarily with the UWES-SS in Europe and the BCSSE in the USA. However, these two instruments have suffered several criticisms ranging from the construct definitions and dimensionality to its application to university students (Campbell & Cabrera 2011; LaNasa et al. 2009; Mills et al. 2012; Wefald & Downey 2009). However, there is still a large debate on the definition (Finn & Zimmer 2012; Fredricks et al. 2011; Schaufeli & Bakker 2010; Schaufeli et al. 2002) and dimensionality (Fredricks & McColskey 2012; LaNasa et al. 2009) of student engagement construct and the need for validated measures is necessary for research in psychology and education in the area of student motivation, burnout and academic success.

In this study, we aimed to evaluate student engagement at the university level, using the expanded three dimensions (behavioural, emotional, and cognitive engagement) previously addressed with students from elementary and high schools in the US. From the different construct dimensionalities observed in the literature (see Fredricks et al. 2011; Kahu 2013; Krause & Coates 2008), our proposed USEI was built upon a scale initially developed by Fredricks and her colleagues for use with elementary students, and which has been also used with older students. Since contextual factors specifically related to the university context have been left out from these previous adaptations, a focus group analysis was conducted to identify specific university activities that could impact on student engagement and a set of new items were developed. Furthermore, there is evidence that the construct dimensionality can be also age-related (Upadaya & Salmela-Aro 2012). Thus, reported differences in the constructs dimensionality may be not only due to school and family related contexts, but also to students’ developmental processes.

Our study adds to the definition of university students’ engagement from an educational perspective in opposition it with the organizational perspective proposed by Schaufeli and colleagues, currently, the most used in Europe (Bresó et al. 2011; Salanova et al. 2010; Schaufeli et al. 2002). The Schaufeli et al. inventory has been criticized for its’ methodological approach (Mills et al. 2012), and difficulties in applying the organizational dimensions to the educational context (e.g., absorption and vigour), (Upadaya & Salmela-Aro 2012; Wefald & Downey 2009). Our proposed inventory addresses these criticisms.

Psychometric data analyses support the behavioural, cognitive and emotional tri-factorial conceptualization of student engagement. Evidence for strong measurement and structural invariance for this 3-factor model was demonstrated in two independent samples of Portuguese university students. Moderate to strong correlations were observed between the three factors, allowing for an empirically supported second-order engagement construct that is reflected on first order behavioural, emotional and cognitive dimensions. This second-order model showed a good fit to university students from both public and private universities majoring in social and human sciences, biological sciences and exact sciences.

Furthermore, the USEI produced reliable estimates of engagement and showed evidence for concurrent validity with the UWES-SS. Although both scales assess overall student engagement, the moderate correlations between their dimensions suggest that different facets of student engagement are being assessed by USEI and UWES-SS as an item’s content analysis would support. The USEI also showed evidence for predictive validity over a set of indicators of student academic achievement, courses failed and intention to dropout. For each extra point in the engagement scale the odds of improving academic achievement, as self-perceived by the student, increases significantly almost 3 times while the chances of failing one or more courses or dropping out of school decreases 2 times.

Our results show that data collected from a new instrument of student engagement has predictive value for important academic variables and can be used to further develop research and psycho-educational interventions on student engagement. Further research should be aimed at developing student engagement as a mediator to improve student achievement, reduce academic failure, promote psychological well-being and reduce university dropout rates.

However, this study has some limitations that need to be pointed out. First, the data of this study is only for a Portuguese population. Therefore, other cross-cultural studies are necessary before the USEI can be used more broadly in research and interventions aimed at improving student engagement. Second, a limitation of this instrument for a U.S. college student population is that it has no items applying to residential life which is a very important aspect of the college experience in the U.S for first year students. Thus the USEI can be used for students living out-of-campus, but may miss an important dimension for the other students’ academic experience. It also did not escape our attention that some items targeted initially to evaluate facets of behavioral engagement (e.g., items encompassing both classroom tasks and school-related extracurricular activities) did not make it to the final 15-item format, either because the experts did not judge these items as fundamental and/or either because they showed weak factor loadings in their proposed factors.

The results observed with the USEI also point to some new research direction to which the USEI maybe useful. For example, when is a student’s engagement at risk during his/her course works? In the earlier semesters or near the end of the academic work? Can the USEI be used as a predictor of student drop-out and or failing courses? Can a relationship between student academic performance, physical and psychological well-being and the students’ engagement in his/her course work be demonstrated? Can psychoeducational interventions aimed at improving students’ engagement have beneficial outcomes for the students’ proficiency and well-being? Can the USEI be used to produce valid and reliable measures of engagement during interventions? These are just a few research questions for an area where much research, supported on valid and reliable data, is needed.

Conclusions

The validated USEI was composed of 15 items, supports the tri-factorial structure of student engagement. We documented evidence of adequate reliability, factorial, convergent and discriminant validities in a sample of Portuguese college students. USEI’s concurrent validity, with the Utrecht Work Engagement Scale-Student Survey, and the predictive validity for self-reported academic achievement and intention to dropout from school were also observed.