Adaptation and preliminary validation evidences of the School Climate Questionnaire – Revised, Elementary and Middle School Version (SCS-MS)

Abstract

This study presents the procedures for adapting the School Climate Survey – Revised, Elementary and Middle School Version (SCS-MS) to the Brazilian context and demonstrates the first evidence of the validity of its use among elementary school students. Participants were 511 children (52,1 % girls) from Porto Alegre, between 3rd and 5th grade. Analysis of internal validity was performed by evaluating the factor structure of the instrument, and analysis of convergent validity by testing the correlation between the SCS-MS, social skills and childhood peer aggression. A confirmatory factor analysis (CFA) and an exploratory structural equation modeling (ESEM) analysis were conducted. The results indicated that the structure provided in the ESEM analysis were more adequate than the original structure for this sample. ESEM provided satisfactory indices of fit for the model of six factors, including modifications in four factors. The overall score of the SCS-MS showed positive and moderate correlations with social skills and negative and moderate correlations with peer aggression. The results suggest that the SCS-MS is a measure that can be used to assess the students’ perception of school climate in Brazilian schools, contributing to overcome the shortage of instruments in this context.

Background

In the early twentieth century, studies related to school climate were based on observable characteristics of the environment, such as availability of material resources and the physical conditions of the school, demonstrating little relationship with student performance (see Anderson, 1982). From the 1980s, these studies began to focus on the perceptions of individuals regarding the organizational behaviors of the teachers and principals and the values shared within the school community (Haynes et al. 1997; Kuperminc et al. 1997). The results indicated that perceived quality of the school climate was negatively associated with different student characteristics, such as behavioral problems (Wang et al. 2010), aggressive behavior (Khoury-Kassabri, 2012) and depression (Jia et al., 2009). Conversely, the perception of the quality of the school climate was positively associated with factors such as school adjustment (Haynes et al., 1997), life satisfaction (Suldo et al. 2008) and mental health (Suldo et al. 2012).

Variation in perceptions of school climate is based on different factors. From a multilevel approach, there are student-, school-, and classroom-related factors that contribute to determine the student perception of school climate, including sex and race, school size and faculty turnover, and characteristics of the teacher, class size, and the concentration of students with behavior problems (Koth et al. 2008). There are also contextual factors that contribute to determine the student perception of school climate, including norms, goals, values, interpersonal relationships, teaching and learning practices, and organizational structure (Cohen et al. 2009). Furthermore, the student perception of school climate is based on student background and motivational factors, and quality and consistency of interpersonal interactions within the school community (Haynes et al., 1997).

School climate is a complex and multidimensional construct that refers to different aspects of school life (Thapa et al. 2013; Zulling et al. 2010). Due to the multiplicity of factors involving the school, there is no consensus about the definition of school climate or its dimensions (Cohen et al., 2009; Thapa et al., 2013). In this study, school climate will be understood as the perception of individuals (students, parents, and teachers/staff) regarding the quality and consistency of the personal interactions that involve the school community (Haynes et al., 1997). This definition focuses on the individual as the unit of analysis. Personal experiences construct the perception of each person regarding the school and this perception may influence their cognitive, social and psychological development (Haynes et al., 1997). Accordingly, although the students share the same physical space, each perceives the school environment in a unique way.

The definition of school climate used in this study consists of six dimensions: fairness, order and discipline, parent involvement, sharing of resources, student interpersonal relations and student-teacher relations (Emmons et al. 2002). Fairness refers to the equal treatment of students without discrimination due to race or socioeconomic status. Order and discipline concern students’ behavior appropriate to the school context. Parent involvement refers to the frequency of parental participation in school activities. Sharing of resources refers to offering equal opportunities for all students to participate in activities of the school and to use its materials and equipment. Student interpersonal relations concern the level of care, respect and trust demonstrated among the students in the school. Finally, student-teacher relations refer to the level of care, respect and trust shown between the students and teachers in the school.

Studies on this subject in the United States have culminated in the realization of a school climate reform in the country. This is an essential component for promoting healthy relationships and reducing school dropout, as well as for the prevention of bullying and for the development of schools (Thapa et al., 2013). Recently, the United States Department of Education published a guide containing resources that can assist the school community in the development of the school climate. The guide presents three principles to improve school climate and discipline: creating a positive school climate with a focus on prevention; the development of expectations and clear, consistent and appropriate consequences to deal with disruptive behaviors of students; and ensuring fairness, equity and continuous development (U.S. Department of Education, 2014).

In Brazil, the few studies that assess school climate have mainly sought to identify issues and solutions related to improving the quality of teaching (Adam & Sales, 2013; Brito & Costa, 2010). For example, Brito and Costa (2010) investigated the perception shared by teachers about the school climate and its relationship with the teaching profession, and Adam and Salles (2013), with violence in the school. Both aimed to characterize the school environment as a whole from the perspective of teachers, and not the individualized perception about the school. This information is relevant to the educational context, however, it does not meet the need for studies that assess the perception of each individual about the school, particularly the students. Although the importance of evaluating school climate in all contexts, there is a lack of validated measure in Brazil.

Positive school climate is related to prevention of student misbehavior (U.S. Department of Education, 2014). It is also important to assess the relation between student perception of school climate and behaviors at school as social skills and peer aggression. The results would be used to test convergent validity of school climate measures.

School Climate Survey - Revised, Elementary and Middle School Version (SCS-MS)

The original version of the School Climate Survey (Haynes et al. 1994) was developed to evaluate the Comer School Development Program in the United States (Haynes, 1998). This program was implemented in more than 1.150 schools nationwide and aimed to improve the quality of the educational experiences of students from minority groups (Lunenburg, 2011). Subsequently, the SCS started to be used in other studies aiming to assess the relationship between the school climate and variables of the students, such as levels of well-being (Briggs et al. 2010), behavioral and emotional problems (Kuperminc et al., 1997; Kuperminc et al. 2001), life satisfaction (Suldo et al., 2008) and mental health (Suldo et al., 2012).

The SCS aims to evaluate the perceptions of students, parents and teachers regarding the school climate. The SCS is composed of four separate questionnaires, two versions for students (one for elementary and one for high school), one version for parents and one version for teachers. The School Climate Survey – Revised, Elementary and Middle School Version (SCS-MS, Emmons et al., 2002) is the most recent edition of the SCS, which is answered by the students in elementary and middle schools. The age range of the children is from about 8 to 12 years old. In the present study it has been translated into Brazilian Portuguese as the Questionário de Clima Escolar – Revisado, Versão para Ensino Fundamental (QCE-EF).

(Ware, The school climate survey, unpublished) conducted confirmatory factor analysis (CFA) to demonstrate the evidences of validity of the SCS-MS and of each independent subscale. The participants were 2749 students of the third to the eighth year of elementary education of two cities (Asheville and Davidson) of the United States. The overall instrument was analyzed using CFA with maximum likelihood estimation. Results presented the follow fit indices: χ2 (614) = 3832.21, p < .001; NNFI = .90; CFI = .91; GFI = .92; SRMR = .046; RMSEA = .044. According to (Ware, The school climate survey, unpublished), the GFI, NNFI and CFI were below the threshold of .95. However, the SRMR and the RMSEA were in their respective acceptable ranges (.08 and .06). Although the results have indicated mixed fit indices and there is a lack of exploratory studies with this instrument, the SCS-MS have been used in many schools in the United States.

The present study aimed to present the procedures of cultural adaptation of the SCS-MS (Emmons et al., 2002) and to demonstrate the first evidence of validity for its use in the Brazilian context. Evidence of validity was demonstrated through the evaluation of the factor structure of the instrument and the convergent validity analysis. The hypotheses for the convergent validity analysis are that positive school climate will be positively correlated with social skills and negatively correlated with peer aggression.

Methods

Procedures for adapting the SCS-MS

The procedures of translation and adaptation of the SCS-MS followed the recommendations of Borsa et al. (2012). Initially, the scale was translated by three independent bilingual translators, which resulted in three independent Portuguese versions. With these initial versions, a synthesis was conducted by selecting the items which presented the most adequate translation in terms of semantic, idiomatic, conceptual, linguistic and contextual issues. This synthetized version was evaluated by a group of experts, which judged it as adequate (Borsa et al., 2012). The synthetized version of the SCS-MS was then evaluated by a group of three middle-school students, which where oriented to verify the adequacy of the scale instructions, items readability and response options. After minor revisions suggested by the group of students from the target population, the scale was back translated from the Brazilian Portuguese to English by other independent translator. The back translated version was evaluated by the authors of the original SCS-MS scale, which judged as adequate the adopted procedures and the Brazilian adapted version. After the agreement of the authors regarding the Brazilian version of the QCE-EF, the scale was considered ready to be used.

Participants

The study included 511 children and adolescents (52.1 % girls), aged from 8 to 12 years (M = 10.18, SD = 1.33), elementary education students of five schools located in the city of Porto Alegre. Of the total students, 387 (75.7 %) attended a public school and 124 (24.3 %) attended a private school. Regarding the grade, 137 (26.8 %) students were in their third year, 174 (34.1 %) attended the fourth year, and 200 (39.1 %) were in the fifth year.

Instruments

Brazilian version of the School Climate Survey - Revised, Elementary and Middle School Version (SCS-MS; Appendix). The SCS-MS consists of 37 items, answered on a 3-point scale (1 = I do not agree; 2 = I am not sure; 3 = I agree). These items assess the perception of students regarding the six dimensions of school climate, which showed the following levels of internal consistency in the original study (Ware, The school climate survey, unpublished): fairness (five items; α = .83); order and discipline (seven items; α = .75), parent involvement (five items; α = .68), sharing of resources (four items; α = .75), student interpersonal relations (seven items; α = .84), and student-teacher relations (nine items; α = .87). The scores in the SCS-MS are obtained by summing the items of the instrument, resulting in a general measure regarding the school climate. It is also possible to assess each subscale separately, summing only the items that comprise each dimension. Higher scores refer to a more positive perception of the school climate. Items 1, 6, 9, 12, 14, 20, 21, 23, 24, and 33 are reversed scored because agreement with them reflects negative opinions of school climate (Emmons et al., 2002).

Social Skills Subscale of the Social Skills Rating System (SSRS-BR₂; Freitas & Del Prette, 2015). This self-reported social skills scale evaluates the behaviors of empathy, responsibility, self-control, and assertion of the child. The instrument consists of 20 items, answered on a 3-point scale (0 = never; 1 = sometimes; 2 = very often). The sum of all items indicates the frequency with which the behaviors evaluated occur. The higher the total score of the instrument, the higher the frequency of the social skill behaviors, according to the perception of the child. The Cronbach’s alpha value of this scale was.73 in the Brazilian version (Freitas & Del Prette, 2015).

Peer Aggressive and Reactive Behaviors Questionnaire (PARB-Q; Borsa & Bandeira, 2014). This empirically based, self-reported instrument is composed of two independent scales that assess aggressive behaviors and reactions to peer aggression among children aged 6 to 13 years. This study used only the first scale (Aggressive Behaviors Scale - ABS) consisting of five items assessing the physical and verbal aggressive behaviors of the child, as well as three control items, disregarded for the score. The items are answered using a 4-point scale (1 = happens every day; 2 = happens sometimes; 3 = happens few times; 4 = never happens). The ABS presented an alpha value of .73 in the Brazilian version (Borsa & Bandeira, 2014).

Ethical procedures and data collection

This study was approved by the Ethics and Health Research Committee of the Institute of Psychology of the Federal University of Rio Grande do Sul (protocol No. 06781812.0.0000.5334). Ethical issues were guaranteed for all participants in accordance with the guidelines and rules of Resolution No. 196/96 of the National Health Council. Data collection occurred after authorization from the school principle and signing of the Terms of Free Prior Informed Consent (TFPIC) by the parents of the students. The data collection was carried out collectively in the classroom.

Data analysis procedures

Initially, we assessed the structure of the SCS-MS by means of an Exploratory Structural Equation Modeling (ESEM), using Mplus software (Muthén & Muthén, 1998-2012). The number of factors to be retained in the ESEM was previously assessed by the Hull method, using the FACTOR statistical program (Lorenzo-Seva & Ferrando, 2006). After the ESEM, a confirmatory factor analysis (CFA) was also conducted to evaluate the fit indexes of the original structure of the scale. The rationale to include the ESEM as well as the CFA was to evaluate both the exploratory structure of our sample (ESEM), but also test the plausibility of the original (Ware, The school climate survey, unpublished) six-factor structure (CFA). The following fit indices were used, with their respective reference values considered to be satisfactory (Brown, 2006): Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI) greater than or equal to .90 (preferably greater than .95), and Root Mean Square Error of Approximation (RMSEA) close to or less than .08 (with higher-bound 90 % confidence interval not exceeding .10).

The estimation method used was WLSMV (Weighted Least Squares Mean and Variance-Adjusted), with Geomin rotation, suitable for factor structures from simple to moderate and ordinal data with a distribution that deviates from normality (Asparouhov & Muthén, 2009). Convergent validity for the SCS-MS was tested by means of Pearson’s correlation analyses (AERA, APA, NCME, 1999) between the SCS-MS and the social skills and aggressive behaviors measures.

Results

Factor structure of the SCS-MS

The Hull method suggested a six-factor solution as the most reliable to the data. Thus, we implemented an ESEM on a six-factor structure. Results presented adequate fit indices for the model [χ2 (459) = 628.555; p < .001; CFI = .97; TLI = .95; RMSEA = .027 (CI 90 % .021 - .032)]. However, the six-factor structure was not totally equivalent to the original version (Table 1). The factors ‘sharing of resources’ and ‘student-teacher relations’ remained identical to the original version, with factor loadings above .40. However, the other factors presented some differences compared to the original version of the instrument. The ‘fairness’ dimension presented four items (3, 7, 13, 31) with factor loadings between .45 and .67. However, item 32 (“In my school, teachers are fair with everyone”), that originally belonged to the “fairness” dimension, presented cross-loading lower than .40, loading both on the original dimension and on the ‘student-teacher relations’ dimension. This can be explained due to the content of the item referring equally to the two dimensions, not discriminating the responses of the participants. The ‘order and discipline’ factor had six items (1, 6, 24, 25, 33, 34) with loads between .42 and .71 and three items (9, 18, 20) with cross-loadings lower than .40. Item 25 originally belonged to the ‘student interpersonal relations’ factor, however, its migration to this factor seems pertinent, since it refers to aspects of order and discipline in the school more than the relationship between the students (“The children in my school respect the teachers”). Item 34 (“The children in my school respect each other”) presented cross-loading, also loading on the ‘student interpersonal relations’ dimension. The ‘parent involvement’ dimension presented four items (4, 27, 29, 36) with loadings between .50 and .68. However, item 8 (“My parents usually attend parent meetings at my school”) had factor loadings between .30 and .40 in the ‘parent involvement’ and ‘student-teacher relations’ dimensions, appearing to not discriminate the responses between the dimensions. Finally, the ‘student interpersonal relations’ factor presented the greatest difference compared to the original instrument, which consists of seven items. In the Brazilian version, the results suggested that this dimension was composed of three items (15, 16, 22), the factor loadings of which ranged between .61 and .66. As previously mentioned, item 25 migrated to the ‘order and discipline’ factor and item 34 presented cross-loading, also loading on the ‘order and discipline’ dimension. Items 2 (“The children at my school behave well”) and 5 (“The children at my school are caring with other people”) presented loadings lower than .40 (Table 1).

Table 1 Exploratory Structural Equation Modeling (ESEM) of the Brazilian version of the SCS-MS

The analysis indicated that each factor presented significant positive correlations with at least one other factor of the instrument. These correlations ranged from weak to moderate (r ² = .11 to .43), being weaker between ‘order and discipline’ and ‘parent involvement’ and between ‘sharing of resources’ and ‘parent involvement’. The strongest correlation was found to ‘fairness’ and ‘student interpersonal relations’. All factors presented significant correlations with each other, except the dimension ‘sharing of resources’, which correlated only with ‘order and discipline’, and ‘student-teacher relations’ (Table 2).

Table 2 Pearson correlations between the factors of the SCS-MS with social skills and peer aggressive behavior

In order to evaluate the goodness-of-fit of the original six-factor structure (Ware, The school climate survey, unpublished), a CFA was conducted. The following fit indexes were achieved: CFI = .91; TLI = .90; RMSEA = .038 (.034 - .042). It is important to note that the ESEM presented apparently better results, because this analysis is less restrictive and allows items to load on more than one factor. Considering also that the ESEM presented a more reliable factor structure for our sample, we adopted the ESEM factor structure for the subsequent analyses.

Convergent validity

Convergent validity was assessed using Pearson’s correlation analysis between the SCS-MS and measures of social skills and of peer aggression (Table 2). The school climate factors showed positive correlations with social skills and negative correlations with peer aggression. There was a weak correlation of the general school climate with social skills (r ² = .37) as well as with peer aggression (r ² = -.32). Among the factors that make up the school climate, the strongest correlations were between social skills and the quality of student-teacher relations (r ² = .31) and between peer aggression and order and discipline at school (r ² = -.30). Conversely, the weakest or insignificant correlations were for sharing of resources with social skills (r ² = .11) and with peer aggression (r ² = -.02).

Discussion

The aim of this study was to present the procedures for adapting the SCS-MS (Emmons et al., 2002) to the Brazilian context and demonstrate the first evidence of the validity of its use among elementary school students. Analysis of internal validity was performed by evaluating the factor structure of the instrument, and analysis of convergent validity by testing the correlation between the SCS-MS, social skills and childhood peer aggression.

Confirmatory factor analysis performed using the original version of the SCS-MS indicated mixed fit indices (Ware, The school climate survey, unpublished). In the present study, exploratory structural equation modeling (ESEM) was performed due to the lack of exploratory studies with this instrument and the existence of possible differences in the factor structure of the instrument to the Brazilian sample. The results presented suitable fit indices for the six factor solution of the SCS-MS to the Brazilian sample. However, there were differences regarding the number of items in the Brazilian version and the composition of four factors (fairness, order and discipline, parent involvement, student interpersonal relations). Of the 37 items that compose the original instrument, seven presented factor loadings below .40 and one presented cross-loading, suggesting a structure of 29 items for the Brazilian version of the SCS-MS.

The differences between these studies could be due to the sociocultural characteristics of the countries. The school is part of a broader social context, reflecting the standards, values and culture of the society (Bronfenbrenner, 1979). Item 20 (“Some children bring guns or knives to my school”) can be regarded as an example of the differences between Brazil and the United States, which have different laws regarding the possession and use of weapons. The Brazilian population, for example, has more restricted access to firearms than the American population. For the children in this study, item 20 did not appear to be related to the other items linked to school climate, with factor loadings below the cutoff point. Therefore, the factors of the macrosystem of a country can relate to the students’ perception of the school climate, confirming the importance of exploratory studies in the SCS-MS validation process. Future investigations could evaluate the adequacy of the factor structure of the adapted version of the SCS-MS in other Brazilian samples, and compare this with the structure of the original instrument.

The majority of the latent factors correlations of the SCS-MS were significant, ranging from weak to moderate. ‘Sharing of resources’ presented no correlations with the other factors. Unlike the other dimensions, the ‘sharing of resources’ content does not directly assess the quality of the interpersonal relationships in the school, which may explain the absence of correlation between the factors. Future investigations may indicate how Brazilian schools function regarding the ‘sharing of resources’ and its relationship with different school climate factors. It is important to note that this pattern of factor correlations is influenced by the ESEM analysis. Once ESEM allows items to have factor loadings in more than one factor, the factor co-variance tend to diminish, because the observed variables (items) are already correlated altogether.

The analyses of convergent validity demonstrated that the SCS-MS presented positive correlations with the social skills measures and negative correlations with peer aggression. These analyses provide information on the correlation patterns of the score of the instrument with other variables and present indicators regarding the predictive ability in relation to other factors of direct interest (AERA, APA, NCME, 1999). These results also corroborate previous studies that demonstrate a negative association between the perception of students regarding the school climate and behavioral problems (Goldstein et al. 2008; Wang et al., 2010), and a positive association with school adjustment (Haynes et al., 1997; Kuperminc et al., 1997). Students who perceive the school climate as positive tend to view the school as an environment where there is respect, trust and generosity among people, reducing the use of violence (Haynes et al., 1997). In addition, they may feel more supported, are less involved in behavior problems, and seek more support to deal with bullying situations (Gregory et al. 2011).

Conclusion

In Brazil, studies related to school climate have evaluated the collective perception of teachers regarding the school environment in order to improve the teaching and learning process. This study aimed to adapt an instrument that assesses the perceptions of students regarding the school climate and present evidence of convergent validity for its use in Brazil. The SCS-MS was shown to be a measure that can be used in schools, contributing to overcome the shortage of instruments for use in this context. Similarly, the SCS-MS may assist in conducting new studies aimed at comprehending the perception of students regarding the school climate and its relationship with socioemotional development.

The concept of school climate adopted in this study refers to the students’ perception about the quality and consistency of the personal interactions that occur in the school and involve the entire school community. This perception is subjective and results from personal experiences inside and outside the school. Accordingly, the SCS-MS can provide important information related to which dimensions of the school climate are more relevant to the students, encouraging the implementation of programs aimed at improving these dimensions in the school (Kuperminc et al. 1997).

One of the limitations of this study was the use of a local sample from Rio Grande do Sul for the adaptation and validation of a measure to the Brazilian context. Expanding the sample is necessary to represent the diversity of the country population and school context. Future studies could provide further evidence for the validity of the SCS-MS to different Brazilian states and also seek to investigate the relationship between school climate and other individual and contextual variables.

Appendix

Table 3 Brazilian Portuguese items of the School Climate Survey – Revised, Elementary and Middle School Version (SCS-MS)