Construct validity and reliability of Olweus Bully/Victim Questionnaire – Brazilian version

Bullying, one of the most common forms of violence in schools, is defined as power asymmetry associated with differences in age, gender, or race which is exploited by one or more individuals with the intention of hurting or humiliating another (Olweus, 1993). Recurrence over time is also a key aspect of bullying (Berger, 2007), along with the involvement of a bully, or perpetrator, and of a victim, the target of the aggression. Some individuals may be at the same time perpetrators and victims, and are therefore classified as bully-victims (Malta et al., 2010).

In broad terms, bullying may be classified as direct or indirect (Lopes Neto, 2005). Direct, face-to-face bullying draws more attention because it involves open aggression, including public verbal abuse, intentional exclusion from groups, punching or pushing, or other types of physical aggression. Indirect bullying involves spreading negative rumors or accusations about a person who is not present to defend him or herself, or indirect negative comments in the presence of the target (Lopes Neto, 2005).

There is often only a thin line between “normal” and healthy teasing between peers and behaviors that tend to be classified as bullying (Volk et al. 2012). In fact, bullying is understood as a social phenomenon rather than a psychiatric disorder (Lopes Neto, 2005). Nevertheless, studies have shown that bullying has a severe negative impact on academic performance (Webster-Stratton et al. 2008), with consequences that may extend into adulthood for both victims and perpetrators (Malta, et al., 2010).

Bullying is usually assessed based on self-report instruments (Kert et al. 2010). Therefore, the findings of a systematic review of 31 articles describing 27 self-report instruments used to evaluate bullying are a reason for concern (Vessey et al. 2014) – the review reports only “limited evidence supporting the reliability, validity, and responsiveness of existing youth bullying measures” (pg. 819). That finding challenges the usefulness of self-report to assess bullying (Vessey, et al., 2014). In this context, determining the validity and reliability of these instruments, that is, the extent to which they discriminate bullying from normative peer conflicts, is crucial to ensure that data reflect the trends of phenomena under observation. This is also true for translation and cultural adaptations of instruments to different languages.

Among the instruments cited in the systematic review by Vessey et al. (2014), the Revised Olweus Bully/Victim Questionnaire (OBVQ) is among the few with well-established psychometric properties in different countries (Kyriakides et al. 2006). The OBVQ contains two separate scales, one focusing on acts of victimization and one focusing on acts of bullying. The answers to each question are chosen from a multiple choice Likert scale, an aspect that has been criticized. According to Kyriakides et al. (2006), the use of a Likert scale “disregards the subjective nature of the data by making unwarranted assumptions” about the meaning of each choice because “the relative value of each response category across all items is treated as being the same” (p. 784). To circumvent this limitation, the authors propose the use of Item Response Theory (IRT), a mathematical method that analyzes the scores in relation to each other in order to evaluate whether the instrument is indeed capable of achieving its goal in a universal manner, that is, across different populations. Kyriakides et al. (2006) found that the OBVQ had satisfactory psychometric properties in a Greek sample (construct validity and reliability). That study also encourages the use of IRT to test other cultural adaptations of the OBVQ.

A Brazilian Portuguese version of the OBVQ, Questionário de Bullying de Olweus (QBO), is also available. The QBO contains 23 items that investigate the frequency with which individuals experience and/or engage in bullying behaviors 30 days before the survey (Olweus, 1996; Fischer et al., 2010). Subjects who experience or perpetrate any of the behaviors at least three times a month are classified as victims or bullies respectively. However, the psychometric properties of the QBO have not yet been determined.

In light of the above, the aim of the present study was to verify the construct validity and reliability of the bully and victim scales of the QBO using an IRT model.

This methodological study was approved by the Research Ethics Committee of the Federal University of Rio Grande do Sul, and by the Municipal Department of Health of the city of Porto Alegre (protocol number: CAAE 19651113.5.0000.5338). All parents and/or guardians provided written consent for the participation of their children in the study, and all adolescents signed an assent form prior to enrollment.

Firstly, the QBO with its four response categories was analyzed using the proposed IRT models, and the resulting performance curves were compared (Table 1). After that, analysis of the ICC led to the combination of response categories “Cerca de uma vez por semana/Around once a week” and “Várias vezes por semana/Several times a week.” That was true for all 23 items. Figure 1 shows the graphs corresponding to item 1 in both scales, modeled with four (Fig. 1a) and three alternatives (Fig. 1b). Because the curves corresponding to items 2 to 23 were very similar to the graph plotted for item 1, they are not shown.

Modelsa	Victim scale		Bully scale
	Four categories	Three categories	Four categories	Three categories
Generalized Partial Credit Model (GPCM)
a) Discriminating power = 1	94.9	87.1	87.6	77.1
b) Constant discrimination	98.0	94.8	98.2	97.3
c) Variable discrimination	98.6	93.4	98.5	97.8
Graded Response Model (GRM)
d) Constant discrimination	88.0	91.6	85.0	92.2
e) Variable discrimination	86.4	89.7	91.5	94.9

^aValues represent the area under the curve in %

Because the probability that an adolescent would select category 3 (“Cerca de uma vez por semana/Around once a week”) was zero in both scales of the QBO, both the victim and the bully scales were reformulated to include only three response categories: (1) “Nunca/Never”, (2) “Uma ou duas vezes por mês/Once or twice a month”, (3) “Uma ou mais vezes por semana/Once or more than once a week”. All 23 items were recoded accordingly.

After this change, participant scores were reanalyzed using the five mathematical models. The results of this procedure are shown in Table 1. For the bully scale, the largest area was enclosed by the GPCM with non-uniform discriminative power (area = 97.8 %). In the victim version, the largest area, by a small margin, was that enclosed by the GPCM with constant discriminating power (area = 93.4 %). The GPCM with variable discriminating power was selected as the best model for both versions of the scale for two reasons: firstly, the difference between its area and that of the GPCM with constant discriminating power was very small. Secondly, this model had been selected as the most adequate for this data set since the beginning of the analysis (area = 94.8 %).

The response parameters for item 1 following transformation are shown in Fig. 1c. The topmost curve indicates the most likely response by participants in different score intervals. As can be seen in Figure 1, in the first item of the victim version of the questionnaire, adolescents with total scores up to 34.9 were likely to respond “Nunca/Never” (Category 1); those with scores between 34.9 and 43.6 were likely to respond with “Uma ou duas vezes no mês/Once or twice per month.” Finally, subjects with scores above 43.6 were most likely respond with “Uma ou mais vezes por semana/Once or more than once a week”. The same results were observed in the first item of the bully scale.

Once final scores were developed for the three-category version of the scale, using the aforementioned discriminating and severity parameters, the mean (standard deviation) of victim scores was 29.3 (SD = 5.39). The reliability (Cronbach alpha) of this scale was α = 0.85. The bully scale had a mean score of 26.8 (SD = 3.92) and a reliability of α = 0.87. The reliability of each item is shown in Table 2 (victim scale) and Table 3 (bully scale).

Unidimensionality analysis revealed that the first factor of the victim QBO scale explained 26.27 % of the variance, whereas the first factor of the bully QBO scale explained 31.05 % of the variance. A full Brazilian Portuguese version of the validated QBO appears in Additional files 1 and 2.

The aim of the present study was to determine construct validity (using IRT) and reliability of the QBO. The findings showed satisfactory validity and reliability for both bully and victim scales of the QBO.

Given the complexity associated with the assessment of bullying, and the lack of validated instruments to evaluate this construct, the use of IRT to investigate the construct validity of both scales of the QBO, define the adequate number of response categories, and verify item discriminating power and severity was an important contribution to the literature. A recent review of 25 Brazilian articles found that in most studies involving the assessment of bullying, this phenomenon is identified using measures developed by the researchers themselves or with unknown validity for the Brazilian populations. The authors concluded that the absence of validated instruments for this purpose is a significant methodological limitation (Alckmin-Carvalho, et al., 2014). The use of IRT to determine construct validity is useful to assess latent traits, such as anxiety level, stress, and quality of life, which correlate with different items in an assessment measure. A relationship is expected between the presence of a particular condition and certain latent traits (Andrade, et al., 2000; Sartes & Souza-Formigoni, 2013).

The present results revealed the need to combine response categories 3 and 4, so that only three response categories were kept in both scales (victim/bully) of the QBO: (1) “Nunca/Never”, (2) “Uma ou duas vezes no mês/Once or twice a month”, (3) “Uma ou mais vezes por semana/Once or more than once a week”. Although some items could be further modified to include only two response categories, the three alternatives were maintained for all items to ensure uniformity between the bully and victim scales. The presence of multiple categories allows for an estimation of behavior frequency, which is especially important since repetition is a core feature of bullying (Malta et al., 2010). Thus, use of the IRT model confirmed that the behaviors measured by the scale are expressions of the underlying construct, and also allowed us to determine the performance of each item of the QBO construct for Brazilian adolescents (Andrade et al., 2000; Pasquali, & Primi, 2003). As previously mentioned, a Greek study employed a similar model to evaluate construct validity and reliability of a cultural adaptation of OBVQ. That study also found satisfactory psychometric properties for both victim and bully questionnaires (Kyriakides et al. 2006).

Our findings also revealed that the items in the QBO differ in their loading to the latent variables in question. In this population, being the object of hurtful comments, persecution, or threats had high power to discriminate victims of bullying. Conversely, forcing people to be physically aggressive to others, persecuting students inside or outside the school, and issuing threats were most likely to identify bullies. These results are in line with the defining feature of bullying, which is the intention to humiliate, threaten, and harm (Olweus, 1996, Berger, 2007).

The items with the least discriminant ability for bullying victims were: being teased and being forced to hand over money or belongings, or having those taken without consent, and being humiliated in association with skin color or ethnicity. The least discriminating items in the bully scale were damaging the belongings of others and using the Internet to hurt others (cyberbullying). The fact that being teased figures among the least discriminative items for bullying victims suggests that this type of behavior may be interpreted as a friendly exchange between peers rather than an attempt to cause harm or humiliate (Volk et al., 2012).

Discriminating power is used to indicate that item estimates will remain relatively constant in future applications (Sartes & Souza-Formigoni, 2013). Concerning the QBO, that means that items with more strength to discriminate victims or bullies in our culture would be useful to assess bullying in schools in other samples of Brazilian adolescents.

The severity parameter is related to another central characteristic of bullying – the frequency of behaviors (the higher the severity parameter, the more frequent the behavior). For both, bullies and victims, the highest severity parameters were observed for direct bullying items; for bullies, the highest severity parameters were recorded for “I snitched money or things from others,” “I used the Internet or cell phone to harm/offend a classmate,” and “I sexually harassed someone”. For victims, the highest severity parameters were observed for “I was forced to hand over my money or belongings,” “I was sexually harassed,” “I was forced to physically harm a classmate,” and “I was humiliated because of my sexual preference of mannerisms”. Also, the results show satisfactory reliability of the final scores, with α > 0.85 for both the victim and bully QBO scales.

The present study had some limitations. Although the replication of our method by other researchers is extremely desirable, we were unable to develop a syntax of our procedures for use in other statistical packages. Additionally, we did not provide a cutoff for the classification of bullies or victims. Nevertheless, the scores obtained by other samples on the victim and bully scales of the QBO can be calculated using IRT parameters estimated from our original data through the interactive method and tutorial available on the website www.professor.ufrgs.br/eheldt, in files model_vit.Rdata and model_agr.Rdata.

We found that simply adding up the scores on all items of the QBO without considering the relative weight of each item may interfere with the validity of this measure and, consequently, with the findings of studies which use the traditional versions of the QBO. Given the relevance of this topic, it is important that future studies continue to investigate the psychometric properties of this instrument, using factor analysis, for instance, to verify whether additional dimensions of bullying (e.g. direct and indirect bullying) can be identified using the QBO. Future studies focusing on the development of effective tools to identify and define the types of bullying behavior present in different samples will be essential to guide the implementation of prevention programs targeting bullying in school environments.

We found that simply adding up the scores on all items of the OBVQ without considering the relative weight of each item may interfere with the validity of this measure and, consequently, with the findings of studies which use the traditional versions of the OBVQ. Given the relevance of this topic, it is important that future studies continue to investigate the psychometric properties of this instrument, using factor analysis, for instance, to verify whether additional dimensions of bullying (e.g. direct and indirect) can be identified using the OBVQ.

Future studies which develop effective tools to identify and define the types of bullying behavior present in a given sample will be essential to allow for the implementation of prevention programs targeting bullying in school environments.