Search:
Does participation in Football Beyond Borders improve the mental wellbeing of at-risk young people?
What is Football Beyond Borders?
Football Beyond Borders (FBB) is a targeted, school-based social and emotional learning (SEL) intervention. Young people who participate in the programme are considered at-risk by their school, because of concerns about behaviour, a history of fixed-term or permanent exclusions, socio-economic disadvantage, adverse childhood experiences, and/or special educational needs. FBB also works with passive learners (those predicted not to meet national expectations in English and Maths, emerging behavioural issues, low homework completion rates, and/or special educational needs) and role models (top 20% academic achievement, play for school football team, and/or predicted to exceed national expectations in English and Maths).
The programme is run throughout the school year and includes: (i) weekly project-based classroom SEL sessions, implemented alongside weekly pitch-based football activities by a youth practitioner; (ii) weekly 1:1 sessions for those requiring therapeutic support led by a counsellor; (iii) engagement with parents/carers to build a picture of home life, provide updates on progress in FBB, and identify ways in which participating young people can be better supported; and, (iv) reward trips, including visits, residentials, and work experience opportunities. The FBB theory of change suggests that these activities improve young people’s social and emotional skills, mental wellbeing, behaviour and attendance via mechanisms of change that include the development of consistent and long-term relationships, a sense of belonging to the FBB group, an engaging and relatable curriculum, and an asset-based approach. You can read more about FBB here.
What did we do?
Our aim was to provide a preliminary robust independent evaluation of the efficacy of FBB in improving the mental wellbeing of at-risk young people, drawing on the #BeeWell dataset. To do so, we used difference-in-differences (DiD) estimation and propensity score matching (PSM). DiD is a framework for analysis that can be used to test the effects of an intervention when it isn’t possible to conduct a randomised trial. PSM is a statistical technique that can be used to ensure that an intervention and comparison group in an evaluation are well matched. In combination, DiD and PSM enable us to rigorously test the effects of interventions like FBB.
FBB securely shared mental wellbeing data (assessed using the SWEMWBS measure, which we also use in #BeeWell) for 46 at-risk young people, 72 passive learners and 35 role models collected prior to their participation in the programme and one year later, in addition to socio-demographic information (e.g. gender). Using PSM, we identified a well-matched comparison sample for each of these three groups from within the #BeeWell longitudinal cohort. Having done so, we were able to compare changes in mental wellbeing over one year among young people who had participated in FBB, compared to young people with very similar characteristics who had not participated in the programme.
What did we find?
For at-risk young people, we found a positive, statistically significant impact of FBB on mental wellbeing, equivalent to just under 2 and a half points on the SWEMWBS scale (an effect size of d=0.44, or a 17-percentile point increase). To check the robustness of this effect, we ran a series of additional checks (e.g. using different PSM approaches to the one we had chosen for our main analysis). In most cases, this did not substantively change the intervention effect we had identified, leading us to conclude that FBB improves the mental wellbeing of at-risk young people.
We also ran analyses to assess whether FBB improved the mental wellbeing of passive learners or role models, but these tests did not identify any intervention effects.
Building on the above, our colleagues at Pro Bono Economics performed an evaluation designed to assess the economic benefits of FBB. Using #BeeWell data, we provided them with estimates of how the observed gains in mental wellbeing would translate into improvements in life satisfaction, and thus, ‘wellbeing years’ (known as WELLBYS), which can be monetized (see here). Pro Bono Economics performed analyses using these data, alongside information about the costs of delivering FBB, ultimately determining that for every £1 spent, the programme provides £2.20 of wellbeing benefits. You can read their report here.
What does this mean?
Our analysis provides preliminary robust and independent evidence that FBB improves the mental wellbeing of at-risk young people. The size of the intervention effect we observed compares favourably with the average effects reported for other targeted school-based interventions in a recent review, and is within the range noted as meeting thresholds for statistically important change in a responsiveness evaluation of the SWEMWBS measure. Overall, this means that the improvements in mental wellbeing brought about by FBB are likely to be meaningful and practically significant. The economic analysis performed by Pro Bono Economics also indicates that FBB is likely to offer good value for money.
Several next steps are recommended. First, we need to know if FBB also improves social and emotional skills, behaviour, attainment and attendance as these are key outcomes in the programme’s theory of change. Second, we need to better understand how and why FBB works, through rigorous evaluation of potential ‘treatment effect moderators’ (e.g., the number of sessions attended by young people). Third, we need to determine if the impact of FBB on mental wellbeing observed in our study is maintained in the longer term.
Though our evaluation had numerous strengths, including the use of DiD, PSM, and our additional checks to test the robustness of the intervention effect, there are also limitations that need to be borne in mind. First, the available FBB samples were relatively small, limiting statistical power to detect intervention effects. This may have impacted on the analyses for passive learners and role models, in which might reasonably expect a much smaller intervention effect, requiring a considerably larger sample than was available in order to be detectable. Second, although the intervention effect identified in our main analysis was not substantively different in most of our additional checks, there were some exceptions, which reinforces the need for caution and replication in a large, randomised trial.
This evaluation showcases the utility of the #BeeWell dataset in helping to understand how best to support young people’s wellbeing. We hope to undertake similar evaluations of other interventions in the future.
You can read more about our study here.
