Abstract
The UCL Careers Extra Internships Scheme was launched in 2023 to provide ring-fenced, funded internships with charities and SMEs for undergraduates from under-represented groups. Integral to the scheme is a programme of wraparound support which includes one-to-one discussions pre-internship to identify areas for skills development, follow-up reflections on skills learned, and finally the opportunity to present to others, simulating the experience of speaking at an interview about previous experience and skills gained. Alongside enabling students to make the most of their internships, these activities yield rich evaluation data, thus integrating evaluation within the skills development activity.
This post explores the Theory of Change framework for the scheme, outlines the evaluation process undertaken and showcases individual students’ experiences to bring to life the impact of their internships.
Aims of the scheme
Research shows that progressing quickly into employment post-graduation impacts individuals’ lifelong earnings and outcomes. Internships and other work experience make a critical difference in the competition for graduate roles. However, internships are in short supply, and students with high levels of social, cultural, and/or economic capital have an advantage in the application process. Students from advantaged backgrounds are also over-represented in many professions.
The aim of the UCL Careers Extra Internship scheme is to support social mobility by enabling students from under-represented groups to access quality internships, building their skills and confidence, and contributing to their competitiveness when applying for graduate roles. This approach builds on evidence that internships and work experience are effective interventions to improve equity of access to graduate roles.
The scheme has also been designed to support employers in the target sectors — namely, small and medium-sized enterprises (SMEs) and charities — to improve diversity within their organisations and, through this, to pave the way for greater equity and improved opportunities for students from under-represented groups.
Our evaluation framework
The Careers Extra internships were funded by a generous donation from an alumnus who expressed a keen interest in the evaluation of the impact of the scheme, prompting us to give extra consideration to how we could embed as many evaluation points into the scheme as possible. This was especially important as evaluating the direct impact of Careers interventions is notoriously difficult for a number of reasons, including:
- Multiple factors feed into students’ successful progression into graduate work, and it is not usually possible to reliably attribute success to a particular intervention; therefore, a simple measure of graduate progression is limited in terms of evaluating the impact of an individual intervention.
- The length of time that may elapse between an intervention and the collection of graduate outcomes data inhibits timely conclusions of impact and contributes further to the difficulty of evaluating the impact of any one particular thing.
- Response rates from students after they have finished an experience/activity tend to be low.
- Ensuring the accuracy and validity of feedback gathered from students grateful for an opportunity is problematic (it is very difficult to craft questions and mechanisms which do not encourage grateful students to give answers they think are ‘expected’).
Working with the Evaluation team within Access and Widening Participation, we first drew up a Theory of Change framework for the intervention, which is shown below.
ToC for Careers Extra internships scheme (revised)
We then studied the TASO (Transforming Access and Student Outcomes in Higher Education) report on Impact Evaluation with Small Cohorts: Methodology Guidance to select a methodology, concluding a Comparative Case Study approach would be most suitable. We had to adapt to suit our circumstances as our cohort was small even compared to those discussed in the report – we were aiming for 7 to 10 internships in this, our pilot year, although we finished up with 15.
Building on a process previously used with a Careers Extra work-based experience bursary scheme we built pre- and post-internship evaluation meetings into the scheme. We created these as a learning opportunity to help the students make the most of the internship and then to help them reflect and articulate their learning in order to feed into future applications. For these discussions we used a template combining free text discussion of what students hoped to gain from the experience/felt they had gained from it with the more quantitative measure of students scoring themselves before and after the internship on the skills from the UCL Pillars of Employability
A third question was used to try to assess and progress in students’ ‘career decision-making stage’ as defined through the Careers Registration questions they answer when re-enrolling each year.
We also asked each student to follow up the one to one discussion by providing us with a brief write-up of their experience and their learning, using a template we provided. This not only generated additional data to feed into a bid for future funding but also provided a further opportunity for them to practise articulating their learning (this time in a written rather than oral medium) as well as creating content to advertise the value of the scheme to future students. A final learning and evaluation activity was a presentation evening, where the students were asked to present briefly to each other, giving them the opportunity to practise speaking in a supportive environment about a work-based experience and a skill they had strengthened – as they might be asked to do in a future interview.
To triangulate the data collected from students we also surveyed employers near the start and at the end of the internships, asking them to score the students on the same set of skills from the Pillars of Employability, as well as providing the opportunity to give more qualitative feedback. We also asked questions designed to evaluate what impact the scheme had had on employers themselves, particularly in relation to their own equity, diversity and inclusion practices.
Challenges
The employers found it difficult to assess students early in their internships when they hadn’t had long to observe their work. This made it difficult to use their evidence to measure students’ progress.
The third question used with students pre- and post-internship proved less useful than the other measures, because ‘progress’ indicated by their responses was partly attributable to the different stage they had reached in their degree.
Devoting an hour of one-to-one discussion (30 minutes each pre- and post-internship) to each student is resource-intensive and therefore unlikely to be scalable to a larger scheme. Nevertheless, for the individual students it provided a valuable opportunity to engage with UCL Careers and through doing so, to develop their own skills of reflection and articulation – the over-arching focus of the UCL Pillars of Employability.
What worked well
The internships went well! The students’ skills self-assessments changed significantly between the pre- and post-internship discussions, with a strong trend of improvement. Whilst the employers’ scoring of the students’ skills was less useful than we’d hoped because of the difficulty of providing a ‘base-line’ assessment near the start of the internship, their qualitative comments about the students were overwhelmingly positive. Over 90% of the employers said the interns produced tangible work which was useful to them. Employers also confirmed that taking part had given them insight into the skills they could expect from a UCL undergraduate at this stage in their degree, they gained insight into the lived experience of a student from an under-represented group, it helped them establish a model for offering a similar internship in future and their intern brought new perspectives/taught them new skills.
We are confident our approach of embedding the data collection with learning opportunities maximised both quantity and quality of data gathered from both students. All but one student engaged with the one-to-one discussions pre- and post-internships and 12 out of 15 followed this up with a write-up. This in-depth approach gave us so much more insight than we could have gathered through a survey/feedback form, while also adding to the students’ learning.
We were also successful in eliciting feedback from a majority of employers, which runs counter to our usual experiences with employers offering internships. We received feedback at the start and end of 13 out of the 15 internships.
Students who took part in the final presentation evening spoke coherently about their experiences, with many of them naming presenting as a skill they were grateful to have developed through the scheme. They also spoke eloquently about their increased confidence. This short video from the event gives a flavour of what they said:
Over 90% of the employers said they would be interested in being part of the scheme again (subject to business capacity at the time). We were also particularly encouraged that over 90% of them also said the scheme’s focus on students from under-represented groups was a reason they were attracted to it.
This comment from one employer addresses many of our aims from the scheme:
“Working with a UCL graduate through this scheme is incredibly rewarding for team members. UCL interns bring energy and enthusiasm, along with creative thinking and ‘can-do’ attitude which is motivating for employees at all levels. It is great to see cross fertilisation of ideas and learnings for all that are involved in the programme.
We…especially love the challenges UCL interns put forward when working on projects that help us re-evaluate how to approach existing processes and projects. Through working with UCL interns we also become more aware of some of the challenges they face and how we could accommodate and support those starting out in their careers when they start work, especially if they are from an under-represented demographic in the workforce. This also provides ‘at the coalface’ experiences that we can share with other organisations we work with so that collectively we build more inclusive working environments.”
Hear more about the scheme and the employers’ responses to it in this video from the final presentation evening:
Next steps
Our plan was always to run the scheme on a similar scale in 2023–24. With a further donation from the same alumnus, we have been able to expand it slightly and are now offering 19 internships. Now that our data has been analysed we are working with OVPA to prepare materials which can be used to encourage other philanthropic donations, with the aim of both expanding the scheme and consolidating its funding for future years.