Earlier this week, a team of academic staff from the Faculty of Medical Sciences and the Institute of Education (IOE) facilitated an AI assessment hackathon, aiming to investigate innovative ways of incorporating language models into assessments. The project was made possible through a small education grant provided by the Competition for Funding Education Enhancement (CofFEE!) initiative at UCL Faculty of Medical Sciences. Grants ranging from £500 to £2000 (£3000 total available) were allocated to multi-division projects focusing on future education.

Planning and Information Collection

To kickstart the project, the team distributed an online questionnaire to students across the faculty, garnering over 90 anonymised responses. Additionally, a student-only focus group, consisting of 13 undergraduate and postgraduate students, was conducted to brainstorm ideas on how students envisioned AI being incorporated into assessments. A recurring theme among the student volunteers was the desire for universities to consider how graduates could utilise AI in the workplace. Students emphasised the importance of aligning teaching and assessment methods with real-world practices, recognising the inevitable use of AI in professional settings. While students acknowledged the risks of AI models generating inaccurate or biased information, they highlighted its value in organising arguments, formatting text, and enhancing visual presentations.

Given the high probability of AI adoption by future graduates, it becomes crucial for both students and teachers to understand the benefits and limitations of AI. While it may not be feasible to restrict the use of AI in home-based assessments, it is imperative to comprehend how students currently employ available AI platforms and their perceptions of AI strengths and limitations. Furthermore, assessors need to grasp how AI usage might influence the final output of assessments and the development of students’ transferable skills.

Hackathon Day

On Monday, 26th June, an all-day hackathon took place from 9 am to 5 pm, involving ten Level 6 and 7 students. The participants were divided into teams of two, with each team assigned a specific task. Throughout the day, the students were provided with snacks and lunch, and the winning team was promised an award of £300 per person. Staff members were designated as judges for the afternoon presentations. In the morning, the students received an introduction to AI and AI tools – the primary ones being Chat-GPT, Google Bard, Evidence Hunt and Tome, followed by pre- and post-hackathon questionnaires to gauge their attitudes towards AI and their previous experience using AI. Subsequently, they were given a question relevant to the field of medical sciences, accompanied by common misconceptions, which formed the basis of their assigned task.

The task is given below:

Task A- Topic Selection  
Choose one of the following: 

- What causes obesity?  
- Is being an elite athlete healthy?  

Task B- Public Belief  
Find out what the public thinks and believes about your chosen topic.   

Task C- Scientific Evidence  
Time to explore the literature and learn what the experts in the field have reported about your chosen topic.   

Task D- Comparing Tasks B and C  
Compare the outcome of your search results for the previous two tasks: Task B and C.   

Task E- Report Writing  
You are now requested to write a 500-word document in scientific language to report your findings from the above tasks. Upload your report on Teams’ space provided. Please ensure to include your reference list at the end and structure your report in a way that your group feels is more appropriate. References are not included in the word count and please use the Harvard or Vancouver style of referencing. Your report will be read and judged by a panel of academics. 

Task F- Presentation  
Prepare and deliver a 3-minute presentation pitched to a lay audience. You are free to use any form of visual that your group select. Your presentation will be judged by a panel of academics.  

Throughout the hackathon, data was collected on how the students approached each of the five tasks, utilising a brief questionnaire. The questionnaire included the following details: Whether they employed AI for each step and if so which AI tools were used and how were they utilised. Data was collected on how students approached each of the 5 steps using AI, for example was the AI tool useful and did it provide accurate information? Each group was also asked to submit prompts used.

At the end of the day each group gave a quick presentation on their work and judges provided a mark for the written scientific summary (submitted anonymously) and for the lay communication piece. The grading form covered Literature searching (document and presentation); knowledge and understanding (document and presentation); critical appraisal (document and presentation); scientific communication (document); lay communication (presentation) and visual representation (presentation).

The presentations and scientific summaries were incredibly varied with some students going ‘above and beyond’ with their use of AI in both their research and delivery (there was even an AI generated Barack Obama introduction!), while others opted for more conventional methods such as for reframing text and summarising content.  AI often illustrated the rather blunt and less-nuanced public understanding in quite a dramatic way – for example who would have known that when provided only with the keyword “Obesity” the tool Midjourney would create a picture of a large man surrounded by doughnuts. AI also supported with processing large volumes of data and allowed students to quickly and efficiently navigate through extensive research materials.

First prize – Muyiwa Aruna and Cameron Dent

The winning prize was awarded to the team who used ChatGPT to perform sentiment analysis on public tweets related to overeating and to sift through research content, and concluded their presentation with segmented images of the snacks in the room using Segment Anything – a tool which categorises objects within image.

Second  prize – Edoardo Beltazar and Praveen Mathiventhan

Second prize was awarded to the team who used Chat-GPT to support their research but focused on clear structure and scholarly referencing their presentation.

Video case study

In the following video case study (13 minutes 16 seconds)  Dr Flaminia Ronca, Associate Professor (Teaching), Department of Targeted Intervention, Div of Surgery & Interventional Sci, talks about how the rationale behind the hackathon, what the main learnings have been and what the team plan to do next.

“We already know that a good student will produce a good piece of work…but if we can provide them with additional skills, teach them what tools are available, how to write prompts and how to use these tools effectively, then this could enable them to produce something very creative and exciting. And it can add real value to what they’re delivering.“

What next?

The day provided some highly valuable insights into student use of AI for assessments, for example challenges around AI literacy, equity and access to tools (some students had paid for accounts while others didn’t) and clarity of AI requirements for assessment tasks.  The next steps are for the team to write up a case study and academic paper which will include the following:

• A flow diagram that shows if students chose to implement AI at each step of the process, which tool they used, how they used it, and what is was/wasn’t useful for. 
• Evaluation on whether the use of AI throughout the process has an impact on the marks obtained. 
• Evaluation on whether attitudes towards AI changed before and after the Hackathon