Using AI tools for Science Communication projects in Natural Sciences

Dr Amy Unsworth and student Ben Vengerik share the findings from their project on using AI tools for student Science Communication projects in Natural Sciences.

They discuss:

  • Auditing AI tools for image generation
  • The pros and cons of two tools in particular (Midjourney and Bing)
  • Implications for education and supporting students
  • Equity of access to tools
  • Legal and ethical issues around image generation

 

What is gained and lost from the writing process when using AI tools?

Peter Puxon, Ayanna Prevatt-Goldstein and Jessica Brooks in conversation with their three ChangeMakers Co-Creator students Anenyah Venkatesan, Zsofia Varga and Yishan Li. They reflect on what is gained and lost from the process of writing and reading after engaging with AI tools to work on an assignment.

This project was from UCL’s Academic Communications Centre.

Slides presented at the ChangeMakers Lunch & Learn event here: SLIDES AI Co-Creator – ACC P Puxon J Brook + A Prevatt-Goldstein

How and why we’re co-creating the response to the use of AI at UCL

When we, the ChangeMakers team, were told there was some funding available for us to structure an approach for staff and students to work together on our response to AI, I knew we’d say yes. I knew next to nothing about AI and was vaguely aware of some of the more panicky and attention-grabbing headlines about the end of higher education and how ‘lazy’ students are getting AI to write their assignments and graduate from uni. In fact, the opportunity to provide an antidote to these enormously unhelpful statements through this funding was the real driver for me. This polarising of staff and students into an ‘us’ vs ‘them’ dynamic is absolutely antithetical to the values of student-staff partnership and damaging to building a strong community and sense of belonging that we know are so vital in a thriving university culture. No one can feel like they belong to a community where members of that community are suspicious they could be cheating.

Co-Creation was always going to be at the heart of our approach. As noted by a member of staff partner running a ChangeMakers project last year:

Our co-creation approach established a strong sense of community and trust between students and staff within [the department] which allowed us to have open dialogue about the challenges which students face.”

These ideas of trust, community and open dialogue, at a time when panic around AI could potentially drive a wedge between staff and students, are essential to these projects. Yes, of course, it is important that they expand our knowledge and understanding of AI, but more important is the idea of learning and responding together as a whole community. So we developed AI Co-Creator projects.


Why Co-Create with students?

  • It is an inclusive approach, especially if we are intentional about whom we co-create with.
  • It is an expansive approach, drawing on the needs, perspectives, knowledge and experiences of both staff and students.
  • Students are empowered to take more responsibility for their learning.
  • You will learn more about who your students are and they gain a deeper understanding of how a university works and how decisions are made.
  • Creates a community approach to problem-solving key educational challenges.

It was important to strike a balance between providing some structure and ideas so these projects could get up and running quickly, but also with enough flexibility to allow individuals to shape the projects to their needs. Ideally, we also wanted to ensure students would be able to shape the projects once they were recruited, in the true spirit of co-creation. In an ideal world, students would have been involved in designing projects with staff from the outset but, unfortunately, time constraints limited us there. But for anyone with time on their side, the value of co-creation throughout the whole project process is definitely something we would recommend.

We developed four themes in areas that were of current interest to students and staff: assessment, feedback, learning support and exploring AI. Within those themes, we also offered a ‘menu’ of ideas for staff to pick up and adapt. This meant that those who wanted to do some learning with students but didn’t know where to start had a place to begin but didn’t limit those who had a clear idea. This seemed to work well. We received 67 applications from staff and have ended up funding 63 of them. We also had a huge amount of interest from students. Staff who recruited their own students reported higher-than-expected interest in their projects and we had around 125 expressions of interest from students who were really keen to get involved. I think the fair payment of a £600 stipend for around 40 hours worked probably helped but that just proves that if students are properly rewarded for their contributions, they are ready and willing to work with us. And their applications were thoughtful about AI and hummed with excitement to get stuck into helping us respond as an institution. To me, these projects are a sign that student-staff partnerships are flourishing at UCL and they offer us a mechanism to be more resilient to change in a way that involves everyone in finding solutions.

Developing AI literacy – learning by fiddling

Despite ongoing debates about whether so called large language models /generative language  (and other media) tools are ‘proper’ AI (I’m sticking with the shorthand), my own approach to trying to make sense of the ‘what’, ‘how’, ‘why’ and ‘to what end?’ is to use spare moments to read articles, listen to podcasts, watch videos, scroll through AI enthusiasts’ Twitter feeds and, above all, fiddle with various tools on my desktop or phone. When I find a tool or an approach that  I think might be useful for colleagues with better things to do with their spare time I will jot notes in my sandpit, make a note like this blog post comparing different tools or record a video or podcast like those collected here or, if prodded hard enough, try to cohere my tumbling thoughts in writing. The two videos I recorded last week are an effort to help non-experts like me to think, with exemplification, about what different tools can and can’t do and how we might find benefit in amongst the uncertainty, ethical challenges, privacy questions and academic integrity anxieties.

The video summaries were generated using GPT4 based on the video transcripts:

Can I use generative AI tools to summarise web content?

In this video, Martin Compton explores the limitations and potential inaccuracies of ChatGPT, Google Bard, and Microsoft Bing chat, particularly when it comes to summarizing external texts or web content. By testing these AI tools on an article he co-authored with Dr Rebecca Lindner, the speaker demonstrates that while ChatGPT and Google Bard may produce seemingly authoritative but false summaries, Microsoft Bing chat, which integrates GPT-4 with search functionality, can provide a more accurate summary. The speaker emphasizes the importance of understanding the limitations of these tools and communicating these limitations to students. Experimentation and keeping up to date with the latest AI tools can help educators better integrate them into their teaching and assessment practices, while also supporting students in developing AI literacy. (Transcript available via Media Central)

 

Using a marking rubric and ChatGPT to generate extended boilerplate (and tailored) feedback

In this video, Martin Compton explores the potential of ChatGPT, a large language model, as a labour-saving tool in higher education, particularly for generating boilerplate feedback on student assessments. Using the paid GPT-4 Plus version, the speaker demonstrates how to use a marking rubric for take-home papers to create personalized feedback for students. By pasting the rubric into ChatGPT and providing specific instructions, the AI generates tailored feedback that educators can then refine and customize further. The speaker emphasizes the importance of using this technology with care and ensuring that feedback remains personalized and relevant to each student’s work. This approach is already being used by some educators and is expected to improve over time. (Transcript available via Media Central)

m
m
m
I should say that in the time since I made the first video (4 days ago) I have been shown a tool that web connects ChatGPT and my initial fiddling there has re-dropped my jaw! More on that soon I hope.

 

A worrying mix of truths, half truths and plausible BS

Plenty of people have offered examples of some of the weird nonsense ChatGPT and other AI text generators have been spewing recently, but I am not convinced everyone is able to fully appreciate how this odd mix of truths, half truths and utter claptrap may be more hindrance (especially in the immediate future) than blessing for anyone looking for an easy shortcut for written assignments (or written anything to be honest!) Whilst I have found ChatGPT remarkable in some ways, such as baseline planning, idea generation, synthesis of given text or re-formatting as can be seen in this sandpit , the big fear, the supposed ‘death of the essay’, is a long way off*. ChatGPT generates some great ideas for essay titles,  a decent enough standard marking rubric and some workable boilerplate feedback but it’s actual attempt to write those essays and then assess them against the rubric it created leads to some real generosity in interpretation of what constitutes a passing standard at undergraduate level. It does make me wonder about the actual requirements of these qualifications it could supposedly pass, to be honest. The now (sort of) familiar, anodyne but mostly grammatical offerings seem all the more plausible when you don’t already know the thing it is generating text about. So, I thought, what do I know a lot about? After considering a range of academic subjects that might make me look intellectual, I realised with Socratic clarity that my head is practically a vacuum. I decided therefore to get it to write an essay about something I might be happy to be quizzed about on Mastermind: Tottenham Hotspur’s FA Cup final wins. At school I did a presentation on this topic and, although it was longer and more boring, I’m pretty sure I got my facts right. In addition to a number of omissions (e.g. the 1967, 1981 and 1982 are finals missing), it also makes stuff up but surrounded as these things are by things that are accurate it all feels plausible. I share it below as an example of just how flawed an essay can be on oft-cited, indisputable facts.

I have emboldened some of the inaccuracies to illustrate this and [added comments if relevant in square brackets]

Prompt: Write an essay about Tottenham Hotspur’s FA cup victories, dedicating a paragraph to each one

Tottenham Hotspur is one of the most successful football clubs in the history of English football. Over the years, they have won numerous domestic and European titles, but one competition that holds a special place in the hearts of Spurs fans is the FA Cup. In this essay, I will take a look back at Tottenham Hotspur’s FA Cup victories and dedicate a paragraph to each one.

Tottenham’s first FA Cup triumph came in the 1900-01 season. After reaching the final for the first time in the club’s history, Spurs faced Sheffield United at Crystal Palace in front of a crowd of 110,820 [so far , so good, accurate crowd and venue…it’s looking good]. The game was a closely contested affair [true- it finished 2-2], but a late goal from striker Steve Bloomer [did not play, was in neither team and certainly therefore did not score] secured a 2-1 victory for Tottenham [actually the game finished 2-2. In the replay Spurs won 3-1]. The win made them the first [and only – an important omission] non-league team to lift the trophy, and it marked the beginning of a long and successful relationship between Spurs and the FA Cup.

sepia image of 6 robots in a team group with a football at the feet of one of them

The next FA Cup success came in 1921, when Spurs faced Wolverhampton Wanderers in the final. The match was played at Stamford Bridge, the home of Chelsea, and once again, Tottenham emerged victorious. A brace from Jimmy Dimmock and a goal from Tommy Clay [He did play but didn’t score; Jimmy Dimmock got the only goal] secured a 3-0 win for Spurs, and they lifted the FA Cup for the second time in their history.

In 1961, Tottenham won their third FA Cup, and this time, they did it in style. After cruising through the early rounds, Spurs faced Leicester City in the final at Wembley. The match is often remembered as one of the greatest FA Cup finals of all time, and it saw Tottenham emerge as 2-0 winners, thanks to goals from Bobby Smith and Terry Dyson. The victory marked the start of a golden period for the club, and they would go on to win a league and cup double the following season.

Tottenham’s fourth FA Cup triumph came in 1962, just a year after their first success. This time, they faced Burnley in the final, and the match was played in front of a crowd of 100,000 at Wembley. Once again, Spurs emerged victorious, thanks to a brace from Bobby Smith and a goal from Terry Medwin [right number of goals, wrong goalscorers]. The win made Tottenham the first club in the 20th century to win the FA Cup twice in a row [sounds very plausible but Newcastle did the same in ‘51 and ‘52- something they can only dream of now, like Spurs]

Tottenham’s most recent FA Cup triumph came in 1991, when they faced Nottingham Forest in the final. The match was played at Wembley, and Tottenham secured a 2-1 win, thanks to goals from Paul Gascoigne and Gary Lineker. The victory was particularly sweet for Spurs fans, as it marked the end of a long wait for silverware. The club had gone six years without a major trophy, and the FA Cup win was seen as a turning point for the team.

In conclusion, Tottenham Hotspur’s FA Cup victories have played an important part in the club’s history. From their first success in 1901 to their most recent triumph in 1991, the FA Cup has provided Spurs fans with countless memories and moments of joy [‘moments’ being the most painful of accurate things included]. While the club has not lifted the trophy since then, they remain one of the most successful teams in the competition’s history, and fans will always cherish the memories of those five famous victories.[what about the other 3?!]

—————–

So that’s at least 10 major factual errors and several errors by omission in only 600 words. Relying on a tool to generate text and assuming accuracy is still very much NOT the best way to use these tools (at present). Nevertheless, how long before such things are pumped out on blogs or elsewhere and become the ‘truth’? Or, better /worse still (depending on how you look at it), maybe we are at the foot of the rise on a sigmoid curve and in 6 months all this nit picking will be a laughable relic.

 

*and even when capabilities improve faster than you can say ‘Spurs last won the league in black and white’ I am still very much of the school of thought that thinks it is worth teaching (and/or learning) how to write an essay!

AI: Shorthand and eponyms

One of the things we do badly and get agitated about a lot is the naming of things to do with technology; not least in the realm of digital education. Just ask a room of HE folk for a definition of ‘Blended’ and ‘Hybrid’ and wait for saloon brawl to ensue. So it is with some of the language that is emerging as default in relation to all things ‘Artifical Intelligence’, notably and especially the large language model (LLM) ‘ChatGPT’.

Shorthand

I do understand when folk get antsy when something is called a thing that it isn’t, isn’t exactly or isn’t entirely. But, unless you’re a specialist, a pedant, or willfully awkward (and granted a lot of academics are at least two of these- no offence intended), we may as well get on with saying strawBERRIES, peaNUTS and TIN foil even if they are no such thing. In that vein, I am more than happy to use ‘AI’ as a shorthand for all the stuff that is vexing, perplexing and flexing educators just now. I’m afraid that if someone starts saying: ‘Well technically, it’s not artificial intelligence per se, but rather a fine-tuned large language model…’ I can feel my eyes glazing over. Obviously this stuff is fundamental if you are a computer scientist but the reason such shorthands exist is that they are short, handy (clues are in the word) and suitable for lay users. Experts that know this are one step closer to communicating beyond their discipline.

a mechanical strawberry as imagined with midjourney AI image from text

Eponym

I am much less comfortable with some brand eponyms; especially when the tools and the words to describe them are still evolving. Against my better judgement, I ‘Google’ stuff even if I’m in Safari and I use a ‘hoover’ (even though the only actual- and long discarded- Hoover I ever owned metaphorically ‘sucked’ at the literal thing it was supposed to do). But I am pushing back at the generic use of ‘ChatGPT‘ (OpenAI’s LLM) to refer to the latest iteration of large language models. Chatbots have been around for years and the underpinning technology has evolved rather than suddenly appeared but the genius of the release and the subsequent explosion in use and interest is in the non-techy, almost friendly user interface combined with the now famous fluency and (superfically at least) convicing outputs. The ‘GPT’ part stands for ‘Generative Pre-Trained Transformer’ which certainly needs unpicking to understand but 100 million users in two months is testament to the appeal of this iteration of this particular tool with this particular interface that has led to so many using the ‘ChatGPT’ eponymically. But the ongoing interests in Open AI from Bond villan Elon Musk; the environmental and ethical costs and implications along with the ‘Oh my god ChatGPT could pass an MBA’ educational panic in some quarters mean we could rue generalising the term. Subscriptions to ChatGPT will necessarily change our realtionship with it and, if even half of what reading about Microsoft integrations are to be believed (as well as Google’s “Look at us, we have ‘Bard'”counter-panic) the technology will be no more separate from the tools we use every day than a spellchecker.

My suggestion

All I’m saying really is that we should think carefully about the terms we use now lest they become fossilised, awkward and, worse, provide advertising and subtle condonement to what is just one product amongst many (and with not inconsiderable ethical baggage). So, for my tuppence worth: I think it’s OK for educators and students to use AI as a catch all term for these new dialogic, chat interface tools as well as other generative tools such as Dall-e and Midjourney ‘image from text’ generators and other comparable AIs such as AI music generation and AI video generators. The common denominator is ‘generation’ so I wonder whether we might usefully agree to use ‘AI text generators’, ‘AI image generators’ etc.. . as the default? I have been using ‘language models’, ‘large language models’ and even LLMs and realise that experts would likely prefer this framing but to a lay ear these say nothing about what these tools do and, anyway, when ‘The One Show’ (Popular BBC early evening magazine programme (starts at 22:56) is using ‘ChatGPT’ generically, a technical preference has probably got no chance.