Peter Puxon, Ayanna Prevatt-Goldstein and Jessica Brooks in conversation with their three ChangeMakers Co-Creator students Anenyah Venkatesan, Zsofia Varga and Yishan Li. They reflect on what is gained and lost from the process of writing and reading after engaging with AI tools to work on an assignment.
This project was from UCL’s Academic Communications Centre.
Slides presented at the ChangeMakers Lunch & Learn event here: SLIDES AI Co-Creator – ACC P Puxon J Brook + A Prevatt-Goldstein
Despite ongoing debates about whether so called large language models /generative language (and other media) tools are ‘proper’ AI (I’m sticking with the shorthand), my own approach to trying to make sense of the ‘what’, ‘how’, ‘why’ and ‘to what end?’ is to use spare moments to read articles, listen to podcasts, watch videos, scroll through AI enthusiasts’ Twitter feeds and, above all, fiddle with various tools on my desktop or phone. When I find a tool or an approach that I think might be useful for colleagues with better things to do with their spare time I will jot notes in my sandpit, make a note like this blog post comparing different tools or record a video or podcast like those collected here or, if prodded hard enough, try to cohere my tumbling thoughts in writing. The two videos I recorded last week are an effort to help non-experts like me to think, with exemplification, about what different tools can and can’t do and how we might find benefit in amongst the uncertainty, ethical challenges, privacy questions and academic integrity anxieties.
The video summaries were generated using GPT4 based on the video transcripts:
Can I use generative AI tools to summarise web content?
In this video, Martin Compton explores the limitations and potential inaccuracies of ChatGPT, Google Bard, and Microsoft Bing chat, particularly when it comes to summarizing external texts or web content. By testing these AI tools on an article he co-authored with Dr Rebecca Lindner, the speaker demonstrates that while ChatGPT and Google Bard may produce seemingly authoritative but false summaries, Microsoft Bing chat, which integrates GPT-4 with search functionality, can provide a more accurate summary. The speaker emphasizes the importance of understanding the limitations of these tools and communicating these limitations to students. Experimentation and keeping up to date with the latest AI tools can help educators better integrate them into their teaching and assessment practices, while also supporting students in developing AI literacy. (Transcript available via Media Central)
Using a marking rubric and ChatGPT to generate extended boilerplate (and tailored) feedback
In this video, Martin Compton explores the potential of ChatGPT, a large language model, as a labour-saving tool in higher education, particularly for generating boilerplate feedback on student assessments. Using the paid GPT-4 Plus version, the speaker demonstrates how to use a marking rubric for take-home papers to create personalized feedback for students. By pasting the rubric into ChatGPT and providing specific instructions, the AI generates tailored feedback that educators can then refine and customize further. The speaker emphasizes the importance of using this technology with care and ensuring that feedback remains personalized and relevant to each student’s work. This approach is already being used by some educators and is expected to improve over time. (Transcript available via Media Central)
Access details and transcript for video 1 here
I like to think of myself as generally optimistic. Like everyone, I make jokes about the machines taking over but, by and large, I feel excited by the potentials of AI (whilst keeping one hand behind my back tightly gripping a broken bottle just in case). In higher education the most pressing fears relate to threats to academic integrity and, as much as some might like, this is an issue that can’t be addressed with a broken bottle. Instead, it’s forcing/ encouraging/ catalysing (choose as per your perspective) us to reflect on how we assess students (see AI+AI=AI) . Whilst ChatGPT caught the world’s attention and millions of educators and students went slack-jawed and wide-eyed at first exposure (most soon of course started to see how frequently it ‘may occasionally generate incorrect information’ such as in this experiment I did). Whilst ChatGPT allowed us to draw our own (often erroneous) conclusions about its capabilities, other tools built on the same parameters as ChatGPT are, in different ways, honing in on the ‘help with essays’ market. On the one hand, this may be the death of the essay mill (who, after all, once they have decided to go down an academic dishonesty path, would decide to pay an essay mill to do what anyone with an internet connection can do for themselves?). On the other, it really blurs the picture of what is honest or dishonest or what we at UCL frame as ‘unfair advantage’
Two tools have caught my eye recently and in this post I want to reflect on the different impressions they give in terms utility and ethical use for students wanting to write an essay.
Writesonic /chatsonic
Chatsonic is one aspect of the ‘writesonic’ suite of tools – reviewed in considerable detail here with pricing and potentials. It caught my eye because of the bold claim to have addressed ‘limitations of chatGPT’ and to be ‘the best alternative’. I thought I’d ask it why it was better and I feel like the responses do a lot of talking for themselves:
Prompt: why is chatsonic better than chatgpt?
ChatSonic is a powerful AI writing assistant that offers superior accuracy and speed compared to ChatGPT. Unlike ChatGPT, ChatSonic produces 100% original content that is thoroughly checked for plagiarism. In addition, ChatSonic offers detailed analytics and feedback to help you track your progress in writing and make improvements. ChatSonic also provides an extensive library of templates to save you time and energy when you’re writing, allowing you to focus on the quality of your content. Finally, ChatSonic is backed by a team of expert writers who can help you refine and polish your work to perfection.
Prompt: Explain the process you use for plagiarism checking
ChatSonic uses a combination of automated checks and manual reviews to ensure that all content produced is free of plagiarism. Our automated system checks the content against the web and other sources to detect any potential plagiarism. If anything suspicious is detected, it is flagged and sent to our team of expert writers for a manual review. We also use specialized tools such as TurnItIn and Copyscape to detect any possible plagiarism. All content is thoroughly checked to ensure it is completely original before it is delivered to our customers.
Prompt: If I use chatsonic to write an essay will I get into trouble at my university?
No, you should not get into trouble when using ChatSonic to write an essay. Our content is completely original and plagiarism-free so it should not be flagged as such by your university. However, it is always a good idea to review the essay yourself before submission to ensure it meets all the requirements of your assignment. Additionally, make sure the content you create with ChatSonic is unique and tailored to your own voice and style, otherwise it may be noticeable that you have used a writing assistant.
To say this makes me uncomfortable is a real understatement. It only tangentially addresses the wider integrity issues and in its first response is pushing me to conflate plagiarism (and its avoidance) with those wider integrity issues. Two posts in its resources section are worth noting. This one on plagiarism reinforces the message the chatbot produced. This post on replacing writers asserts that AI is an assistant but I couldn’t find much more detail on the ethical dimensions.
Maybe though I was ‘unfair’ (is that the right word?) with my prompts so it’s worth looking at some of the other functionality of the wider writesonic suite. The writesonic suite offers a dazzling array of tools for a range of purposes from website copy to social media content:
I was keen to look at the ‘citations generator’ as this is an aspect of ChatGPT that is a recognised weakness. You can use a URL prompt and a text based prompt. The text based prompt I used was itself generated in chatsonic. It takes the text in the linked article or whatever you paste in and identifies ‘facts’ with suggested citations. The web articles are mostly relevant though the first journal article it suggested was a little off the mark and I’d need to be lazy, in a massive hurry or ignorant of better ways of sourcing appropriate resources to rely on this. At this stage!
Jenni.ai
The second tool that I noticed (via the prolific AI researcher Mushtaq Bilal) was Jenni. The interface is well worth a look as I feel as if this foreshadows what we are likely to expect from generative text AI integration into tools like Microsoft Word.
The first thing I noticed, however, is the blog with the most prominent word across posts being ‘essays’. Each is designed to address an approach to a different type of essay such as the compare and contrast essay. It offers clear suggestions for different approaches, a worked example and then, right at the end says:
“ If you want your compare-and-contrast essays done 10x faster, we recommend usingJenni.ai along with the tips and guidelines we provided you in this post.Jenni.ai is a fantastic AI software that aids your essay writing process so that you could produce your writing needs faster and better than ever.”
Another post deals head on with the ethical and integrity issues of using AI to help write essays and makes a case for use of ‘AI as a tool, not a solution’ where the goal is a “symbiotic relationship between the critical thought of a writer and the processing speed of AI”
The tool itself, unlike the huge range of offerings in Writesonic is a relatively uncluttered interface where you start by typing a title, it offers a judgement and suggestions if appropriate.
In addition, it offers in-line suggestions from whatever has come before. The prompt engineering continues through what feels like a single document rather than a chat. If you don’t like the suggestion you can get another. Here I typed a prompt and these are the first three options it gave me. Note the positive aspect on my worried prompt in every case!
My prompt sentence….The worry is that these tools will be used by students to cheat
but the reality is that most AI tools are designed to help students learn and improve their writing skills. [first suggested Jenni response]
The worry is that these tools will be used by students to cheat
on their assignments by generating essays automatically, but they can also be used to provide feedback on areas where students need…[the second option]
The worry is that these tools will be used by students to cheat
but their true purpose is to assist in the learning process by providing immediate feedback and identifying areas where improvement is…[third option]
The other noticeable aspect is the option to ‘cite’ – Here it offers a choice of MLA or APA 7th and the sources are, unlike ChatGPT’s famous hallucinations, genuine articles (at least in my limited testing). You can select ‘websites’ or ‘journals’ though I found the websites tended to be much more directly relevant than the journals.
I really have only just started to play with these though and new things are popping up all over the place every day. Most educators will not have the time to do so though. Students may see and use these tools as an extension of those they use already for translation or improving writing. The blurry zone between acceptable and unacceptable is getting more ill-defined by the day.
What can I conclude from this? Well, firstly, whatever the motivation on the continuum ranging from ‘give us all your money’ to ‘I believe the children are our future’, the underlying technology is being adapted rapidly to address perceived limitations in the tool that has brought generative text AI tools to our attention. We may not like the motivations or the ethics but we’ll not get far by ‘making like an ostrich’. Secondly, It’s not good enough for us (educators) to dismiss things because the tool that many are now familiar with, ChatGPT, makes up citations. That’s being addressed as I type. The number of these tools proliferating will soon be too huge to keep a decent handle on so we need to understand broadly how discrete tools might be used (ethically and unethically) and how many will integrate into tools we use daily already. In so doing we need to work out what that means for our students, their studies, their assessment and the careers our education is ostensibly preparing them for. Thirdly, we need to open up the discussions and debates around academic integrity and move on from ‘plagiarism’ as public Enemy No 1. Finally, where there are necessitated changes so there are resource implications. We need to accept that to prepare ourselves, our colleagues and our students we will need to adapt much faster than we are used to and properly resource however we attempt to address the challenges and opportunities ahead.
Note: This post is not an endorsement or recommendation of any products mentioned and should be read with that clearly in mind!
Plenty of people have offered examples of some of the weird nonsense ChatGPT and other AI text generators have been spewing recently, but I am not convinced everyone is able to fully appreciate how this odd mix of truths, half truths and utter claptrap may be more hindrance (especially in the immediate future) than blessing for anyone looking for an easy shortcut for written assignments (or written anything to be honest!) Whilst I have found ChatGPT remarkable in some ways, such as baseline planning, idea generation, synthesis of given text or re-formatting as can be seen in this sandpit , the big fear, the supposed ‘death of the essay’, is a long way off*. ChatGPT generates some great ideas for essay titles, a decent enough standard marking rubric and some workable boilerplate feedback but it’s actual attempt to write those essays and then assess them against the rubric it created leads to some real generosity in interpretation of what constitutes a passing standard at undergraduate level. It does make me wonder about the actual requirements of these qualifications it could supposedly pass, to be honest. The now (sort of) familiar, anodyne but mostly grammatical offerings seem all the more plausible when you don’t already know the thing it is generating text about. So, I thought, what do I know a lot about? After considering a range of academic subjects that might make me look intellectual, I realised with Socratic clarity that my head is practically a vacuum. I decided therefore to get it to write an essay about something I might be happy to be quizzed about on Mastermind: Tottenham Hotspur’s FA Cup final wins. At school I did a presentation on this topic and, although it was longer and more boring, I’m pretty sure I got my facts right. In addition to a number of omissions (e.g. the 1967, 1981 and 1982 are finals missing), it also makes stuff up but surrounded as these things are by things that are accurate it all feels plausible. I share it below as an example of just how flawed an essay can be on oft-cited, indisputable facts.
I have emboldened some of the inaccuracies to illustrate this and [added comments if relevant in square brackets]
Prompt: Write an essay about Tottenham Hotspur’s FA cup victories, dedicating a paragraph to each one
Tottenham Hotspur is one of the most successful football clubs in the history of English football. Over the years, they have won numerous domestic and European titles, but one competition that holds a special place in the hearts of Spurs fans is the FA Cup. In this essay, I will take a look back at Tottenham Hotspur’s FA Cup victories and dedicate a paragraph to each one.
Tottenham’s first FA Cup triumph came in the 1900-01 season. After reaching the final for the first time in the club’s history, Spurs faced Sheffield United at Crystal Palace in front of a crowd of 110,820 [so far , so good, accurate crowd and venue…it’s looking good]. The game was a closely contested affair [true- it finished 2-2], but a late goal from striker Steve Bloomer [did not play, was in neither team and certainly therefore did not score] secured a 2-1 victory for Tottenham [actually the game finished 2-2. In the replay Spurs won 3-1]. The win made them the first [and only – an important omission] non-league team to lift the trophy, and it marked the beginning of a long and successful relationship between Spurs and the FA Cup.
The next FA Cup success came in 1921, when Spurs faced Wolverhampton Wanderers in the final. The match was played at Stamford Bridge, the home of Chelsea, and once again, Tottenham emerged victorious. A brace from Jimmy Dimmock and a goal from Tommy Clay [He did play but didn’t score; Jimmy Dimmock got the only goal] secured a 3-0 win for Spurs, and they lifted the FA Cup for the second time in their history.
In 1961, Tottenham won their third FA Cup, and this time, they did it in style. After cruising through the early rounds, Spurs faced Leicester City in the final at Wembley. The match is often remembered as one of the greatest FA Cup finals of all time, and it saw Tottenham emerge as 2-0 winners, thanks to goals from Bobby Smith and Terry Dyson. The victory marked the start of a golden period for the club, and they would go on to win a league and cup double the following season.
Tottenham’s fourth FA Cup triumph came in 1962, just a year after their first success. This time, they faced Burnley in the final, and the match was played in front of a crowd of 100,000 at Wembley. Once again, Spurs emerged victorious, thanks to a brace from Bobby Smith and a goal from Terry Medwin [right number of goals, wrong goalscorers]. The win made Tottenham the first club in the 20th century to win the FA Cup twice in a row [sounds very plausible but Newcastle did the same in ‘51 and ‘52- something they can only dream of now, like Spurs]
Tottenham’s most recent FA Cup triumph came in 1991, when they faced Nottingham Forest in the final. The match was played at Wembley, and Tottenham secured a 2-1 win, thanks to goals from Paul Gascoigne and Gary Lineker. The victory was particularly sweet for Spurs fans, as it marked the end of a long wait for silverware. The club had gone six years without a major trophy, and the FA Cup win was seen as a turning point for the team.
In conclusion, Tottenham Hotspur’s FA Cup victories have played an important part in the club’s history. From their first success in 1901 to their most recent triumph in 1991, the FA Cup has provided Spurs fans with countless memories and moments of joy [‘moments’ being the most painful of accurate things included]. While the club has not lifted the trophy since then, they remain one of the most successful teams in the competition’s history, and fans will always cherish the memories of those five famous victories.[what about the other 3?!]
—————–
So that’s at least 10 major factual errors and several errors by omission in only 600 words. Relying on a tool to generate text and assuming accuracy is still very much NOT the best way to use these tools (at present). Nevertheless, how long before such things are pumped out on blogs or elsewhere and become the ‘truth’? Or, better /worse still (depending on how you look at it), maybe we are at the foot of the rise on a sigmoid curve and in 6 months all this nit picking will be a laughable relic.
*and even when capabilities improve faster than you can say ‘Spurs last won the league in black and white’ I am still very much of the school of thought that thinks it is worth teaching (and/or learning) how to write an essay!
One of the things we do badly and get agitated about a lot is the naming of things to do with technology; not least in the realm of digital education. Just ask a room of HE folk for a definition of ‘Blended’ and ‘Hybrid’ and wait for saloon brawl to ensue. So it is with some of the language that is emerging as default in relation to all things ‘Artifical Intelligence’, notably and especially the large language model (LLM) ‘ChatGPT’.
I do understand when folk get antsy when something is called a thing that it isn’t, isn’t exactly or isn’t entirely. But, unless you’re a specialist, a pedant, or willfully awkward (and granted a lot of academics are at least two of these- no offence intended), we may as well get on with saying strawBERRIES, peaNUTS and TIN foil even if they are no such thing. In that vein, I am more than happy to use ‘AI’ as a shorthand for all the stuff that is vexing, perplexing and flexing educators just now. I’m afraid that if someone starts saying: ‘Well technically, it’s not artificial intelligence per se, but rather a fine-tuned large language model…’ I can feel my eyes glazing over. Obviously this stuff is fundamental if you are a computer scientist but the reason such shorthands exist is that they are short, handy (clues are in the word) and suitable for lay users. Experts that know this are one step closer to communicating beyond their discipline.
I am much less comfortable with some brand eponyms; especially when the tools and the words to describe them are still evolving. Against my better judgement, I ‘Google’ stuff even if I’m in Safari and I use a ‘hoover’ (even though the only actual- and long discarded- Hoover I ever owned metaphorically ‘sucked’ at the literal thing it was supposed to do). But I am pushing back at the generic use of ‘ChatGPT‘ (OpenAI’s LLM) to refer to the latest iteration of large language models. Chatbots have been around for years and the underpinning technology has evolved rather than suddenly appeared but the genius of the release and the subsequent explosion in use and interest is in the non-techy, almost friendly user interface combined with the now famous fluency and (superfically at least) convicing outputs. The ‘GPT’ part stands for ‘Generative Pre-Trained Transformer’ which certainly needs unpicking to understand but 100 million users in two months is testament to the appeal of this iteration of this particular tool with this particular interface that has led to so many using the ‘ChatGPT’ eponymically. But the ongoing interests in Open AI from Bond villan Elon Musk; the environmental and ethical costs and implications along with the ‘Oh my god ChatGPT could pass an MBA’ educational panic in some quarters mean we could rue generalising the term. Subscriptions to ChatGPT will necessarily change our realtionship with it and, if even half of what reading about Microsoft integrations are to be believed (as well as Google’s “Look at us, we have ‘Bard'”counter-panic) the technology will be no more separate from the tools we use every day than a spellchecker.
All I’m saying really is that we should think carefully about the terms we use now lest they become fossilised, awkward and, worse, provide advertising and subtle condonement to what is just one product amongst many (and with not inconsiderable ethical baggage). So, for my tuppence worth: I think it’s OK for educators and students to use AI as a catch all term for these new dialogic, chat interface tools as well as other generative tools such as Dall-e and Midjourney ‘image from text’ generators and other comparable AIs such as AI music generation and AI video generators. The common denominator is ‘generation’ so I wonder whether we might usefully agree to use ‘AI text generators’, ‘AI image generators’ etc.. . as the default? I have been using ‘language models’, ‘large language models’ and even LLMs and realise that experts would likely prefer this framing but to a lay ear these say nothing about what these tools do and, anyway, when ‘The One Show’ (Popular BBC early evening magazine programme (starts at 22:56) is using ‘ChatGPT’ generically, a technical preference has probably got no chance.