Designing Assessment for an AI enabled world

Designing Assessment for an AI enabled world

Assessment in AI enabled world - title slideAs part of AI and Education experts’ group at UCL, the Assessment design workstream, led by Mary Richardson, recognized the need to provide timely and practical staff guidance on how to navigate both the opportunities and challenges presented by the increasing availability of GenAI.    Assessment was (and is) a particularly fraught area for many staff,  with concerns around academic integrity rising, the capabilities and range of GenAI tools advancing at pace and a short time frame in which to make changes to their assessment design.

A phased approach was required, the first involving strategies for making small scale adaptations to current assessment practice which would be viable within the existing UCL regulatory framework.  The second phase involved planning ahead for larger changes.

My contribution was to produce a set of resources for both phases that could be used this academic year by staff, depending on where they are in their planning;  some may need low key interventions, some may be ready for more radical changes and some may be looking at a mix of the two.  These resources, called ‘Designing assessment in an AI enabled world’  are now part of the newly released Generative AI hub for UCL staff.

To support larger changes in assessment I put together an assessment menu with suggestions for either integrating AI or for which AI would find it difficult to generate a response. The inspiration for this was Lydia Arnold’s wonderful resource Top Trumps consisting of 50 suggestions for making assessment more authentic and published under a creative commons licence 4.0.    Many of the examples included in the menu are customised versions of Lydia Arnold’s ideas as well as ideas from the sector ( list of contributors).

My colleague Lene-Marie Kjems from Digital Advisory helped me get the menu into shape, advising on visual design and accessibility.  We were about publish the menu on the UCL website when Jisc (via our manager, Marieke Guy) broached the idea of collaborating on an interactive, searchable version for their website.  They had been thinking along the same lines of AI-ing Top Trumps and, rather than reinvent the wheel, it was obvious we should join forces.

The collaboration was extremely valuable. It really helped to have a group of experts reviewing and revising the resource and, in addition, we invited Lydia Arnold to be a consultant on the project.

The Jisc design team did a fabulous job of turning the original PPT version into the interactive resource you can now find on the Publications and resources page  (scroll down to locate it). Using clickable icons representing each of the above (on slide 7), users can be presented with options to consider.  

There is an accompanying Blog post on the Jisc National Centre for AI website which describes the development of the resource in more detail, including the rationale for the search categories.

Our UCL version of the assessment menu is the same content but only available in PDF download.

As with all of the AI guidance, the assessment design work will have to continue as our experience and expertise in AI evolves over the coming academic year.