#assessment

ChatGPT: What do we know now? What must we learn next?

I was honoured to join a TEQSA/CRADLE panel last week, the third in a series on the implications of ChatGPT (or GenAI more broadly) for higher education. In the second panel in March, I flagged the absence (at that early stage) of any evidence about whether students have the capacity to engage critically with ChatGPT. So many people were proposing to do interesting, creative things with students — but we didn’t know how it would turn out.

But three months on, we now have:

* myriad demos of GPT’s capabilities given the right prompts

* a few systematic evaluations of that capability

* myriad proposals for how this can enable engaging student learning and a small but growing stream of educators’ stories from the field with peer reviewed research about to hit the streets.

I also urge us to harness the diverse brilliance of our student community in navigating this system shock, sharing what we’re learning from our Student Partnership in AI. 

The following is an edited transcript of my presentation to the panel.

Educators are now beginning to be able to articulate what ChatGPT literacy looks like,  they have a sense of what the range of ability is within their cohort, and they’re beginning to gain insights into how to better scaffold their students. So for example, I’ve been talking to some of my most exciting colleagues here at UTS, asking them to tell me, how are you using ChatGPT? And in particular, what’s the capacity of your students to engage critically with its output? That is something we hear a lot about all the time. Three months ago, we really couldn’t articulate what that looked like. Now we can. So let me give you four glimpses of what some of my colleagues were saying to me. 


Antonette Shibani  teaches applied NLP to data science master’s students – they have to write a critical summary and a visual map of ethical issues using NLP. They’re encouraged to use ChatGPT for various purposes and to reflect on their use of it, and how useful it was. So the most able students, she tells me, could engage in deep conversations with AI, and they were using excellent prompts and follow up replies to the agent, whereas the less able students were using simple prompts to access content they didn’t have deeper discussions with the AI.


Here’s Baki Kocaballi, teaching interaction design, and the students are using GPT to develop user personas and scenarios, ideating solutions and reflecting critically. The most able students were doing this, they were generating rich scenarios with ChatGPT. Yet he was not seeing any critical reflection on what made an AI output an appropriate or accurate response. And Baki is reflecting that this may be something to do with the subjective nature of design practice. The less able students could still get some good responses but not much good reflection going on. And he notes that he needs to provide more scaffolding and examples for the students. So we see this this professional learning as well amongst the teachers. 


Here’s Anna Lidfors Lindqvist, training student teams to work together to build robots, and again encouraging their use of ChatGPT and reflecting critically on it. The most able students could use it in quite fluent and effective ways. But the less able students  she notes, they’re  not really validating and checking GPT’s calculations. They’re struggling to apply information in the context of the project. Some actually just dropped GPT altogether. It’s just too much work to actually get it to do anything useful.


And a final example from Behzad Fatahi, teaching Yr 2-3 students, they’re using ChatGPT but they’re also using a simulation tool called Plexus to analyze soil structure interaction problems. The most engaged students were behaving as shown, but the least engaged students were struggling and were behaving like this. So, the point is not so much the details — the point is that our academics are starting to know, what does good look like? What can I expect from my students? There is clearly a diversity of ability to engage critically with a conversational, generative AI.

And when you step back from these particular examples and asked again, what are going to be the foundational concepts and evidence as it grows, around what we could call generative AI literacy, for learning, not for marketing, not for any other purposes that can be useful — for learning.

Conversational agents are not new in the world of education. They’ve been around in the AI and research literature for donkey’s years, but used well, they should move us towards more dialogical learning and feedback. So we’re all used to thinking about student peer feedback, learning designs, they’re now going to be interacting with agents. Those agents will be interacting with them and potentially with other agents as well, playing different roles and we will learn how to orchestrate these agents and define the roles they need to play. 

And every turn in this conversation is a form of feedback. The question is what move does the student make next? How do they engage with that feedback from humans and machines?

Now, we have concepts and evidence from pre-generative AI research around this. We have concepts such as student feedback literacy, and we have been taking inspiration from that and talking about student automated feedback literacy now. There is the notion of teacher feedback literacy as well, and similarly, we’re working on teacher automated feedback literacy. So these are powerful concepts I think, for helping us think about how we can study and equip students to engage in powerful learning conversations.

The final point I want to make is we need to work with our students.

We’ve been working with our Students’ Association here at UTS. We had over 150 applicants for a workshop where we took a stratified sample of 20. They engaged in pre-workshop readings where we presented them with a range of dilemmas involving ChatGPT and Turnitin, took part in online discussions and had a face to face workshop. They got briefings from UTS experts introducing generative AI, explaining how it’s being used creatively at UTS, such as the examples I just showed you,alking about the ethical issues around generative AI and talking about Turnitin what do we know about it? And should we turn it on? That is a decision we’re trying to make at UTS at the moment.reakout groups, a plenary discussion and we have a report currently under review by the students as to whether they have they’re happy with that as a summary of what they talked about.

But let me just share three examples of what they told us and you’ll see some echoes here with what we heard from Rowena Harper earlier. 

  • Firstly, they are asking, please equip us to use ChatGPT for learning. We are aware that it could actually undermine our learning if we don’t use it well, but what does that mean? You’re the educators — you should be able to tell us how to use it effectively for learning and not in a way that torpedoes our learning. 
  • Secondly, can we have more assessments, integrating ChatGPT in sensible ways. They were very excited to see the examples such as the ones I showed you because not all of them have experienced that yet. And finally, Turnitin. Well, yes, it may have a role to play as part of an overall approach to academic integrity. But please handle with care. If there are any questions about our academic integrity, we want to be invited for a respectful conversation, and not be accused of misconduct when, as we are already hearing, Turnitin is backing off from some of its original claims about how good its software is. It’s a very fast-moving arms race. 

So just to wrap up,  here are three questions about what we need to learn next. 

  • What do we mean by generative AI literacy and how do we scaffold it? 
  • How well do generative AI learning designs translate across contexts? They may look very promising, but we have to actually deploy those and study them in context. 
  • And finally, how are we going to engage our students in codesigning this radical shift with us? We talk a lot about diversity of voices and the design of AI. We absolutely need them on board, trusting the way we’re using this technology, seeing that we’re using it responsibly and ethically, and bringing the perspectives that they have. They’re the ones on the receiving end of all this policy we’re talking about.

Simon Buckingham-Shum is the director of the Connected Intelligence Centre at the University of Technology Sydney. He has a career-long fascination with the potential of software to make thinking visible. His work sits at the intersection of the multidisciplinary fields of Human-Computer Interaction, Educational Technology, Hypertext, Computer-Supported Collaboration and Educational Data Science (also known as Learning Analytics).

Six factors that impact student mental health

Young people attending university experience higher rates of stress, depression and anxiety than their peers who are primarily engaged in full or part-time work. What is going on here? And how can universities better support their students’ mental health?

I’ve been investigating the mental wellbeing of undergraduate and postgraduate students at The University of Melbourne since 2011.  Results from our most recent (pre-Covid-19) survey of the mental wellbeing of coursework students in eleven Bachelor and coursework Masters programs have just been published. Our study confirms many university students are struggling with mental health difficulties: more than 4,500 students responded to our anonymous, online survey and, of these, almost one-third reported ‘severe’ or ‘extremely severe’ levels of depressive symptoms, anxiety or (chronic) stress (as measured by the DASS-21). These symptom levels would impede students’ daily functioning (sleep, memory, cognitive processing), social interactions and learning. So there is definitely a problem.

What is going on here?

It seems that university study and university experiences are highly variable. For many, and even most, young people, university study affords positive experiences of personal and intellectual growth, exploration and achievement; university is a time and place for thriving. For some, however – around one in three in our study – their experiences are negative, and this appears to contribute to a spiral of stress, dis-engagement and mental health difficulties.

What can universities do about it?

Most universities are taking active steps to improve mental health services and information for their students and many now offer stress-management tools and strategies such as mindfulness. But such services are responsive, rather than preventive. My colleagues and I at the Melbourne Centre for the Study of Higher Education want to understand what university administrators and educators can do to ensure that more students thrive, not just survive, during their time at university. 

Our latest study identified six common course experiences that predict substantial variations in students’ levels of mental wellbeing. This is really important because it means that universities, and university teachers (academics) in particular, can do more to make a difference to student mental wellbeing than they, and others, might imagine.

But before explaining those course experiences, I need to first explain that we didn’t only consider course experiences. We all have lots of ideas about why university students might be experiencing high levels of stress and mental strain these days. For example, it’s widely discussed that the increasing costs involved in attending uni are putting many students under financial pressure. Most Australian uni students graduate with considerable Fee Help debts that may take decades to pay off and, in many disciplines, there is no guarantee – or even a reasonable prospect – that all graduates will find a job in their chosen field at the end of a degree. 

As a result, there’s pressure on uni students to do more than pass; they may need to graduate in the top X%, or to build a grade-point-average that is, ironically, ‘above average’ to secure a job. Such expectations might be particularly onerous for local students whose educational backgrounds haven’t prepared them for the styles of teaching and learning at university, those who speak English as an additional language, international/overseas students, those who are the first in their immediate family to attend a university, and so on.

We took stressors like these into account in our study. What we wanted to find out is: Do students’ course experiences – that universities can influence through curriculum design and teaching practices – predict mental wellbeing after students’ individual circumstances and backgrounds are taken into account? The short answer is, yes, they do. In fact, our study found that the six coursework experiences are stronger predictors of students’ mental health than demographic factors (such as age, gender, living situation, hours of paid work and family care per week etc) and individual stressors including financial stress, worry about job prospects or English language difficulties.

For example, in our statistical model of DASS-Depression, demographic factors accounted for 6% of variance in scores, individual stressors accounted for an additional 7%, while course experiences accounted for 16% (after taking all other factors into account). 

What this means is that ensuring that all students have positive course experiences can make a substantial difference to their mental wellbeing, notwithstanding common pressures on uni students, such as juggling family care and paid work commitments; financial strain; career uncertainty; or English language difficulties. 

Which course experiences predict students’ mental wellbeing? 

Course experiencesExample items
Sense of belonging (in the course, or to the university)‘I sometimes feel that I don’t belong here’
Assessment pressures‘When I work on assessment tasks I think about how poorly I am doing compared with other students’
Positive engagement with other students in the year/program/school‘I have made at least one or two close friendships with other students in my course’
Finding course content interesting or useful‘I’m really interested in what I’m studying’
Teachers perceived as supportive of students‘My teachers convey confidence in my ability to do well in the course’
Academic workload ‘The course workload is too heavy’

Feelings of not belonging and worry about assessments were most strongly predictive of both DASS-depressive and DASS-anxiety symptoms; while perceiving the course workload as unreasonable is independently associated with DASS-Stress. ‘Autonomous’ motivation for study – being interested or perceiving value in what you are studying – is strongly predictive of positive Psychological Wellbeing and Satisfaction with Life. Engagement with other students in the course is independently associated with all – positive and negative – mental health outcomes; while perceptions of Teacher Support are most strongly associated with DASS-Depression and Satisfaction With Life.

So. What can universities do to better support students’ mental wellbeing? Our findings suggest a twin focus – on (course)work and relationships. Rationalising and organising students’ academic workload will be helpful, as will ensuring that assessment tasks are well-spaced and designed to assess and promote learning, not only to normatively rank student performance (see, eg, Slavin, Schindler & Chibnall, 2014; Tang & Ferguson, 2014). Promoting respectful and accepting relationships among students and between staff and students is also vital if students are to form positive, rather than competitive, peer relationships (see Larcombe et al., 2013) and feel that they ‘belong’ within the program or university (van Gijn-Grosvenor & Huisman, 2020). The value of academic teachers showing interest in their students is difficult to over-state (Baik, Larcombe & Brooker, 2019). Our findings confirm previous studies that have found students thrive psychologically when teachers acknowledge their perspectives, offer meaningful choices and justify course requirements (see Su & Reeve, 2011). Student motivation for study is also known to be maintained when teachers support students to pursue their intrinsic interests and goals (Ryan & Deci, 2000), and when feedback on assessment is consistent and informative (Hoskins & Newstead, 2009). Recent research also suggests that students’ sense of belonging is strengthened when teachers value and respect all students’ contributions in class, and create welcoming and empathic learning environments (Wilson, Murray & Clarke, 2018; see also http://unistudentwellbeing.edu.au/).

Of course, there are already many pressures on curriculum designers and academic teachers. If supporting student mental wellbeing is to be added to, or elevated among, the list of priorities for academic teachers, they will need to be supported and resourced to explore evidence-based strategies and approaches. And those strategies and approaches in turn need appropriate resourcing. 

We hope that our research prompts university administrators and funders, not only university teachers, to think about how students’ course experiences might impact their mental health – both positively and negatively – and about the steps that can be taken to address students’ high levels of psychological distress. We have long known how much teachers matter for student learning; our study provides new insight into how much teachers and teaching practices matter to student wellbeing.

Dr Wendy Larcombe is a Principal Fellow at the Melbourne Centre for the Study of Higher Education and Melbourne Law School.