Anyone have any recommended tools to use to detect AI generated student essays

[gallopingissuchfun]

Sick of reading assignments that are clearly styled by AI and at odds with students’ previous critical abilities. University’s TurnitIn software doesn’t detect AI text. What tools are out there to prove it was AI generated?

There are none that work well. My writing kept being flagged as AI by Originality AI, which I believe is the gold standard, so I got in touch with them, and they said that a 100 percent AI score only meant that AI had been used somewhere along the line, and could be something as simple as using Grammarly to correct text.

It's also ridiculously easy to get a low AI score from a 100 percent generated text by simply rewriting a few paragraphs here and there.

There is no tool that can prove it is written by AI. If you try and use a system that isn't in your institution's agreed approach/policy then you will be the one who is potentially doing something wrong - and you also have no hope of getting the case successfully through your academic offence procedure. If you can tell it's AI that's presumably because it's generic/ unoriginal/ doesn't reflect the course content. Mark it down accordingly as a bad piece of work. If it actually does merit a decent mark despite being clearly AI then consider why you're setting assessment in uncontrolled conditions that AI can do well and whether this is a productive strategy.

Absolutely what @MoreDangerousThanAWomanScorned said!

I read recently about someone (in the humanities) who set an assignment for students to get AI to write an essay in response to a set question and then their task was to research and discuss why and how it was wrong. These are really the skills we need to be developing in students, rather than trying to stop them using it (which they all are)

Don’t think they’re accurate. A colleague of mine found an old assignment of theirs on a memory stick from over a decade ago. Uploaded it and it came back at 63% AI.

But they can only critique an essay IF they know what good writing looks like. And what critical thinking looks like. If they don’t (and they can only do that by practising writing and thinking for themselves) then their critique of AI will be limited.

I can spot what AI has misunderstood or phrased badly because I have spent 20 years studying something. But a first year undergraduate has neither the critical skills nor the depth of knowledge to spot this.

Most of the cases I deal with re AI are ones with false or inappropriate references. They're easy to catch. Generic writing that doesn't address the question is also a sign but that's harder to prove.

We're not allowed to use detection tools as they are extremely inaccurate.

I am getting frequent ads on Facebook for AI services that write the literature section and maybe a lot more in university essays (and articles for submission etc). I hit first time I saw one of these ads, so naturally Facebook flooded me with more and more such ads.

I feel strongly that going back to model where essays have to be written in class, as a lesson, live by the students, is best solution. On computers that only have MS word working, and maybe google Scholar.

If we had world enough and time …

we’d viva each and every one of them with forensic focus on the bits where they’ve cheated.

I suspect we may have to bring back invigilated exams, written by hand.

Do you also get them to write essays by hand? Ideally using a quill? Can't be having all this new fangled technology being used, it's just not cricket!

the medical students now do nationalised exams on computer but the computers are locked down so they can't just google the answers. this works fine. tech can find solutions that make them use their brains after all

We have either locked computers or handwritten exams routinely - but the big problem is coursework and dissertations. At present, AI doesn't seem able to get the top marks but it can certainly get a solid pass. I imagine AI plus an essay mill type human input could probably do pretty well though. @MoreDangerousThanAWomanScorned makes the central point that current assessments are just no longer fit for purpose - and that purpose is also shifting. Students need to be working both with and without AI. I just did a really impressive course at my uni on using AI in biology - yet it's not yet part of our curriculum to pass this on to students. Madness.

I completely agree with needing to think about curricula that include rather than run away from AI. On assessment specifically, though, I actually think it's more that AI has highlighted/made more acute an existing problem. If your coursework assignment could be done by someone at an essay mill, or indeed if you can google the title you've given them and find example answers instantly, then you've always had this problem, just not at this scale and not in a way that is so 'in your face' as a marker.

I think the problem is actually more practical: almost all the alternatives to setting the kind of assignment that AI can do are more work-intensive for the setter, assessment system and/or marker, and it's not a great time in UK HE to have a big new resource demand. Even the path of least resistance of just going back to handwritten exams for everything isn't actually achievable for many institutions - if you can't balance the books anyway, then suddenly needing many times more space and invigilators for exams isn't an easy ask.

@MoreDangerousThanAWomanScorned

I agree with you to an extent and do try to design my CA such that it can't be produced by AI. I also get them to use AI appropriately in CA.

However, the problem is that there are skills they need that they are just not learning. For example, writing skills, critical reading skills, critical thinking skills. They are outsourcing these tasks to AI. How can we get them to learn these skills that would have traditionally been learned via CA?

I imagine all marks need to start being earned doing a written exam in a classroom from now on.

I think the only answer anyone has so far is that you do a lot more assessment of process as opposed to assessing just the product - that there are plans, drafts, key milestones like annotated bibliographies along the way and (because AI can make all those too!) that there is some 'live' interaction between educator and student as part of the assessment of those elements. This would almost certainly also be a much more effective way of teaching those skills. But we come back to the resource issue - that what I'm describing is essentially large elements of one-to-one teaching and assessment and you, of course, can't possibly deliver that and can't be expected to within the current set-up of HE.

Plus, on top of the basic issues of resources: there is also an issue of whether students would actually engage with all elements in this much more intensive assessment regime, how we ensure we're not creating an assessment system that massively advantages certain types of student (which is one of the reasons that so many institutions abandoned exam-only/heavy assessment in the first place) and, as part of that, whether we're ok with abandoning anonymous marking, and what issues that might throw up... There really aren't any easy answers on this, sadly. But I think pinning your hopes on something that will 'catch' the AI using students as OP has (and as many of my colleagues want to do) is just as unrealistic.

Completely agree. There is absolutely no way I could do that with current resources. My general approach is to use assessment for learning rather than assessment of learning but that really only works because I have relatively small classes (max of 70). I do practical sessions with a flipped classroom and then their submission requires a lot of reflection on what and how they learned and needs to include specific examples and evidence.

I think what will happen will be that students show their working, so they’ll have to submit their drafts, or as google docs work you can see the edit history.

I think then that you probably are doing as much and as well to tackle the problem as anyone can do right now (and a lot more than many)! Ultimately, I do think we're at a point where we need to start properly, actually having the conversation about what HE is for and what it takes to deliver that on a national (and beyond) level - but I don't see that happening anytime soon.

One of my essays was assessed as 100% AI written by "AI detection software" but I had written every word of it myself. I didn't even use a spell check!
I wouldn't rely on more AI to try and resolve this issue.

There’s also the risk of discrimination with so-called AI detectors. My understanding is that they commonly flag ND students and those whose English is not their primary language.

I suspect that if I put my writing into an AI detector it would flag as AI generated. I often use formal language and em dashes (I even know the difference between an em dash and an en dash!).

It’s a tricky one. I’m doing a humanities degree with the OU and getting to grips either writing academic essays. It’s hard! There are loads of skills I’m learning, and hopefully I’ll improve every time. But as I’m new to it, it currently it takes me ages to write an assignment I’m happy with. I did 6 revisions of a recent philosophy essay. I couldn’t sit down and produce the same essay in exam conditions. I suppose this is a skill to be learnt too.

I have quite a formal writing style as I spent many years as a lawyer and I do wonder if that is why mine got flagged.

That's brilliant and exactly what's needed.

The real skill now is to be able to differentiate between the plausible sounding nonsense that AI produces that often says a lot but includes nothing useful and content written by someone who actually understands the subject to a good level.

Get AI to write about something you know a lot about and you'll see what I mean.

Yes! Law is my discipline and it seems align with AI style writing.