Anyone have any recommended tools to use to detect AI generated student essays

[gallopingissuchfun]

Sick of reading assignments that are clearly styled by AI and at odds with students’ previous critical abilities. University’s TurnitIn software doesn’t detect AI text. What tools are out there to prove it was AI generated?

I’m studying a PhD (English Literature), and have passed my confirmation so writing at PhD level, however it is of course a huge challenge to synthesise different texts, to write with clarity on complex ideas at sufficient depth whilst keeping an eye on the thesis argument. I don’t use AI apart from when Google serves up responses and the first section includes these, I drill down to the sources to ensure accuracy. I do wonder if I’m being a philistine though, in not using it to make my life easier if everyone else is taking advantage of it and I’m sat going through the pain of multiple drafts using my tired organic human brain.

I think you're doing it the right way. I mean, the whole point of a PhD (in the humanities) is to show you have the ability to "snthesise different texts [and] write with clarity on complex ideas at sufficient depth whilst keeping an eye on the thesis argument".

Thank you, I know that it is a huge issue in our English Literature department (more for UG). My BA was in the early 1990s, and all we had then was a typewriter and the short-term loan section of the university and scholarly discourse amongst your peers and academic staff.

No. I wonder if the lecturers use it to eg produce slides for their lectures?

AI will make us all stupid, so I do see the problem, but all you can do is go back to invigilated exams. Which don't prepare you for the workplace. Far better to teach students how to use it so they aren't made stupid by it.

I ignore the AI summaries on online searches, I always go down to the actual results

You don't have to assess students by getting them to write essays. We did 15 min vivas for a module this year and it worked brilliantly. The students had to prepare a 4 slide long presentation and then did not present them but were instead asked questions on the content which was used to assess their comprehension of the material. All marking was over and done with almost within the session.

This sounds good.

I’ve seen this work well at university. It brings in rigour and helps to develop other transferable skills.

There are loads of other ways to examine students which prevent ai use too, you just have to be a bit more creative and think about what is being assessed rather than just falling back on traditional formats.

OTOH, @Geneticsbunny part of what I'm teaching is expressing & developing a long form argument via the essay. The ability to research, think through, and construct a complex argument over 3,000 to 5,000 words is one of my ILOs.

Just to play devil's advocate @ParmaVioletTea , why is that one of your learning outcomes? I mean obviously, researching and then thinking through arguments is a vital skill to learn at university, but why does the output need to be in an essay form? Noone writes essays after uni so that skill in itself is not all that useful.
The essay itself is just a means to an end, which could be achieved by the student preparing something else.

I'm sure you would love to be treated nya doctor that used AI for assignments or your kids taught by teachers that used AI throughout their courses etc

Noone writes essays after uni so that skill in itself is not all that useful.

But in my field (humanities) people write reports. They go into journalism. They go into many fields in which writing, and research, and disseminating the results in long-form argument are central activities.

As a student Ds is always worried he will be accused of using AI. He has a writing style that is a little quirky and when I read over one of his assignments I said I would swap the sentence around (my degree is English Lit from a million years ago) and Ds argues that is his stylistic choice.

In secondary he was accused of copy pasting because of he wrote above his age vocabulary wise but the teachers realised he wrote that way in class. That was a case of getting to know him. He attends every lecture in person at uni and loves his course.

For uni he just hopes that his word document shows enough of his edits and his history shows all the websites he visits which he may not quote or refer to in the document. Plus if ever questioned he would be able to show his thinking process for the work.

Yes, so the skill which is important is combining source material, comprehending the content fully and then being able to weave the corresponding information together in a way which communicates the content to the appropriate audience. I agree that essays are one way to teach this skill set but there are many others.

I think its going to be increasingly difficult to detect AI written work, I think software tools would struggle to detect it.

To be honest I personally am not going to bother fighting what might be a loosing battle. Trying to investigate which scripts are AI vs legit is going to become a huge time suck. Currently the marking time allocated in my workload is far lower then it normally takes. So hunting for AI scripts personally would just make the work of marking even more miserable.

Yes this is a drop in academic standards, but that already happened at my failing seaside institution.

My course has been dumbed down significantly, our entry tariff has dropped. Perhaps AI plagourism could be easily fought by introducing more formal written exams into the course. We currently don't have exams and the degree is 100% coursework - this one of the big selling points of our degree (alongside the main selling point of we are in a party town -woop). In my institution management would resist traditional exams because the "customers" would complain/melt-down. We already are under pressure to simplify our already basic assessments due to our pretty weak cohort struggling with the basics.

In my school management would likely see AI written assessments as a net positive. Grades could go up, more students pass, retention improved. They don't seem to care about academic standards beyond maximising student numbers and giving them a "smooth - safe" path to an easily attained degree.

Personal I think HE is dumpster fire and AI assessments are not the hill I'd personally die on. Management isn't going to help by either improving curriculum/exams, they certainly won't allocate more time/resources to discover the misconduct. Then you have all the extra meetings around misconduct panels - a huge amount of time not reflected in the workload. And if the customer errr. I mean... "student"is found "guilty" they will just get a gentle slap on the wrist with minimal consequences.

I'm personally working very hard on an escape plan and focusing on getting out of HE. In the meantime I'm not going to be looking too hard for academic misconduct (AI or otherwise) when I'm trying to streamline my workload to free up time to moon light.

Cynical yes - but I think in my institution with its current (not very good) management, the battle has already been lost. When you become "recruit at all costs" you've already lost the tools to retain academic standards.

We've been forced to remove all exams just as AI came in. So we get ai slop for lost answers and the occasional one which demonstrates they can prompt ai well.

But it's getting to the point where we are encouraged to use ai to design the assessments, design the marking rubrics, the students ai the assignment and eventually ai will mark it.

To be honest I personally am not going to bother fighting what might be a loosing battle. Trying to investigate which scripts are AI vs legit is going to become a huge time suck. Currently the marking time allocated in my workload is far lower then it normally takes. So hunting for AI scripts personally would just make the work of marking even more miserable.

Agree with this, just had an assignment submitted with a really high AI score, over 80%. The time spent looking at references, etc and trying to see if any of them are fake references is substantial. They all actually look ok so next question is how do we prove it? We can’t. I’ve no idea how accurate the software is, if the student denies it the university says we can’t do anything.

Our allocated marking time isn’t enough for this. Though I have started dabbling with using AI to mark assignments. So far have been marking them myself and then asking AI what it thinks and getting very similar grades. Seems like we just might use AI to mark AI. 🙄🙈

I don't think you can upload student assignments to ai to check grades because then you've let ai scrape/use it unless you have a specific data sharing agreement with the ai company (which are generally set up as business contracts with the ai companies).

Indeed @Barrellturn the more assignments that are uploaded the better it gets at doing assignments. I think that horse has well and truly bolted though. We've gone back to hand written exams and vivas whenever possible - but dissertations are a nightmare and coursework and I agree with @Spronkles that HE is a 'dumpster fire' - AI is just the accelerant. It is fundamentally not fit for purpose and I expect enthusiasm to get into debt to experience it to wane considerably over the next decade given the collapsing graduate employment market.

Our uni has such an agreement so anything we upload can’t be scraped

My main issue is how clueless management are about the reality of AI. How pervasive and their solution is to put all work in policing it on the (overworked) teaching staff.

In our uni it seems like they are working against us - for instance they are the reason we lost every in person exam. So they can market the course as "100% coursework". We had exam's as late as last year but they finally killed off the last one (and pedagogy was never discussed in the process)

Our course is media focused and our team did find getting students to do video essays as partial solution. Even if students are using AI in their scripting, they are forced to research and edit clips and assemble an edited visual sequences. Not yet possible to to on AI. So even if AI is involved we could still see the students were doing some form of "work". We've been using video essays for about 5 or 6 years with no problems and students generally liked the assessments. However this year many students are resisting the assessment complaining about it and requesting we go back to a written essay..... Why would students actually request "more writing" over a practical assessment? Oh yes making just words is easier with AI.

Unfortunately management's approach currently is to over analyse student feedback and force changes on modules/assessment to apease our new masters (students). So we are potentially about to be forced to remove some of the video essay tasks to make them text only... yay.

As a "recruting" low tariff course I would say at least 50% of the cohort are incredibly lazy. Any attempt by teaching staff to get the students to some actual work, to challenge them... is met by poor module feedback.. this is then jumped on by management with the result (usually) of "streamlining the module to enhance student success"... or make it easier because the lazy ones moan

What would you say to a student who doesn't use AI but feels they will be at a disadvantage when they join the job market because they will be in competition with people who get the same grade using AI and on top of that employers will assume this batch of students all use AI? How would you reassure them?

I know they are learning the skills they are meant to be learning and that if they do find employment those skills will emerge and will benefit hugely but the challenge is getting the job in the first place.

I thought most of the skill in using LLMs ( I presume this entire thread is about LLMs and not machine learning or some kind of AI I haven't thought of )

was in the prompt engineering skill. Yet also, the "enormous benefit" of using LLMs is soon you won't need to be clever in prompt engineering; the LLM will figure out what you need and ask questions to get the results you want it to write.

For the programmers designing the LLMs: lots of job opportunities. For people focusing on prompt engineering as an important job skill: very limited short-lived career path.

Maybe PP means everyone should learn to OOP and C++ to be competitive in future job markets ?

I would encourage them to use ai for some things that save time and are reliable. Grammar checking is to be expected now. I've not seen a typo in an essay for a while now.

I also use it for rubber ducking.

But I think it's important students continue to read full articles. Don't ask for summaries until you've read the actual article and use it to check your own understanding. Don't use it for constructing argument.

I think it's important to understand what it can and can't do, how it approaches things, what its drawbacks are. You have to use it to understand that. So getting used to promoting is an important skill for any graduates in the next year or so.