UCC study confirms AI cannot fully write like a human

Olivia Kelleher

Researchers at University College Cork (UCC) have completed a study which indicates that AI-generated writing continues to display distinct stylistic patterns which set it apart from human prose.

The study is the first in the world to use literary stylometry, computational methods traditionally used to identify authorship, to compare the writing styles of humans and large language models like ChatGPT across creative texts.

Researchers found that AI can generate polished, fluent prose, but its writing continues to follow a narrow and uniform pattern.

Human authors display far greater stylistic range, shaped by personal voice, creative intent, and individual experience.

Dr James O’Sullivan, of the UCC School of English and Digital Humanities, analysed hundreds of short stories written by people alongside pieces generated by AI systems.

By examining subtle linguistic markers, including the frequency of common words, the research reveals clear and consistent stylistic differences.

Dr O’Sullivan said analysis shows a clear stylistic divide between human writing and text generated by large language models.

AI systems like GPT-3.5, GPT-4, and Llama 70B produce tightly grouped clusters, each reflecting the uniform patterns typical of a given model. In contrast, human texts show far greater variation and individuality

Researchers found that GPT-4 writes with even more consistency than GPT-3.5, but both remain distinct from human work. While GPT-3.5 occasionally comes close to human style, such moments are rare.

Dr O’Sullivan says that AI models produce compact, predictable styles, while human writing remains more varied and idiosyncratic, traits that reflect individuality and creative intention.

“While AI writing is often polished and coherent, it tends to show more uniformity in word choice and rhythm. In contrast, human writing remains more varied and idiosyncratic, reflecting individual habits, preferences and creative choices.

“Even when ChatGPT tries to sound human, its writing still carries a detectable fingerprint, which suggests that computers and people don’t yet write in quite the same style.”

The researchers caution against using stylometry as an AI detection tool in education. However, they highlight its value in understanding how human expression differs from algorithmic generation, offering new insights into what makes writing recognisably human.

Dr O’Sullivan said stylometry can reveal broad patterns across large bodies of text. However, he insists that it has no place in judging authorship in education.

“Students’ writing shifts from task to task and is shaped by context, support, and lived experience, which makes stylometric detection both unreliable and ethically questionable in academic integrity cases.”

Dr O’Sullivan said that the findings point to the need for broader datasets, new prompts, and testing with emerging models, as well as closer attention to the ethical and creative questions raised by the growing use of generative AI.

He added that its one thing for a Large Language Model (LLM) to reliably produce an email or summary report.

However, the capacity to automate the production of literature raises “profound ethical and philosophical concerns about authenticity, originality, and the very nature of authorship".

The research was led by Dr O’Sullivan and is published in Nature — Humanities and Social Sciences Communications.