There’s an intriguing paper featured in WINE2023’s tutorials:
ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models
Yikang Liu, Ziyin Zhang, Wanyang Zhang, Shisen Yue, Xiaojing Zhao, Xinyuan Cheng, Yiwen Zhang, Hai Hu
ABSTRACT
AI generated content (AIGC) presents considerable challenge to educators around the world. Instructors need to be able to detect such text generated by large language models, either with the naked eye or with the help of some tools. There is also growing need to understand the lexical, syntactic and stylistic features of AIGC. To address these challenges in English language teaching, we first present ArguGPT, a balanced corpus of 4,038 argumentative essays generated by 7 GPT models in response to essay prompts from three sources: (1) in-class or homework exercises, (2) TOEFL and (3) GRE writing tasks. Machine-generated texts are paired with roughly equal number of human-written essays with three score levels matched in essay prompts. We then hire English instructors to distinguish machine essays from human ones. Results show that when first exposed to machine-generated essays, the instructors only have an accuracy of 61% in detecting them. But the number rises to 67% after one round of minimal self-training. Next, we perform linguistic analyses of these essays, which show that machines produce sentences with more complex syntactic structures while human essays tend to be lexically more complex. Finally, we test existing AIGC detectors and build our own detectors using SVMs and RoBERTa. Results suggest that a RoBERTa fine-tuned with the training set of ArguGPT achieves above 90% accuracy in both essay- and sentence-level classification. To the best of our knowledge, this is the first comprehensive analysis of argumentative essays produced by generative large language models. Machine-authored essays in ArguGPT and our models will be made publicly available at this https URL
So, ArguGPT is an AI designed to discriminate whether essays, particularly argumentative ones, and responses to GRE and TOEFL writing tasks, are penned by humans or GPTs. Its author team diverses among machine learning experts as well as profs. and students from the School of Foreign Language, SJTU. Hence using some student essay corpus would be quite handy a choice.
From an undergraduate perspective, I find the ArguGPT tool they developed somewhat interesting. The authors have generously open-sourced their software and even established an online platform on HuggingFace for public experimentation. So, I tested it with some GPT-generated content from my collection, including my application essay:
(take a look, and at the end we’ll unveil how much it’s GPT-generated)
My proceeding work involves redesigning Amazon’s Buy-Box. The Buy-Box of each merchandise highlights one seller by imposing a search friction on all others. Current design of the Buy-Box yields a non-stable market dynamic reflecting wild fluctuation in prices. We proved the non-existence of any pure Nash Equilibrium in the current system and developed another optimal mechanism that stabilizes market outcomes, maximizes social welfare and benefits all market participants.
This research and my academic pursuits are rooted in the interdisciplinary of market design and algorithmic game theory (AGT). I aspire to leverage the power and beauty of mathematics to economic theory, applications ranges from design of online advertising auctions to organ donation systems. This ambition has been underpinned by solid mathematical training over the past few years.
Moreover, I believe that game theorists need to be sensitive to cultural and psychological factors to understand the incentives and constraints that people face. (90% AI) Acute perception, mathematical skills and a deep sense of empathy embodies the essence of a modern economist, and is what I’m committed to becoming.
the prediction result of ArguGPT
According to ArguGPT, the essay appears to be entirely human-written, with only one sentence flagged as potentially AI-generated. But here’s the twist: that lone sentence is actually the only part that is not written by GPT. Ha!