Technique for Identifying Texts Generated by Large Language Models

The article presents a method for identifying Russian-language texts generated by large language models (LLMs). The method was developed with a focus on short messages from 100 to 200 characters long. The relevance of the work is due to the widespread use of generative models, such as GPT-3.5, GPT-4o, LLaMA, GigaChat, DeepSeek, and Yandex GPT. The method is based on an ensemble of machine learning models, and features of three levels are also used: linguistic (structure, punctuation, morphology, lexical diversity), statistical (entropy, perplexity, n-gram frequency), and semantic (RuBERT embeddings). LightGBM, BiLSTM, and the pre-trained transformer model RuRoBERTa are used as basic models, combined by stacking through logistic regression. The choice of a hybrid ensemble approach is due to the desire to take into account features at different levels of the text hierarchy and to ensure the reliability of classification in the context of different topics of generated texts, versions, and types of language models. The use of an ensemble is an advantage in the analysis of short texts, since LightGBM, based on averaged indicators, is less sensitive to length (the perplexity metric is already averaged over the entire text), while BiLSTM and RoBERTa are able to identify local features of an LLM text, and not just global ones. The dataset of natural texts includes more than 2.8 million user comments from the VK social network. The LLM text dataset contains 700 thousand texts generated by seven relevant large language models. Topic modeling (LDA) and role generation using prompt engineering were used in the text generation. The methodology was evaluated on open datasets of Russian-language LLM texts. The experimental results showed an accuracy of up to 0.95 in the binary classification task (Human–LLM) and up to 0.89 in the multi-class task of determining the model-generator. The method demonstrates robustness to the diversity of sources, styles, and LLM versions.

large language models, neural networks, machine learning, text generation, ensemble of classifiers, text features