Precision of academic plagiarism detection: A descriptive analysis of Artificial Intelligence verifiers

Luis Ebano Amor Oliva; Erika Guadalupe May Guillermo

doi:10.19136/etie.v8n16.6338

Authors

M.A.Y.D.E. Luis Ebano Amor Oliva Universidad Juárez Autónoma de Tabasco
Dra. Erika Guadalupe May Guillermo Tecnológico Nacional De México/ ITS de la Región Sierra

DOI:

https://doi.org/10.19136/etie.v8n16.6338

Keywords:

Artificial intelligence, ChatGPT, Large language models, Plagiarism

Abstract

Some authors argue that the use of Artificial Intelligence (AI) should be considered plagiarism and have proposed the use of verifier tools to combat this type of plagiarism. Among these Artificial Intelligences is ChatGPT, which has revolutionized the world with its ability to produce human-like text. The purpose of this work is to identify the level of accuracy in detecting plagiarism using Artificial Intelligence detection tools in literature writing. Fifty samples of books written and published before November 2022 were collected, and ChatGPT was asked to generate another 50 literature samples from different genres. The original human and AI-generated content were analyzed using four plagiarism detection tools, which were moderately successful in identifying human content but had varying degrees of effectiveness in detecting AI-generated content. Copy Leaks scored 99% on the F-score, Content at Scale 79%, Scribber 25%, and ZeroGPT 69%. This paper has an explanatory approach with a cross-sectional design, with quantitative analysis of the data collected. ChatGPT as the potential to displace human writers and the use of these AI verifiers can aid schools and editorial houses to distinguish original human content and that generated by IA. We exhort AI verifiers to improve their algorithms used to identify plagiarism, and for schools to incorporate these types of tools in the design of strategic pedagogies for future papers.

References

Adeshola, I., & Adepoju, A. P. (2023). The opportunities and challenges of ChatGPT in education. Interactive Learning Environments, 32(10), 6159–6172. https://doi.org/10.1080/10494820.2023.2253858

Almeida, R. M. V., de Albuquerque Rocha, K., Catelani, F., Fontes-Pereira, A. J., & Vasconcelos, S. M. (2016). Plagiarism allegations account for most retractions in major Latin American/Caribbean databases. Science and engineering ethics, 22(5), 1447-1456. https://doi.org/10.1007/s11948-015-9714-5

Biörck, J., & Eriksson, S. (2023). Detecting Plagiarism with ChatGPT Using Prompt Engineering [Tesis de licenciatura, Escuela Electrical Engineering and Computer Science]. https://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-330996

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., & Xie, X. (2018). A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 37(4), 1-45. https://doi.org/10.48550/arXiv.2307.03109

Content at Scale (2024). Introducing WriteWell to Humanize Your AI Writing with Unlimited AI Detector Credits. BrandWell. https://brandwell.ai/ai-content-detector/

Cotton, D. R., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148

Copyleaks (2024). AI Detector. Copyleaks. https://copyleaks.com/ai-content-detector

Currie, G. M. (2023). Academic integrity and artificial intelligence: is ChatGPT hype, hero or heresy? Seminars in Nuclear Medicine. WB Saunders, 53 (5) https://doi.org/10.1053/j.semnuclmed.2023.04.008

Dahl, S. (2007). Turnitin®: The student perspective on using plagiarism detection software. Active Learning in Higher Education, 8(2), 173-191. https://doi.org/10.1177/1469787407074110

Deng, J., & Lin, Y. (2022). The benefits and challenges of ChatGPT: An overview. Frontiers in Computing and Intelligent Systems, 2(2), 81-83. https://doi.org/10.54097/fcis.v2i2.4465

Fan, J., Upadhye, S., & Worster, A. (2006). Understanding receiver operating characteristic (ROC) curves. Canadian Journal of Emergency Medicine, 8(1), 19-20. doi:10.1017/S1481803500013336

Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. BioRxiv, 1-18. https://doi.org/10.1101/2022.12.23.521610

Goutte, C., & Gaussier, E. (2005). A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. En 27th European Conference on IR Research, ECIR 2005, Santiago de Compostela, Spain. https://doi.org/10.1007/978-3-540-31865-1_25

Grassini, S. (2023). Shaping the future of education: exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692

Hua, H. U., Kaakour, A. H., Rachitskaya, A., Srivastava, S., Sharma, S., & Mammo, D. A. (2023). Evaluation and comparison of ophthalmic scientific abstracts and references by current artificial intelligence chatbots. JAMA ophthalmology, 141(9), 819-824. DOI:10.1001/jamaophthalmol.2023.3119

Jarrah, A. M., Wardat, Y., & Fidalgo, P. (2023). Using ChatGPT in academic writing is (not) a form of plagiarism: What does the literature say. Online Journal of Communication and Media Technologies, 13(4), e202346. https://doi.org/10.30935/ojcmt/13572

Kacena, M. A., Plotkin, L. I., & Fehrenbacher, J. C. (2024). The Use of Artificial Intelligence in Writing Scientific Review Articles. Current Osteoporosis Reports, 22, 115-121. https://doi.org/10.1007/s11914-023-00852-0

Khalil, M., & Er, E. (2023). Will chatgpt get you caught? rethinking of plagiarism detection. En In: Zaphiris, P., Ioannou, A. (Coordinadores), 10th International Conference, LCT 2023, Held as Part of the 25th HCI International Conference, HCII 2023, Copenhagen, Denmark. https://doi.org/10.1007/978-3-031-34411-4_32

Khaled, F., & Al-Tamimi, M. S. H. (2021). Plagiarism detection methods and tools: An overview. Iraqi Journal of Science,62(8), 2771-2783. https://doi.org/10.24996/ijs.2021.62.8.30

Kumar, P. M., Priya, N. S., Musalaiah, S. V. V. S., & Nagasree, M. (2014). Knowing and avoiding plagiarism during scientific writing. Annals of medical and health sciences research, 4(3), 193-198.DOI: 10.4103/2141-9248.141957

Lancaster, T. (2003). Effective and efficient plagiarism detection [Tesis de Doctorado, School of Computing, Information Systems and Mathematics South Bank University]. https://www.academia.edu/168972/Effective_and_Efficient_Plagiarism_Detection

Landero, E. A. R., Basurto, A. C., & Ramírez, S. G. V. (2024). ChatGPT en la elaboración de trabajos académicos. Ingenio y Conciencia Boletín Científico de la Escuela Superior Ciudad Sahagún, 11(22), 157-161. https://doi.org/10.29057/escs.v11i22.12553

Masic, I. (2012). Plagiarism in scientific publishing. Acta Inform Med., 20(4), 208-213. DOI:10.5455/aim.2012.20.208-213

Naik, R. R., Landge, M. B., & Mahender, C. N. (2015). A review on plagiarism detection tools. International Journal of Computer Applications, 125(11). DOI: 10.5120/ijca2015906113

Rathore, B. (2023). Future of AI & generation alpha: ChatGPT beyond boundaries. Eduzone: International Peer Reviewed/Refereed Multidisciplinary Journal, 12(1), 63-68. https://doi.org/10.56614/eiprmj.v12i1y23.254

Rivera, H. (2023). La integridad científica ante los plagios fabricados con el ChatGPT. Revista Médica del Instituto Mexicano del Seguro Social, 61(6), 857-862. doi: 10.5281/zenodo.10064458

Scribber (2024). Free AI Detector. Scribber. https://www.scribbr.com/ai-detector/

Sharma, H., & Verma, S. (2020). Insight into modern-day plagiarism: The science of pseudo research. Tzu-Chi Medical Journal, 32(3), 240-244. DOI: 10.4103/tcmj.tcmj_210_19

Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006). Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. En Sattar, A., Kang, Bh. (Coordinadores), AI 2006: Advances in Artificial Intelligence. Berlin Heidelberg. https://doi.org/10.1007/11941439_114

Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large language models in medicine. Nature medicine, 29, 1930-1940. https://doi.org/10.1038/s41591-023-02448-8

Weber-Wulff, D. (2014). False feathers: A perspective on academic plagiarism. Springer Science & Busines. DOI: 10.1007/978-3-642-39961-9

Wu, T., He, S., Liu, J., Sun, S., Liu, K., Han, Q. L., & Tang, Y. (2023). A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5), 1122-1136. DOI: 10.1109/JAS.2023.123618

Youmans, R. J. (2011). Does the adoption of plagiarism-detection software in higher education reduce plagiarism?. Studies in Higher Education, 36(7), 749-761. https://doi.org/10.1080/03075079.2010.523457

ZeroGPT (2024). Trusted GPT-4 & ChatGPT AI Content Detector AI Checker Tool. ZEROGPT.cc. https://zerogpt.cc/

Zhou, Z. H. (2021). Machine learning. Springer Nature. Singapore https://doi.org/10.1007/978-981-15-1967-3