Artificial intelligence tools develop rapidly and provide automated assistance across various stages of the research process. This includes mapping, organizing, selecting, and assessing research literature, as well as conducting literature reviews. Furthermore, artificial intelligence, particularly in the form of large language models (LLMs), can contribute to the writing process by supporting several tasks, from developing research questions and structuring your text to composing and proofreading academic documents. This raises important questions about the usefulness, appropriateness, legitimacy, and ethical consequences of using such tools.
The following section will mainly focus on the use of generative AI in the academic writing process but will also address related activities such as literature search, assessment, and publishing.
AI Literacy
Artificial intelligence (AI) refers to computer systems capable of performing tasks that typically require human intelligence, such as problem-solving, speech and pattern recognition, decision-making, and language translation. Large language models (LLMs), so-called "chatbots", are a generative subset of AI with the ability to process and generate human-like text. These models can assist scholarly writers in drafting, editing, translating and summarizing academic content.
Large language models are trained on large amounts of texts and generate language by statistically predicting the likelihood of what word should come next in a sentence. Although LLMs can solve a wide range of tasks in a seemingly sensible way, it is important to understand that they are essentially sophisticated guessing machines. This inclination for uncritical mixing of fact and fiction is ill-suited for academic contexts and involves many inherent challenges that need careful consideration.
In response to these challenges, researchers need to develop their AI literacy. This literacy encompasses a set of competences that enable individuals to effectively communicate and collaborate with AI technologies, and to critically evaluate these technologies (Long & Magerko, 2020). Essential in this regard is the ability to assess the output of AI, as documents produced by LLMs challenge the traditional notion of authorship that academic texts are based on. Consequently, it is important to subject AI-based results to rigorous source criticism.
Challenges
All texts produced by LLMs – which also can be described as synthetic texts – must be carefully examined and considered before they are used in academic output, as the quality of synthetic texts is limited by the caliber and volume of the texts they have been trained on. In addition, LLMs are prone to violate copyright and plagiarize sources that are not properly referenced, as well as to hallucinate. Thus, LLMs and other AI tools require active human interaction in order to contribute to innovation (Jabotinsky & Sarel, 2023)..
The “Halo effect”: While texts produced by LLMs are often eloquent, this eloquence may also be deceptive. These synthetic texts are good at producing a “halo effect” (Smith et al., 2024), that is an impressive textual surface that nevertheless conceals a vague and superficial content, in particular when it comes to more complex questions. Texts generated by LLMs ultimately lack the precision that is essential to academic writing in its quest for truth. These models are trained at imitating form, but do not understand the context or the content of your research. Be also aware that longer texts produced by LLMs may contain contradictory statements. In any case, as the author of an academic work, it is you who bear the responsibility for its content.
Hallucination: The most obvious risk is LLMs’ tendency to hallucinate, that is to make false claims. These claims are typically presented in an eloquent and convincing manner, often backed up by sources that may or may not exist in the real world. Synthetic texts are indifferent towards truth; they are first and foremost designed to imitate human language (Hicks, Humphries & Slater, 2024). All claims and sources should therefore be carefully revised before you proceed.
AI and authorship
The notion of authorship is fundamental to our academic culture and implies integrity and responsibility for the published work. By contrast, LLMs are not committed to notions of responsibility, integrity and not even to truth. Because of this, journals and publishers do not accept LLMs, such as ChatGPT, as co-authors of submitted texts. Instead of expressing an author's intentions, LLMs imitate texts they have been trained on, and may therefore (re)produce false information, fake news and biases.
Bias: By their nature, LLMs are limited by their training, and this could lead to various forms of bias in the generated texts. These tools often have a defined training period and cannot update their knowledge in real-time. This means that they might be ignorant of the latest development within research and society. The data used to train an LLM might also be biased by their cultural context or censorship.
Cognitive offloading: While LLMs and other AI tools do have a huge potential to effectivize and even automatize research, they may also eliminate substantial parts of the research process, above all those that are made up of human, intellectual engagement, creativity and innovation. A worst-case scenario is the “potential atrophy of critical thinking skills essential for any kind of critique, innovation and human development (Sætra, 2025). Future researchers should therefore make careful considerations as to what extent their work will benefit from AI tools.
Opportunities
The wave of generative AI-tools offers some new opportunities for your research process. Generative AI can serve as a valuable assistant in your creative process. It is up to you to determine how—and to what extent—you wish to integrate this technology into your own research practice.
As a start you should find out what kind of AI-based tools your university provides and supports. Different local guidelines regarding the use of AI tools are usually provided. You can find a good overview of ethical guidelines for using generative intelligence in the “Living guidelines on the responsible use of generative AI in research” developed by the European Union (European Research Area Platform, 2024). However, note that ethical considerations are your own responsibility.
"Continuously learn how to use generative AI tools properly to maximise their benefits, including by undertaking training. Generative AI tools are evolving quickly, and new ways to use them are regularly discovered.”
(European Research Area Platform, 2024)
Initial stage: It can be a good idea to check the guidelines on what is acceptable use of LLMs among relevant journals and publishers. Journals’ author instructions may require that you declare and document how AI was used.
In the initial phase AI tools can contribute in various ways, for example:
- Searching for literature:
In the process of mapping earlier research, AI tools may be used to make a search strategy, identify search terms and keywords and the choices of databases. Several major databases now integrate AI tools in their services. - Analyzing data:
In the process of analyzing data, such as transcribed interviews, AI may be used as a supportive tool to discover additional themes that you did not identify in your initial analysis. Be aware of which data is allowed to use in relation to the AI tools supported by your university.
A view of LLMs as a conversation partner:
“This new conversational AI model can be your friend, philosopher, and guide ... and even your worst enemy.” (Chatterjee, J., & Dethlefs, N., 2023)
Writing stage: AI tools have the ability to be your creative partner. Using LLMs as a study-buddy when brainstorming can be fruitful. Sometimes it is difficult to start the writing process. One way to use LLMs is to brainstorm in terms of asking for topics and concepts related to your main research question, or for a possible way to structure your text. However, you should never ask an LLM to produce new text, since this would be unethical, and the results would often be of insufficient quality. Also uploading longer texts to have them rewritten would violate the idea of authorship.
Revision stage: At this stage, checking grammar and asking for suggestions on how to rephrase a sentence is normally acceptable. However, you need to check the journal’s rules for AI use and also consider declaring the use of AI for revision in your acknowledgement section.
Publishing stage: In the process of publishing a paper one often needs to deliver additional documentation like CV, statements of co-working activities, and other formal documents. An LLM may help you put together such documents.
References
Chatterjee, J., & Dethlefs, N. (2023). This new conversational AI model can be your friend, philosopher, and guide ... and even your worst enemy. Patterns, 4(1), 100676–100676. https://doi.org/10.1016/j.patter.2022.100676
Hicks, M.T., Humphries, J. & Slater, J. (2024). ChatGPT is bullshit. Ethics and Information Technology, 26, Article 38. https://doi.org/10.1007/s10676-024-09775-5
Jabotinsky, H. Y. & Sarel, R. (2023). Co-authoring with an AI? Ethical Dilemmas and Artificial Intelligence. Arizona State Law Journal, Forthcoming. http://dx.doi.org/10.2139/ssrn.4303959
Living guidelines on the responsible use of generative AI in research published | European Research Area Platform. (2024). https://european-research-area.ec.europa.eu/news/living-guidelines-responsible-use-generative-ai-research-published
Long, D., & Magerko, B. (2020). What is AI Literacy? Competencies and Design Considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 598–598. https://doi.org/10.1145/3313831.3376727
Smith, G.R. et al. (2024). Ten simple rules for using large language models in science, version 1.0. PLoS Comput Biol 20(1): e1011767. https://doi.org/10.1371/journal.pcbi.1011767
Sætra, H. (2025) The rise of the research automaton: science as process or product in the era of generative AI? AI & Society. https://doi.org/10.1007/s00146-025-02557-7
