Evaluation von LLM- und Intent-basierten Ansätzen zur Umsetzung eines Chatbots für die Unterstützung bei der Studienorganisation

Conversational user interfaces such as chatbots offer great potential to support students in organizing their
support students in organizing their studies in addition to existing advisory services. In particular
the field of Large Language Models (LLMs) in particular, new approaches to the construction of such
to the construction of such chatbots. However, these are associated with opportunities and risks, so that
so that the choice of a suitable approach must be carefully considered. This article examines three
approaches to creating such chatbots are examined and compared with each other: ChatGPT with Retrieval
Augmented Generation (RAG), the open-source LLM Mistral with RAG and an intent-based chatbot.
chatbot. The approaches are compared in terms of the quality of the answers and risks (e.g. hallucinations).
compared. Overall, it is shown that all approaches can potentially be used to support the organization of studies.
study organization. However, no clear recommendation for one approach can be derived from the findings.
recommendation for one approach, which is why a hybrid chatbot should be investigated in further work.
should be investigated in further work.

Chatbot

Evaluation

Artificial intelligence

Large Language Models

Retrieval
Augmented Generation