Publication

Enhancing Customer Support with Open-Source LLMS: Development Of A Question Answering Application: Evaluating the potential of a LLM-based Question Answering application to improve customer support in Small and Medium-sized Enterprises as a cost-effective solution to lower the barrier of AI adoption

View document open_in_new

Bibliographic Details

Summary:	Today's customers expect 24/7 personalised service across multiple channels, putting immense pressure on organisations to deliver exceptional experiences while maintaining efficiency. Generative Artificial Intelligence (AI) is emerging as a potential solution, promising to automate tasks and personalise interactions. However, this technology remains largely out of reach for Small and Medium-sized Enterprises (SMEs) due to financial and technical constraints. This research investigates the potential of Large Language Models (LLMs) to enhance customer support capabilities in SMEs, with a particular focus on technical question answering. The research develops a Question Answering (QA) application that uses RetrievalAugmented Generation (RAG) to optimise information retrieval from technical documents. The application developed exclusively using open-source tools and LLMs, aims to provide a cost-efficientsolution for SMEsto lower the barrier of AI adoption. Guided by a Design Science Research Methodology (DSRM), the research details the development of the application and evaluates its performance and usability. The results show that while the application successfully returns answers with high factual accuracy and explainability scores, it suffers from high latency issues, with response times unsuitable for real-world use. The method used to incorporate human feedback when the LLM cannot answer a question demonstrated potential as a continuous learning mechanism for the application. However, this mechanism still needs further development to achieve consistent performance. In summary, while the open-source LLM-based QA application shows potential for improving SME customer support, significant improvements in computational resources and alternative judging approaches are required to fully realise its capabilities. Future work should focus on increasing processing speeds and exploring more powerful LLMs to reduce latency and improve answer relevance and correctness.
Main Authors:	Kickler, Jannik
Subject:	Generative Artificial Intelligence Small and Medium-sized Enterprise Artificial Intelligence Adoption Large Language Models Question Answering Retrieval Augmented Generation LLM-as-a-Judge Natural Language Processing SDG 8 - Decent work and economic growth SDG 9 - Industry, innovation and infrastructure
Year:	2024
Country:	Portugal
Document type:	master thesis
Access type:	open access
Associated institution:	Universidade Nova de Lisboa
Language:	English
Origin:	Repositório Institucional da UNL