Document details

The management of external service provider networks requires continuous and rigorous evaluation to ensure consistent service quality. At the company TRUST, the assessment of provider performance and compliance with Service Level Agreements (SLAs) is traditionally a manual, unstructured, and time-consuming process. This inefficiency limits the rapid identification of critical operational issues and delays the implementation of corrective actions. To address this problem, this dissertation proposes and develops an Artificial Intelligence-supported Decision Support System (DSS) designed to automate the evaluation of service providers and generate actionable compliance interventions. To overcome the inherent hallucination risks of generative AI, the implemented prototype introduces a hybrid architecture that strictly decouples deterministic SLA mathematical calculations from probabilistic text generation. By integrating a structured SQL data extraction layer with a custom Retrieval-Augmented Generation (RAG) pipeline, the system automatically analyzes provider metrics—such as the time required for a first consultation—and grounds its generated responses in historical contractual documents and standard operational procedures. To guarantee absolute corporate data privacy, all processing was executed locally. During the experimental phase, various open-weights Large Language Models (LLMs) were evaluated. The results revealed a trade-off that does not track model scale: the two mid-size models (mistral:latest, qwen2.5:latest) never broke the requested schema, whereas phi4-mini and the advanced reasoning model deepseek-v2:16b both fell short of full structural compliance — the latter by over-extending the prompt constraints — while producing markedly superior European Portuguese in expert review. Furthermore, the introduction of a multi-agent “LLM-as-a-Judge” validation pipeline successfully filtered out unformatted or ungrounded outputs before human review. Ultimately, the findings prove that this hybrid, privacy-preserving automated approach provides a more structured and fully traceable analysis than the current manual process. By resolving the compliant majority of the network deterministically, the system confines inference to the cases that require it, demonstrating the operational viability and safety of local open-weights LLM architectures in enterprise decision support.