Publicação
Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks
| Resumo: | Recommender Systems have become essential mechanisms in our digital era, and while traditional evaluation emphasizes accuracy, there is an emerging demand for comprehensive system performance assessment. This study aims to systematically identify frameworks and techniques that evaluate and balance multidimensional quality in Recommender Systems. Specifically, it addresses three core questions: What quality evaluation measures exist in Recommender Systems? Which methods effectively convey and balance multidimensional quality? What novel techniques address current evaluation challenges across various domains? The literature review was conducted using Scopus as the primary source. The inclusion criteria required peer-reviewed journal articles published in English between 2020 and 2024. Studies were excluded if they lacked focus on quality assessment, domain application, or trade-off balancing. Thirty-one studies were identified through a multi-stage process that included automated text mining, NLP-based abstract screening, and full-text analysis guided by the PRISMA 2020 framework. Risk of bias was mitigated by combining automated semantic filtering with manual review, ensuring a high-precision selection process. Results were synthesized through thematic coding and descriptive mapping. The selected studies encompassed a broad range of quality dimensions, in addition to accuracy, with hybrid and deep learning-based models prevailing, although with limited practical validation. Several integrative frameworks have been identified for balancing conflicting metrics. However, their empirical deployment remains scarce. Limitations of the evidence include an overreliance on offline evaluations, a lack of standardized metrics across studies, and limited guidance for realworld implementation. Despite advancements, real-time testing and stakeholder-driven metric weighting remain underexplored. This literature review introduces a layered visual framework to support systematic, stakeholder-aligned evaluation of Recommender Systems. By mapping contextual objectives to measurable outcomes and explicitly balancing trade-offs, it provides a practical guide for both researchers and practitioners seeking to design Recommender Systems that are effective, fair, and transparent across domains. |
|---|---|
| Autores principais: | Del Río, Daniela Sofía Erazo |
| Assunto: | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso embargado |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| _version_ | 1865920626159517696 |
|---|---|
| author | Del Río, Daniela Sofía Erazo |
| author_facet | Del Río, Daniela Sofía Erazo Del Río, Daniela Sofía Erazo |
| author_role | author |
| contributor_name_str_mv | António, Nuno Miguel da Conceição RUN |
| country_str | PT |
| creators_json_str | [{\"Person.name\":\"Del Río, Daniela Sofía Erazo\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | António, Nuno Miguel da Conceição RUN |
| datacite.creators.creator.creatorName.fl_str_mv | Del Río, Daniela Sofía Erazo |
| datacite.date.Accepted.fl_str_mv | 2025-11-03T00:00:00Z |
| datacite.date.available.fl_str_mv | 2028-11-03T00:00:00Z |
| datacite.date.embargoed.fl_str_mv | 2028-11-03T00:00:00Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_f1cf |
| datacite.subjects.subject.fl_str_mv | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| datacite.titles.title.fl_str_mv | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| dc.contributor.none.fl_str_mv | António, Nuno Miguel da Conceição RUN |
| dc.creator.none.fl_str_mv | Del Río, Daniela Sofía Erazo |
| dc.date.Accepted.fl_str_mv | 2025-11-03T00:00:00Z |
| dc.date.available.fl_str_mv | 2028-11-03T00:00:00Z |
| dc.date.embargoed.fl_str_mv | 2028-11-03T00:00:00Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | http://hdl.handle.net/10362/190933 |
| dc.language.none.fl_str_mv | eng |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_f1cf |
| dc.subject.none.fl_str_mv | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| dc.title.fl_str_mv | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_bdcc |
| description | Recommender Systems have become essential mechanisms in our digital era, and while traditional evaluation emphasizes accuracy, there is an emerging demand for comprehensive system performance assessment. This study aims to systematically identify frameworks and techniques that evaluate and balance multidimensional quality in Recommender Systems. Specifically, it addresses three core questions: What quality evaluation measures exist in Recommender Systems? Which methods effectively convey and balance multidimensional quality? What novel techniques address current evaluation challenges across various domains? The literature review was conducted using Scopus as the primary source. The inclusion criteria required peer-reviewed journal articles published in English between 2020 and 2024. Studies were excluded if they lacked focus on quality assessment, domain application, or trade-off balancing. Thirty-one studies were identified through a multi-stage process that included automated text mining, NLP-based abstract screening, and full-text analysis guided by the PRISMA 2020 framework. Risk of bias was mitigated by combining automated semantic filtering with manual review, ensuring a high-precision selection process. Results were synthesized through thematic coding and descriptive mapping. The selected studies encompassed a broad range of quality dimensions, in addition to accuracy, with hybrid and deep learning-based models prevailing, although with limited practical validation. Several integrative frameworks have been identified for balancing conflicting metrics. However, their empirical deployment remains scarce. Limitations of the evidence include an overreliance on offline evaluations, a lack of standardized metrics across studies, and limited guidance for realworld implementation. Despite advancements, real-time testing and stakeholder-driven metric weighting remain underexplored. This literature review introduces a layered visual framework to support systematic, stakeholder-aligned evaluation of Recommender Systems. By mapping contextual objectives to measurable outcomes and explicitly balancing trade-offs, it provides a practical guide for both researchers and practitioners seeking to design Recommender Systems that are effective, fair, and transparent across domains. |
| dirty | 0 |
| eu_rights_str_mv | embargoedAccess |
| format | masterThesis |
| fulltext.url.fl_str_mv | https://run.unl.pt/bitstreams/95f58a5d-7fdf-4b54-9765-22bcd1bce8a9/download |
| id | run_e4f0bf067b2c3cdc9f3aaa99becb6ce0 |
| identifier.url.fl_str_mv | http://hdl.handle.net/10362/190933 |
| instacron_str | unl |
| institution | Universidade Nova de Lisboa |
| instname_str | Universidade Nova de Lisboa |
| language | eng |
| network_acronym_str | run |
| network_name_str | Repositório Institucional da UNL |
| oai_identifier_str | oai:run.unl.pt:10362/190933 |
| organization_str_mv | urn:organizationAcronym:unl |
| person_str_mv | Del Río, Daniela Sofía Erazo |
| publishDate | 2025 |
| reponame_str | Repositório Institucional da UNL |
| repository_id_str | urn:repositoryAcronym:run |
| service_str_mv | urn:repositoryAcronym:run |
| spelling | engpt_PTRecommender Systems have become essential mechanisms in our digital era, and while traditional evaluation emphasizes accuracy, there is an emerging demand for comprehensive system performance assessment. This study aims to systematically identify frameworks and techniques that evaluate and balance multidimensional quality in Recommender Systems. Specifically, it addresses three core questions: What quality evaluation measures exist in Recommender Systems? Which methods effectively convey and balance multidimensional quality? What novel techniques address current evaluation challenges across various domains? The literature review was conducted using Scopus as the primary source. The inclusion criteria required peer-reviewed journal articles published in English between 2020 and 2024. Studies were excluded if they lacked focus on quality assessment, domain application, or trade-off balancing. Thirty-one studies were identified through a multi-stage process that included automated text mining, NLP-based abstract screening, and full-text analysis guided by the PRISMA 2020 framework. Risk of bias was mitigated by combining automated semantic filtering with manual review, ensuring a high-precision selection process. Results were synthesized through thematic coding and descriptive mapping. The selected studies encompassed a broad range of quality dimensions, in addition to accuracy, with hybrid and deep learning-based models prevailing, although with limited practical validation. Several integrative frameworks have been identified for balancing conflicting metrics. However, their empirical deployment remains scarce. Limitations of the evidence include an overreliance on offline evaluations, a lack of standardized metrics across studies, and limited guidance for realworld implementation. Despite advancements, real-time testing and stakeholder-driven metric weighting remain underexplored. This literature review introduces a layered visual framework to support systematic, stakeholder-aligned evaluation of Recommender Systems. By mapping contextual objectives to measurable outcomes and explicitly balancing trade-offs, it provides a practical guide for both researchers and practitioners seeking to design Recommender Systems that are effective, fair, and transparent across domains.application/pdfpt_PTRecommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation FrameworksDel Río, Daniela Sofía ErazoAntónio, Nuno Miguel da ConceiçãoHostingInstitutionOrganizationalRUNe-mailmailto:run@unl.ptrun@unl.ptURNurn:tid:2040734132025-11-032028-11-03T00:00:00Z2025-11-03T00:00:00ZHandlehttp://hdl.handle.net/10362/190933http://purl.org/coar/access_right/c_f1cfembargoed accessRecommender SystemsArtificial IntelligenceNatural Language ProcessingSystematic Literature ReviewSDG 9 - Industry, innovation and infrastructure958698 bytesliteraturehttp://purl.org/coar/resource_type/c_bdccmaster thesis2025-11-03http://creativecommons.org/licenses/by/4.0/http://purl.org/coar/access_right/c_f1cfapplication/pdffulltexthttps://run.unl.pt/bitstreams/95f58a5d-7fdf-4b54-9765-22bcd1bce8a9/download |
| spellingShingle | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks Del Río, Daniela Sofía Erazo Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure Del Río, Daniela Sofía Erazo Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| status | NEW |
| subject.fl_str_mv | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| title | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| title_full | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| title_fullStr | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| title_full_unstemmed | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| title_short | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| title_sort | Recommender Systems Reimagined: A Systematic Review of Quality, Trade-offs, and Evaluation Frameworks |
| topic | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| topic_facet | Recommender Systems Artificial Intelligence Natural Language Processing Systematic Literature Review SDG 9 - Industry, innovation and infrastructure |
| url | http://hdl.handle.net/10362/190933 |
| visible | 1 |