Publication
Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation
| Summary: | Accurate lung cancer diagnosis and staging are critical for personalized treatment and improved patient outcomes. Traditional diagnostic methods, such as manual image examination, are prone to variability and inefficiency. Recent advancements in Deep Learning (DL) have demonstrated potential in automating lung cancer classification and staging, thereby enhancing diagnostic accuracy and efficiency. However, existing solutions often address only isolated aspects of the diagnostic process, such as tumor detection, without offering a unified system for multi-target classification that integrates clinical tools for both clinicians and patients. This study presents a unified framework for lung cancer analysis that combines medical imaging, clinical data, and Large Language Models (LLMs) to support three key tasks: tumor type classification, TNM staging, and automated treatment protocol recommendation. Image-based classification was performed using YOLOv8n, trained on two CT datasets, achieving a maximum mean average precision (mAP50) of 0.418 and an F1 score of 0.44. TNM staging was addressed through a multimodal classifier, combining the ResNet50 model with a multilayer perceptron, which fused imaging and demographic inputs. This approach yielded an average F1 score of 0.389, with the M component showing the strongest performance. For treatment generation, a Retrieval-Augmented Generation (RAG) approach was employed, combining clinical prompts with relevant documents to produce personalized protocols using the Gemini 2.0 Flash LLM. The best-performing configuration achieved a stage match of 0.54 and a BERTScore of 0.83, along with high contextual fidelity across RAGAS metrics (Faithfulness: 0.68, Answer Relevancy: 0.81, Context Precision: 0.99, and Context Recall: 0.93). This framework demonstrates the potential to support both diagnosis and treatment planning within a single integrated system, contributing to more personalized and effective clinical decision-making in oncology. |
|---|---|
| Main Authors: | Silva, Catarina Costa Pereira Nascimento da |
| Subject: | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| Year: | 2025 |
| Country: | Portugal |
| Document type: | master thesis |
| Access type: | open access |
| Associated institution: | Universidade Nova de Lisboa |
| Language: | English |
| Origin: | Repositório Institucional da UNL |
| _version_ | 1868413809032429568 |
|---|---|
| author | Silva, Catarina Costa Pereira Nascimento da |
| author_facet | Silva, Catarina Costa Pereira Nascimento da |
| author_role | author |
| contributor_name_str_mv | Castelli, Mauro Jardim, João Bruno Morais de Sousa RUN |
| country_str | PT |
| creators_json_txt | [{\"Person.name\":\"Silva, Catarina Costa Pereira Nascimento da\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | Castelli, Mauro Jardim, João Bruno Morais de Sousa RUN |
| datacite.creators.creator.creatorName.fl_str_mv | Silva, Catarina Costa Pereira Nascimento da |
| datacite.date.Accepted.fl_str_mv | 2025-10-28T00:00:00Z |
| datacite.date.available.fl_str_mv | 2025-11-10T10:44:23Z |
| datacite.date.embargoed.fl_str_mv | 2025-11-10T10:44:23Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| datacite.subjects.subject.fl_str_mv | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| datacite.titles.title.fl_str_mv | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| dc.contributor.none.fl_str_mv | Castelli, Mauro Jardim, João Bruno Morais de Sousa RUN |
| dc.creator.none.fl_str_mv | Silva, Catarina Costa Pereira Nascimento da |
| dc.date.Accepted.fl_str_mv | 2025-10-28T00:00:00Z |
| dc.date.available.fl_str_mv | 2025-11-10T10:44:23Z |
| dc.date.embargoed.fl_str_mv | 2025-11-10T10:44:23Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | http://hdl.handle.net/10362/190384 |
| dc.language.none.fl_str_mv | eng |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| dc.subject.none.fl_str_mv | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| dc.title.fl_str_mv | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_bdcc |
| description | Accurate lung cancer diagnosis and staging are critical for personalized treatment and improved patient outcomes. Traditional diagnostic methods, such as manual image examination, are prone to variability and inefficiency. Recent advancements in Deep Learning (DL) have demonstrated potential in automating lung cancer classification and staging, thereby enhancing diagnostic accuracy and efficiency. However, existing solutions often address only isolated aspects of the diagnostic process, such as tumor detection, without offering a unified system for multi-target classification that integrates clinical tools for both clinicians and patients. This study presents a unified framework for lung cancer analysis that combines medical imaging, clinical data, and Large Language Models (LLMs) to support three key tasks: tumor type classification, TNM staging, and automated treatment protocol recommendation. Image-based classification was performed using YOLOv8n, trained on two CT datasets, achieving a maximum mean average precision (mAP50) of 0.418 and an F1 score of 0.44. TNM staging was addressed through a multimodal classifier, combining the ResNet50 model with a multilayer perceptron, which fused imaging and demographic inputs. This approach yielded an average F1 score of 0.389, with the M component showing the strongest performance. For treatment generation, a Retrieval-Augmented Generation (RAG) approach was employed, combining clinical prompts with relevant documents to produce personalized protocols using the Gemini 2.0 Flash LLM. The best-performing configuration achieved a stage match of 0.54 and a BERTScore of 0.83, along with high contextual fidelity across RAGAS metrics (Faithfulness: 0.68, Answer Relevancy: 0.81, Context Precision: 0.99, and Context Recall: 0.93). This framework demonstrates the potential to support both diagnosis and treatment planning within a single integrated system, contributing to more personalized and effective clinical decision-making in oncology. |
| dirty | 0 |
| eu_rights_str_mv | openAccess |
| format | masterThesis |
| fulltext.url.fl_str_mv | https://run.unl.pt/bitstreams/73dc5028-a35e-4245-90bf-6ada1d07e0d8/download |
| id | run_6c877dfd4e71e776f486069fb7b12ffb |
| identifier.url.fl_str_mv | http://hdl.handle.net/10362/190384 |
| instacron_str | unl |
| institution | Universidade Nova de Lisboa |
| instname_str | Universidade Nova de Lisboa |
| language | eng |
| network_acronym_str | run |
| network_name_str | Repositório Institucional da UNL |
| oai_identifier_str | oai:run.unl.pt:10362/190384 |
| organization_str_mv | urn:organizationAcronym:unl |
| person_str_mv | Silva, Catarina Costa Pereira Nascimento da |
| publishDate | 2025 |
| reponame_str | Repositório Institucional da UNL |
| repository_id_str | urn:repositoryAcronym:run |
| service_str_mv | urn:repositoryAcronym:run |
| spelling | engpt_PTAccurate lung cancer diagnosis and staging are critical for personalized treatment and improved patient outcomes. Traditional diagnostic methods, such as manual image examination, are prone to variability and inefficiency. Recent advancements in Deep Learning (DL) have demonstrated potential in automating lung cancer classification and staging, thereby enhancing diagnostic accuracy and efficiency. However, existing solutions often address only isolated aspects of the diagnostic process, such as tumor detection, without offering a unified system for multi-target classification that integrates clinical tools for both clinicians and patients. This study presents a unified framework for lung cancer analysis that combines medical imaging, clinical data, and Large Language Models (LLMs) to support three key tasks: tumor type classification, TNM staging, and automated treatment protocol recommendation. Image-based classification was performed using YOLOv8n, trained on two CT datasets, achieving a maximum mean average precision (mAP50) of 0.418 and an F1 score of 0.44. TNM staging was addressed through a multimodal classifier, combining the ResNet50 model with a multilayer perceptron, which fused imaging and demographic inputs. This approach yielded an average F1 score of 0.389, with the M component showing the strongest performance. For treatment generation, a Retrieval-Augmented Generation (RAG) approach was employed, combining clinical prompts with relevant documents to produce personalized protocols using the Gemini 2.0 Flash LLM. The best-performing configuration achieved a stage match of 0.54 and a BERTScore of 0.83, along with high contextual fidelity across RAGAS metrics (Faithfulness: 0.68, Answer Relevancy: 0.81, Context Precision: 0.99, and Context Recall: 0.93). This framework demonstrates the potential to support both diagnosis and treatment planning within a single integrated system, contributing to more personalized and effective clinical decision-making in oncology.application/pdfpt_PTMultimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol GenerationSilva, Catarina Costa Pereira Nascimento daCastelli, MauroJardim, João Bruno Morais de SousaHostingInstitutionOrganizationalRUNe-mailmailto:run@unl.ptrun@unl.ptURNurn:tid:2040730222025-11-10T10:44:23Z2025-10-282025-10-28T00:00:00ZHandlehttp://hdl.handle.net/10362/190384http://purl.org/coar/access_right/c_abf2open accessDeep Learning (DL)Large Language Models (LLMs)Lung CancerRetrieval-Augmented Generation (RAG)Treatment Recommendation7581445 bytesliteraturehttp://purl.org/coar/resource_type/c_bdccmaster thesis2025-10-28http://creativecommons.org/licenses/by/4.0/http://purl.org/coar/access_right/c_abf2application/pdffulltexthttps://run.unl.pt/bitstreams/73dc5028-a35e-4245-90bf-6ada1d07e0d8/download |
| spellingShingle | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation Silva, Catarina Costa Pereira Nascimento da Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| status | SINGLETON |
| subject.fl_str_mv | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| title | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| title_full | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| title_fullStr | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| title_full_unstemmed | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| title_short | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| title_sort | Multimodal Learning for Lung Cancer Diagnosis and Management: A Deep Learning Pipeline for Classification, TNM Staging, and Treatment Protocol Generation |
| topic | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| topic_facet | Deep Learning (DL) Large Language Models (LLMs) Lung Cancer Retrieval-Augmented Generation (RAG) Treatment Recommendation |
| url | http://hdl.handle.net/10362/190384 |
| visible | 1 |