Publicação
Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments
| Resumo: | Pre-Trained Models (PTMs) or Large Language Models (LLMs) are deep neural networks trained on vast amounts of text data, enabling them to make predictions based on learned knowledge. Google has played a significant role in this field, particularly through popularizing the Transformers Architecture. However, the landscape evolved dramatically with the release of ChatGPT by OpenAI in November 2022, marking the advent of the universal artificial intelligence era. This event sparked significant interest and efforts in studying LLMs, prompting industries to adapt their operations, software providers to refine their skills, and society to contemplate ethical implications. This research delves into the use of Open-Source LLMs, focusing particularly on text classification — a critical task in Natural Language Processing (NLP). The study employs techniques such as fine-tuning and model quantization, which are essential for leveraging LLMs effectively in practical applications. Key questions addressed include evaluating the comparability of open-source models with established benchmarks across different text classification approaches. The research aims to identify primary challenges and limitations associated with running modern open-source LLMs in lowresource environments. By exploring these topics, the research aims to contribute insights into optimizing the deployment of open-source LLMs, enhancing their accessibility, and addressing practical constraints that affect their widespread adoption across various sectors and applications in NLP. |
|---|---|
| Autores principais: | Figueiredo, Fernando Niglio de |
| Assunto: | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| Ano: | 2024 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| _version_ | 1865920617045295104 |
|---|---|
| author | Figueiredo, Fernando Niglio de |
| author_facet | Figueiredo, Fernando Niglio de Figueiredo, Fernando Niglio de |
| author_role | author |
| contributor_name_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| country_str | PT |
| creators_json_str | [{\"Person.name\":\"Figueiredo, Fernando Niglio de\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| datacite.creators.creator.creatorName.fl_str_mv | Figueiredo, Fernando Niglio de |
| datacite.date.Accepted.fl_str_mv | 2024-10-24T00:00:00Z |
| datacite.date.available.fl_str_mv | 2024-11-26T15:26:08Z |
| datacite.date.embargoed.fl_str_mv | 2024-11-26T15:26:08Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| datacite.subjects.subject.fl_str_mv | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| datacite.titles.title.fl_str_mv | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| dc.contributor.none.fl_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| dc.creator.none.fl_str_mv | Figueiredo, Fernando Niglio de |
| dc.date.Accepted.fl_str_mv | 2024-10-24T00:00:00Z |
| dc.date.available.fl_str_mv | 2024-11-26T15:26:08Z |
| dc.date.embargoed.fl_str_mv | 2024-11-26T15:26:08Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | http://hdl.handle.net/10362/175829 |
| dc.language.none.fl_str_mv | eng |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| dc.subject.none.fl_str_mv | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| dc.title.fl_str_mv | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_bdcc |
| description | Pre-Trained Models (PTMs) or Large Language Models (LLMs) are deep neural networks trained on vast amounts of text data, enabling them to make predictions based on learned knowledge. Google has played a significant role in this field, particularly through popularizing the Transformers Architecture. However, the landscape evolved dramatically with the release of ChatGPT by OpenAI in November 2022, marking the advent of the universal artificial intelligence era. This event sparked significant interest and efforts in studying LLMs, prompting industries to adapt their operations, software providers to refine their skills, and society to contemplate ethical implications. This research delves into the use of Open-Source LLMs, focusing particularly on text classification — a critical task in Natural Language Processing (NLP). The study employs techniques such as fine-tuning and model quantization, which are essential for leveraging LLMs effectively in practical applications. Key questions addressed include evaluating the comparability of open-source models with established benchmarks across different text classification approaches. The research aims to identify primary challenges and limitations associated with running modern open-source LLMs in lowresource environments. By exploring these topics, the research aims to contribute insights into optimizing the deployment of open-source LLMs, enhancing their accessibility, and addressing practical constraints that affect their widespread adoption across various sectors and applications in NLP. |
| dirty | 0 |
| eu_rights_str_mv | openAccess |
| format | masterThesis |
| fulltext.url.fl_str_mv | https://run.unl.pt/bitstreams/f8b0e29f-5432-4d89-99a1-0de41082c908/download |
| id | run_7b14e00a5107a54ca6c569dbfcd00dbf |
| identifier.url.fl_str_mv | http://hdl.handle.net/10362/175829 |
| instacron_str | unl |
| institution | Universidade Nova de Lisboa |
| instname_str | Universidade Nova de Lisboa |
| language | eng |
| network_acronym_str | run |
| network_name_str | Repositório Institucional da UNL |
| oai_identifier_str | oai:run.unl.pt:10362/175829 |
| organization_str_mv | urn:organizationAcronym:unl |
| person_str_mv | Figueiredo, Fernando Niglio de |
| publishDate | 2024 |
| reponame_str | Repositório Institucional da UNL |
| repository_id_str | urn:repositoryAcronym:run |
| service_str_mv | urn:repositoryAcronym:run |
| spelling | engpt_PTPre-Trained Models (PTMs) or Large Language Models (LLMs) are deep neural networks trained on vast amounts of text data, enabling them to make predictions based on learned knowledge. Google has played a significant role in this field, particularly through popularizing the Transformers Architecture. However, the landscape evolved dramatically with the release of ChatGPT by OpenAI in November 2022, marking the advent of the universal artificial intelligence era. This event sparked significant interest and efforts in studying LLMs, prompting industries to adapt their operations, software providers to refine their skills, and society to contemplate ethical implications. This research delves into the use of Open-Source LLMs, focusing particularly on text classification — a critical task in Natural Language Processing (NLP). The study employs techniques such as fine-tuning and model quantization, which are essential for leveraging LLMs effectively in practical applications. Key questions addressed include evaluating the comparability of open-source models with established benchmarks across different text classification approaches. The research aims to identify primary challenges and limitations associated with running modern open-source LLMs in lowresource environments. By exploring these topics, the research aims to contribute insights into optimizing the deployment of open-source LLMs, enhancing their accessibility, and addressing practical constraints that affect their widespread adoption across various sectors and applications in NLP.application/pdfpt_PTOpen-Source Language Models for News Classification: Implementing Small Models in Low-Resource EnvironmentsFigueiredo, Fernando Niglio deBação, Fernando José Ferreira LucasHostingInstitutionOrganizationalRUNe-mailmailto:run@unl.ptrun@unl.ptURNurn:tid:2037851422024-11-26T15:26:08Z2024-10-242024-10-24T00:00:00ZHandlehttp://hdl.handle.net/10362/175829http://purl.org/coar/access_right/c_abf2open accessLarge Language Models (LLM)Pre-trained models (PTM)Text classificationNews classificationFine-tuningOpen SourceLoRAQLoRAlow-resource environmentSDG 8 - Decent work and economic growth1432890 bytesliteraturehttp://purl.org/coar/resource_type/c_bdccmaster thesis2024-10-24http://creativecommons.org/licenses/by/4.0/http://purl.org/coar/access_right/c_abf2application/pdffulltexthttps://run.unl.pt/bitstreams/f8b0e29f-5432-4d89-99a1-0de41082c908/download |
| spellingShingle | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments Figueiredo, Fernando Niglio de Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth Figueiredo, Fernando Niglio de Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| status | NEW |
| subject.fl_str_mv | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| title | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| title_full | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| title_fullStr | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| title_full_unstemmed | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| title_short | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| title_sort | Open-Source Language Models for News Classification: Implementing Small Models in Low-Resource Environments |
| topic | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| topic_facet | Large Language Models (LLM) Pre-trained models (PTM) Text classification News classification Fine-tuning Open Source LoRA QLoRA low-resource environment SDG 8 - Decent work and economic growth |
| url | http://hdl.handle.net/10362/175829 |
| visible | 1 |