Publicação
Automation of machine learning pipelines for anomaly detection challenges
| Resumo: | Machine Learning (ML) and Data Science can solve different real-world problems. Businesses are becoming increasingly interested in these approaches, and as technology evolves, new challenges can be identified, mostly regarding the ML models development, deployment cycle and data cleansing, which can significantly decrease the accuracy and viability of ML software systems. Development and Operations (DevOps) practices have become popular in operating software systems at scale successfully, but they need to be adapted to deliver the best results when applied to ML systems. This led to the emergence of Machine Learning and Operations (MLOps), a development culture specific for ML systems, derived from DevOps principles. What MLOps attempts to address is the unification of the development cycle of ML based software systems while striving for automation and monitoring, in order to allow continuous integration and delivery. With this thesis, the goal is to study different available frameworks and methods for ML systems, in order to develop an automated ML pipeline to ingest and manipulate high volumes of data. A sensorial system, which simulates the interior of a vehicle, gathers enough data to feed the pipeline. Alongside the development of the ML system, a visual interface which allows control over the overall system and its data is created. |
|---|---|
| Autores principais: | Martins, Ricardo Rodrigues |
| Assunto: | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| Ano: | 2023 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade do Minho |
| Idioma: | inglês |
| Origem: | RepositóriUM - Universidade do Minho |
| _version_ | 1866875729479729152 |
|---|---|
| author | Martins, Ricardo Rodrigues |
| author_facet | Martins, Ricardo Rodrigues |
| author_role | author |
| contributor_name_str_mv | Fernandes, João M. Ferreira, André Leite Universidade do Minho |
| country_str | PT |
| creators_json_txt | [{\"Person.name\":\"Martins, Ricardo Rodrigues\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | Fernandes, João M. Ferreira, André Leite Universidade do Minho |
| datacite.creators.creator.creatorName.fl_str_mv | Martins, Ricardo Rodrigues |
| datacite.date.Accepted.fl_str_mv | 2023-12-11T00:00:00Z |
| datacite.date.available.fl_str_mv | 2024-07-30T10:06:14Z |
| datacite.date.embargoed.fl_str_mv | 2024-07-30T10:06:14Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| datacite.subjects.subject.fl_str_mv | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| datacite.titles.title.fl_str_mv | Automation of machine learning pipelines for anomaly detection challenges |
| dc.contributor.none.fl_str_mv | Fernandes, João M. Ferreira, André Leite Universidade do Minho |
| dc.creator.none.fl_str_mv | Martins, Ricardo Rodrigues |
| dc.date.Accepted.fl_str_mv | 2023-12-11T00:00:00Z |
| dc.date.available.fl_str_mv | 2024-07-30T10:06:14Z |
| dc.date.embargoed.fl_str_mv | 2024-07-30T10:06:14Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | https://hdl.handle.net/1822/92661 |
| dc.language.none.fl_str_mv | eng |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by-nc-sa/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_abf2 |
| dc.rights.rights.copyright.fl_str_mv | openAccess |
| dc.subject.none.fl_str_mv | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| dc.title.fl_str_mv | Automation of machine learning pipelines for anomaly detection challenges |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_bdcc |
| description | Machine Learning (ML) and Data Science can solve different real-world problems. Businesses are becoming increasingly interested in these approaches, and as technology evolves, new challenges can be identified, mostly regarding the ML models development, deployment cycle and data cleansing, which can significantly decrease the accuracy and viability of ML software systems. Development and Operations (DevOps) practices have become popular in operating software systems at scale successfully, but they need to be adapted to deliver the best results when applied to ML systems. This led to the emergence of Machine Learning and Operations (MLOps), a development culture specific for ML systems, derived from DevOps principles. What MLOps attempts to address is the unification of the development cycle of ML based software systems while striving for automation and monitoring, in order to allow continuous integration and delivery. With this thesis, the goal is to study different available frameworks and methods for ML systems, in order to develop an automated ML pipeline to ingest and manipulate high volumes of data. A sensorial system, which simulates the interior of a vehicle, gathers enough data to feed the pipeline. Alongside the development of the ML system, a visual interface which allows control over the overall system and its data is created. |
| dirty | 0 |
| eu_rights_str_mv | openAccess |
| format | masterThesis |
| fulltext.url.fl_str_mv | https://prod-dspace.uminho.pt/bitstreams/181ff798-713e-46e7-8e0b-8afc352202ee/download |
| id | rum_3eea8ccef313e2a47bbc33b70a67e3f4 |
| identifier.url.fl_str_mv | https://hdl.handle.net/1822/92661 |
| instacron_str | repositorium |
| institution | Universidade do Minho |
| instname_str | Universidade do Minho |
| language | eng |
| network_acronym_str | rum |
| network_name_str | RepositóriUM - Universidade do Minho |
| oai_identifier_str | oai:repositorium.uminho.pt:1822/92661 |
| organization_str_mv | urn:organizationAcronym:repositorium |
| person_str_mv | Martins, Ricardo Rodrigues |
| publishDate | 2023 |
| reponame_str | RepositóriUM - Universidade do Minho |
| repository_id_str | urn:repositoryAcronym:rum |
| service_str_mv | urn:repositoryAcronym:rum |
| spelling | engporMachine Learning (ML) and Data Science can solve different real-world problems. Businesses are becoming increasingly interested in these approaches, and as technology evolves, new challenges can be identified, mostly regarding the ML models development, deployment cycle and data cleansing, which can significantly decrease the accuracy and viability of ML software systems. Development and Operations (DevOps) practices have become popular in operating software systems at scale successfully, but they need to be adapted to deliver the best results when applied to ML systems. This led to the emergence of Machine Learning and Operations (MLOps), a development culture specific for ML systems, derived from DevOps principles. What MLOps attempts to address is the unification of the development cycle of ML based software systems while striving for automation and monitoring, in order to allow continuous integration and delivery. With this thesis, the goal is to study different available frameworks and methods for ML systems, in order to develop an automated ML pipeline to ingest and manipulate high volumes of data. A sensorial system, which simulates the interior of a vehicle, gathers enough data to feed the pipeline. Alongside the development of the ML system, a visual interface which allows control over the overall system and its data is created.application/pdfporAutomation of machine learning pipelines for anomaly detection challengesMartins, Ricardo RodriguesFernandes, João M.Ferreira, André LeiteHostingInstitutionOrganizationalUniversidade do Minhoe-mailmailto:repositorium@usdb.uminho.ptrepositorium@usdb.uminho.ptURNurn:tid:2036107172024-07-30T10:06:14Z2023-12-112023-122023-12-11T00:00:00ZHandlehttps://hdl.handle.net/1822/92661http://purl.org/coar/access_right/c_abf2open accessSoftware engineeringMachine learningMLOpsModelAutomationEngenharia de softwareAprendizagem automáticaModeloAutomação941187 bytesliteraturehttp://purl.org/coar/resource_type/c_bdccmaster thesis2023-12-11http://creativecommons.org/licenses/by-nc-sa/4.0/openAccesshttp://purl.org/coar/access_right/c_abf2application/pdffulltexthttps://prod-dspace.uminho.pt/bitstreams/181ff798-713e-46e7-8e0b-8afc352202ee/download |
| spellingShingle | Automation of machine learning pipelines for anomaly detection challenges Martins, Ricardo Rodrigues Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| status | SINGLETON |
| subject.fl_str_mv | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| title | Automation of machine learning pipelines for anomaly detection challenges |
| title_full | Automation of machine learning pipelines for anomaly detection challenges |
| title_fullStr | Automation of machine learning pipelines for anomaly detection challenges |
| title_full_unstemmed | Automation of machine learning pipelines for anomaly detection challenges |
| title_short | Automation of machine learning pipelines for anomaly detection challenges |
| title_sort | Automation of machine learning pipelines for anomaly detection challenges |
| topic | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| topic_facet | Software engineering Machine learning MLOps Model Automation Engenharia de software Aprendizagem automática Modelo Automação |
| url | https://hdl.handle.net/1822/92661 |
| visible | 1 |