Publicação
ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech
| Resumo: | Automatic Speech Recognition (ASR) systems struggle with speech disfluencies like repetitions and filled pauses, which are common in spontaneous speech and disorders such as stuttering. This performance gap limits the accessibility and clinical utility of ASR technology. This research proposes a comprehensive Deep Learning framework to improve the transcription, classification, and interpretation of disfluent speech. The multi-stage methodology begins by fine-tuning state-of-the-art ASR models on the FluencyBank corpus to accurately transcribe disfluent events. Next, a custom-trained ModernBERT model performs token-level classification to identify and label specific disfluencies in the transcript. Finally, a specialized Large Language Model (LLM) provides a structured, context-aware analysis of the identified patterns. The integrated pipeline is demonstrated through a functional prototype, ArtiFlow (Articulate Flow), which showcases the system's end-to-end capabilities. Empirical results validate the framework's effectiveness. After a comparative analysis of different ASR model families, the Whisper Large v3 Turbo model was selected, offering an optimal balance between high transcription accuracy (12.94% Word Error Rate) and enhanced inference speed, while leveraging a stable and accessible implementation framework. The subsequent disfluency classifier performs with high reliability (weighted F1-score of 0.9512) and the selected LLM demonstrates strong capabilities in generating structured analyses. This thesis establishes a robust, modular system for both identifying and interpreting speech disfluencies, providing a foundation for advanced support tools. The work contributes to speech processing and assistive technology, offering significant potential for applications in clinical diagnosis, speech therapy, and the development of more inclusive human-computer interaction systems. |
|---|---|
| Autores principais: | Pérez, Ariel Enrique Cerda |
| Assunto: | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso embargado |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| _version_ | 1868982710970613760 |
|---|---|
| author | Pérez, Ariel Enrique Cerda |
| author_facet | Pérez, Ariel Enrique Cerda |
| author_role | author |
| contributor_name_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| country_str | PT |
| creators_json_txt | [{\"Person.name\":\"Pérez, Ariel Enrique Cerda\"}] |
| datacite.contributors.contributor.contributorName.fl_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| datacite.creators.creator.creatorName.fl_str_mv | Pérez, Ariel Enrique Cerda |
| datacite.date.Accepted.fl_str_mv | 2025-10-28T00:00:00Z |
| datacite.date.available.fl_str_mv | 2025-11-07T15:55:19Z |
| datacite.date.embargoed.fl_str_mv | 2025-11-07T15:55:19Z |
| datacite.rights.fl_str_mv | http://purl.org/coar/access_right/c_f1cf |
| datacite.subjects.subject.fl_str_mv | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| datacite.titles.title.fl_str_mv | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| dc.contributor.none.fl_str_mv | Bação, Fernando José Ferreira Lucas RUN |
| dc.creator.none.fl_str_mv | Pérez, Ariel Enrique Cerda |
| dc.date.Accepted.fl_str_mv | 2025-10-28T00:00:00Z |
| dc.date.available.fl_str_mv | 2025-11-07T15:55:19Z |
| dc.date.embargoed.fl_str_mv | 2025-11-07T15:55:19Z |
| dc.format.none.fl_str_mv | application/pdf |
| dc.identifier.none.fl_str_mv | http://hdl.handle.net/10362/190288 |
| dc.language.none.fl_str_mv | eng |
| dc.rights.cclincense.fl_str_mv | http://creativecommons.org/licenses/by/4.0/ |
| dc.rights.none.fl_str_mv | http://purl.org/coar/access_right/c_f1cf |
| dc.subject.none.fl_str_mv | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| dc.title.fl_str_mv | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| dc.type.none.fl_str_mv | http://purl.org/coar/resource_type/c_bdcc |
| description | Automatic Speech Recognition (ASR) systems struggle with speech disfluencies like repetitions and filled pauses, which are common in spontaneous speech and disorders such as stuttering. This performance gap limits the accessibility and clinical utility of ASR technology. This research proposes a comprehensive Deep Learning framework to improve the transcription, classification, and interpretation of disfluent speech. The multi-stage methodology begins by fine-tuning state-of-the-art ASR models on the FluencyBank corpus to accurately transcribe disfluent events. Next, a custom-trained ModernBERT model performs token-level classification to identify and label specific disfluencies in the transcript. Finally, a specialized Large Language Model (LLM) provides a structured, context-aware analysis of the identified patterns. The integrated pipeline is demonstrated through a functional prototype, ArtiFlow (Articulate Flow), which showcases the system's end-to-end capabilities. Empirical results validate the framework's effectiveness. After a comparative analysis of different ASR model families, the Whisper Large v3 Turbo model was selected, offering an optimal balance between high transcription accuracy (12.94% Word Error Rate) and enhanced inference speed, while leveraging a stable and accessible implementation framework. The subsequent disfluency classifier performs with high reliability (weighted F1-score of 0.9512) and the selected LLM demonstrates strong capabilities in generating structured analyses. This thesis establishes a robust, modular system for both identifying and interpreting speech disfluencies, providing a foundation for advanced support tools. The work contributes to speech processing and assistive technology, offering significant potential for applications in clinical diagnosis, speech therapy, and the development of more inclusive human-computer interaction systems. |
| dirty | 0 |
| eu_rights_str_mv | embargoedAccess |
| format | masterThesis |
| fulltext.url.fl_str_mv | https://run.unl.pt/bitstreams/cd1628a4-c00b-4f8f-928a-8805b90129aa/download |
| id | run_a8db9f919f5e4e3d3d9b00a47ad691f0 |
| identifier.url.fl_str_mv | http://hdl.handle.net/10362/190288 |
| inst_facet_str | urn:organizationAcronym:unl{{{_:::_}}}Universidade Nova de Lisboa |
| instacron_str | unl |
| institution | Universidade Nova de Lisboa |
| instname_str | Universidade Nova de Lisboa |
| language | eng |
| network_acronym_str | run |
| network_name_str | Repositório Institucional da UNL |
| oai_identifier_str | oai:run.unl.pt:10362/190288 |
| organization_str_mv | urn:organizationAcronym:unl |
| person_str_mv | Pérez, Ariel Enrique Cerda |
| publishDate | 2025 |
| repo_facet_str | urn:repositoryAcronym:run{{{_:::_}}}Repositório Institucional da UNL |
| reponame_str | Repositório Institucional da UNL |
| repository_id_str | urn:repositoryAcronym:run |
| service_str_mv | urn:repositoryAcronym:run |
| spelling | engpt_PTAutomatic Speech Recognition (ASR) systems struggle with speech disfluencies like repetitions and filled pauses, which are common in spontaneous speech and disorders such as stuttering. This performance gap limits the accessibility and clinical utility of ASR technology. This research proposes a comprehensive Deep Learning framework to improve the transcription, classification, and interpretation of disfluent speech. The multi-stage methodology begins by fine-tuning state-of-the-art ASR models on the FluencyBank corpus to accurately transcribe disfluent events. Next, a custom-trained ModernBERT model performs token-level classification to identify and label specific disfluencies in the transcript. Finally, a specialized Large Language Model (LLM) provides a structured, context-aware analysis of the identified patterns. The integrated pipeline is demonstrated through a functional prototype, ArtiFlow (Articulate Flow), which showcases the system's end-to-end capabilities. Empirical results validate the framework's effectiveness. After a comparative analysis of different ASR model families, the Whisper Large v3 Turbo model was selected, offering an optimal balance between high transcription accuracy (12.94% Word Error Rate) and enhanced inference speed, while leveraging a stable and accessible implementation framework. The subsequent disfluency classifier performs with high reliability (weighted F1-score of 0.9512) and the selected LLM demonstrates strong capabilities in generating structured analyses. This thesis establishes a robust, modular system for both identifying and interpreting speech disfluencies, providing a foundation for advanced support tools. The work contributes to speech processing and assistive technology, offering significant potential for applications in clinical diagnosis, speech therapy, and the development of more inclusive human-computer interaction systems.application/pdfpt_PTArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent SpeechPérez, Ariel Enrique CerdaBação, Fernando José Ferreira LucasHostingInstitutionOrganizationalRUNe-mailmailto:run@unl.ptrun@unl.ptURNurn:tid:2040714102025-11-07T15:55:19Z2025-10-282025-10-28T00:00:00ZHandlehttp://hdl.handle.net/10362/190288http://purl.org/coar/access_right/c_f1cfembargoed accessAutomatic Speech RecognitionSpeech DisfluenciesLarge Language ModelsSpoken Language ProcessingStutteringSDG 3 - Good health and well-being2659439 bytesliteraturehttp://purl.org/coar/resource_type/c_bdccmaster thesis2025-10-28http://creativecommons.org/licenses/by/4.0/http://purl.org/coar/access_right/c_f1cfapplication/pdffulltexthttps://run.unl.pt/bitstreams/cd1628a4-c00b-4f8f-928a-8805b90129aa/download |
| spellingShingle | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech Pérez, Ariel Enrique Cerda Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| status | SINGLETON |
| subject.fl_str_mv | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| title | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| title_full | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| title_fullStr | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| title_full_unstemmed | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| title_short | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| title_sort | ArtiFlow: An Integrated Pipeline for Identifying and Analyzing Speech Disfluencies: A Deep Learning Framework for Enhanced Transcription, Classification and Interpretation of Disfluent Speech |
| topic | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| topic_facet | Automatic Speech Recognition Speech Disfluencies Large Language Models Spoken Language Processing Stuttering SDG 3 - Good health and well-being |
| url | http://hdl.handle.net/10362/190288 |
| visible | 1 |