Publicação
An ETL pattern for log configuration and analysis
| Resumo: | In many scenarios, such as the ones related to Data Warehousing Extract-Transform-Load (ETL) processes, logging techniques are usually applied for capturing event metrics across system levels for system auditing and system recovery. The diversity of strategies and architectures of the toolset used to support the ETL implementation introduces another layer of complexity, both for system development and audit. Although a valuable system diagnosis resource for the development team, logging is generally underestimated, being used only when the system reveals unexpected behaviours and not to drive the ETL system evolution. We believe that the use of logs for steering ETL development and maintenance can improve significantly global system quality. However, this approach is only effective if flexible and efficient logging systems exist. In this paper, we describe a Log Pattern used in a pattern-oriented approach for ETL systems development, which provides a configurable and flexible component for using to drive ETL development and maintenance phases. |
|---|---|
| Autores principais: | Oliveira, Bruno |
| Outros Autores: | Oliveira, Óscar; Matos, Telmo; Santos, Vasco; Belo, Orlando |
| Assunto: | Data Warehousing ETL Patterns ETL Systems Graph Databases Log Pattern Logging Process Mining |
| Ano: | 2019 |
| País: | Portugal |
| Tipo de documento: | comunicação em conferência |
| Tipo de acesso: | acesso restrito |
| Instituição associada: | Universidade do Minho |
| Idioma: | inglês |
| Origem: | RepositóriUM - Universidade do Minho |
| Resumo: | In many scenarios, such as the ones related to Data Warehousing Extract-Transform-Load (ETL) processes, logging techniques are usually applied for capturing event metrics across system levels for system auditing and system recovery. The diversity of strategies and architectures of the toolset used to support the ETL implementation introduces another layer of complexity, both for system development and audit. Although a valuable system diagnosis resource for the development team, logging is generally underestimated, being used only when the system reveals unexpected behaviours and not to drive the ETL system evolution. We believe that the use of logs for steering ETL development and maintenance can improve significantly global system quality. However, this approach is only effective if flexible and efficient logging systems exist. In this paper, we describe a Log Pattern used in a pattern-oriented approach for ETL systems development, which provides a configurable and flexible component for using to drive ETL development and maintenance phases. |
|---|