Publicação
Predicting Non-Contractual B2C Churn on Small and Sparse Data: An Ensemble-based Framework for Retention-driven Threshold Optimization and Model Robustness
| Resumo: | Churn Prediction Modelling has long been researched and put into practice in subscriptionbased industries in an attempt to study, identify and anticipate Customer behaviour in these settings. Under non-contractual B2C Businesses, however, it has been understudied. This is a consequence of both Business and Data Mining factors such as unclear Churn definitions, irregular visit and purchase patterns, and, in some cases, small and sparse data – which limits feature richness. This study aimed to investigate if, even under such circumstances, Machine Learning Models could accurately predict whether or not a Customer would eventually Churn and if these results could generalize effectively on unseen real-world data. This, in part, was done through the tweaking of Model Hyperparameters and decision thresholds – with the latter being tested against the default 0.5 cutoff that is widely employed in Literature and practice, to assess whether or not a Business-driven decision threshold is optimal under the aforementioned circumstances. The results of this research underscore the possibility of competent Predictive Modelling when working with data that is limited in nature. By constraining tree depth, iteration count, and subsampling to reduce variance, both the Histogram-based Gradient Boosting and LightGBM ensembles employed in this study achieved mean cross-validated ROC AUCs above 0.72 on a 1,462-row dataset, demonstrating reliable discrimination even with inherently noisy and sparse inputs. Furthermore, at the F₁ Scoreoptimized threshold (t = 0.63) for the HGB Model, each 20$ outreach expects a 74.88$ in recovered CLV, compared to the 0.5 default cutoff, which yields a Precision score of 0.770609 and, consequently, an ROI per Outreach of 70.59$. Finally, with careful hyperparameter tuning, early stopping, and repeated stratified cross-validation, the best performing Models exhibited train-to-CV AUC gaps under 0.02 and CV standard deviations below 0.05, indicating stable generalization despite the small sample size. |
|---|---|
| Autores principais: | Folgado, Alexandre Cruz Fernandes Fonseca |
| Assunto: | Churn Prediction non-contractual B2C Customer Retention Overfitting Decision Threshold Small Data SDG 8 - Decent work and economic growth |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso embargado |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| Resumo: | Churn Prediction Modelling has long been researched and put into practice in subscriptionbased industries in an attempt to study, identify and anticipate Customer behaviour in these settings. Under non-contractual B2C Businesses, however, it has been understudied. This is a consequence of both Business and Data Mining factors such as unclear Churn definitions, irregular visit and purchase patterns, and, in some cases, small and sparse data – which limits feature richness. This study aimed to investigate if, even under such circumstances, Machine Learning Models could accurately predict whether or not a Customer would eventually Churn and if these results could generalize effectively on unseen real-world data. This, in part, was done through the tweaking of Model Hyperparameters and decision thresholds – with the latter being tested against the default 0.5 cutoff that is widely employed in Literature and practice, to assess whether or not a Business-driven decision threshold is optimal under the aforementioned circumstances. The results of this research underscore the possibility of competent Predictive Modelling when working with data that is limited in nature. By constraining tree depth, iteration count, and subsampling to reduce variance, both the Histogram-based Gradient Boosting and LightGBM ensembles employed in this study achieved mean cross-validated ROC AUCs above 0.72 on a 1,462-row dataset, demonstrating reliable discrimination even with inherently noisy and sparse inputs. Furthermore, at the F₁ Scoreoptimized threshold (t = 0.63) for the HGB Model, each 20$ outreach expects a 74.88$ in recovered CLV, compared to the 0.5 default cutoff, which yields a Precision score of 0.770609 and, consequently, an ROI per Outreach of 70.59$. Finally, with careful hyperparameter tuning, early stopping, and repeated stratified cross-validation, the best performing Models exhibited train-to-CV AUC gaps under 0.02 and CV standard deviations below 0.05, indicating stable generalization despite the small sample size. |
|---|