Ibraheem Wagih · Fraud & Financial Crime → Analytics
Projects / Predictive Modelling

Hotel Cancellation Prediction

Individual Predictive Analytics (MSIN0097) UCL MSc Business Analytics

A classification pipeline predicting whether a hotel booking will be cancelled, so revenue managers can target retention or set defensible overbooking thresholds. The same discipline as fraud model review: imbalanced classes, a cost on both error types, and a threshold that has to be justified, not defaulted.

37%
of bookings cancelled (class imbalance)
99.4%
cancellation rate on non-refundable deposits
F1
target metric, chosen over accuracy
94%
missing in the sparsest field, handled explicitly

Problem

Accuracy was rejected as the success metric. With a 37/63 split, a model that predicts "not cancelled" every time scores 63% while catching nothing. F1 forces a real trade-off between catching cancellations (recall) and not flagging bookings that were always going to happen (precision), the same tension I worked inside daily when adjudicating fraud-model alerts at Monzo.

What the Data Showed

Exploratory analysis found the strongest signals before any model was trained: non-refundable deposits cancel at 99.4%, and lead time plus market segment (Groups, Online Travel Agents) drive the rest. A Spearman heatmap ruled out multicollinearity, and splits were reproducible with no information from the outcome period bleeding into training.

Cancellation rate by deposit type
Cancellation rate by deposit type. Non-refundable bookings cancel at 99.4%, the single strongest signal in the data.
Confusion matrix of the final model
Final model confusion matrix on the held-out test set: 14,101 true negatives, 7,175 true positives.

Model and Explanation

Multiple classifiers were compared on engineered features, tuned, and threshold-calibrated against F1 rather than left at the 0.5 default. SHAP values then confirmed the model's drivers matched the exploratory findings, deposit type, lead time and country, rather than leaning on leaked or spurious features. An AI-agent coding workflow scaffolded the pipeline, with explicit checks on leakage and seed reproducibility rather than trusting generated code blindly.

SHAP summary plot of booking cancellation drivers
SHAP summary: the model's decisions are driven by deposit type, lead time, special requests and country, consistent with the EDA rather than artefacts.

Outcome

The final model turns a cancellation-probability score into two decisions: which bookings to target with retention outreach, and how far a property can safely overbook given its booking mix. The report documents limitations and when the threshold should be recalibrated.