How We Built a Fraud Detection Model with 99.1% Accuracy

When a US fintech reached out, their fraud loss rate sat at 2.8% of monthly transaction volume — roughly $190,000 a month leaking out the back door. The brief was simple to write and hard to deliver: build a real-time fraud detection model that drops that loss rate below 0.5%, integrates with their existing payments stack, and never blocks more than 1% of legitimate transactions.

This post walks through how we did it.

The Data Problem We started with 14 months of transaction logs — about 38 million rows. Roughly 0.6% of those were labeled fraud, which makes the problem severely imbalanced. Before touching a model we spent two weeks on the unsexy part: cleaning, joining card-on-file metadata, deriving velocity features (transactions per card per minute, per merchant per hour), and building a leak-proof temporal split.

If you only remember one thing from this article: **the model is downstream of the data**. We have never had a project where fixing the data wasn't the biggest unlock.

Feature Engineering Tabular fraud is still XGBoost territory. We engineered ~80 features across three buckets — behavioural (deviation from a card's normal spend profile), graph-derived (shared device/IP signals), and merchant-side (risk score by MCC × geo × time-of-day). We dropped a third of those features after permutation importance on a held-out set.

Model & Validation A single XGBoost classifier with focal loss handled the imbalance well enough that we didn't need SMOTE. Calibration with isotonic regression got the predicted probabilities honest. The validation gate was a precision/recall curve evaluated on a four-week rolling window — not on a fixed test set.

Production accuracy held at 99.1% with a recall of 92% on the fraud class. False positive rate stayed below 0.8%.

Deployment A FastAPI service in a Docker container behind their existing API gateway. Sub-50ms p99 latency at 400 requests/second. We also shipped a small React dashboard for the risk team to inspect flagged transactions and feed reviewed labels back into the next retraining cycle.

What Would We Do Differently Honestly, less feature engineering by hand. Twelve months later we've moved similar projects toward a gradient-boosting + small tabular transformer ensemble that handles a chunk of the feature interactions automatically.

If you have a fraud problem and a year of labeled data, talk to us.