Prove It,
Don't Promise It
Every payment-recovery vendor claims “we recovered X% more.” This is the protocol for telling real, incremental lift from an artefact of the setup, the same standard Slicker commits to being evaluated by.
- The 12-step evidence-driven lift checklist
- The AABB design and its two self-checks, with figures
- How to size, read, and decide with confidence intervals
- The pitfalls this design avoids, each with its antidote
Get the playbook
Read exactly how we prove ~20% relative lift is real, not an artefact of the setup. Enter your work email to download.
By downloading, you agree to our Privacy Policy. No spam, unsubscribe anytime.
A number is not evidence by default
A different customer mix, a favourable date window, a metric quietly redefined, or a rollout that favoured easy cases can all create apparent lift without the method doing anything. Before you accept a lift claim, name the ways it can be wrong.
“How do I know the measured lift is real, and not an artefact of the setup?”
The one question this paper answers.
Confounding
The two groups differ in composition, not just treatment. Hand one arm easier invoices and it outperforms for reasons that have nothing to do with the method.
Selection bias
Non-random assignment changes the comparison set. Letting the system opt in the cases it is likely to recover is the classic example.
Regression to the mean
Retry anything after a failure spike and it will appear to improve, because the spike was partly noise that was always going to subside.
Cherry-picked windows
A favourable fortnight can overstate performance. A test that spans one convenient calendar window inherits that window's luck.
Simpson's paradox
An aggregate lift can reverse inside every segment. The most counter-intuitive failure, and one that survives a casual review.
None of these require intentional bias. Only an uncontrolled test. Every one has the same antidote: a design that controls composition and checks itself.
Why AABB, not bare A/B
Split the control side into two independent groups and the treatment side into two more. The same traffic is stratified and randomised four ways instead of two. It is the same headline comparison, made auditable.
- A1 vs A2An embedded A/A check. Two groups running identical status-quo logic should show no material difference. If they diverge, stop and investigate before trusting the headline.
- B1 vs B2A treatment-stability check. Two independent treatment groups should land in the same place up to noise, or the effect may be unstable or contaminated.
- A vs BThe actual lift. Only once the first two checks pass do we report the pre-registered primary endpoint.
Figures are from the paper's synthetic worked example, used to illustrate the method. They are not a performance guarantee.
What you'll take away
A working protocol you can hold any vendor to, including us. Written for the buyer who has to decide whether to roll out a change, keep a vendor, or extend a trial.
The 12-step evidence checklist
From defining the decision to deciding against pre-registered criteria, the full operating protocol on one page.
Why AABB beats bare A/B
Two embedded diagnostics, an A/A check and a treatment-stability check, that have to look healthy before the lift is believed.
Balanced groups by design
Covariate-adaptive randomisation that keeps the high-value tail and long tail of small markets from pooling into one arm by chance.
Sizing and power, up front
Why roughly ~1,900 failed invoices power a ~20% read, and why a 90/10 split needs about 2.8x the volume.
Reading results honestly
A visual field guide to what good and bad look like: covariate balance, the AA check, and separation versus noise.
A decision framework
Significant win, extend under a pre-committed rule, or inconclusive, with guardrails that can block a rollout even when the headline wins.
Ten questions to ask before believing a lift claim
Put these to any recovery vendor, including Slicker. If a claim cannot survive them, it is a promise, not proof.
Hold your next vendor to this standard
You've seen the method. Take the full protocol with you: a working evaluation standard you can apply to any recovery vendor, including us. 33 pages, 15 figures, no fluff.
Get the playbook
Read exactly how we prove ~20% relative lift is real, not an artefact of the setup. Enter your work email to download.
By downloading, you agree to our Privacy Policy. No spam, unsubscribe anytime.
See the protocol run on your traffic
Slicker builds this evaluation into every engagement. Control-arm baseline, balanced groups, and a headline you can reproduce from your own billing data.