Chargeback app vs doing it yourself: a test template

Every chargeback app has a story about a store that doubled its recoveries. You have read a few. None of them is about your store, your products or your customers, and you suspect the best examples were picked from a long list. So when you search for chargeback app vs doing it yourself, what you really want is not another vendor's case study. You want your own, with your numbers, written before you spend real money.

You can build one in about two months and a spreadsheet. Measure how you handle disputes now, change one thing, measure again and compare on the same yardsticks. This guide gives you a copy-ready template: a baseline sheet, a dispute log, a before-and-after scorecard, a table of fair test designs and a one-page write-up. It also includes a worked example with clearly labelled illustrative numbers, and an honest warning about what a small store can and cannot prove from a few dozen disputes.

This guide is for you if:

You handle chargebacks yourself and want evidence, not claims, before you pay for a tool.
You already use an app and want to find out whether it is really earning its fee.
You need a one-page summary to show a partner, an accountant or a boss.

Last updated: 30 October 2026. Written by the ChargeFix team. Reviewed against sources on 30 October 2026.

A note on bias before we start: we make ChargeFix, so we gain if your test favours an app. ChargeFix is a new product with no App Store reviews yet, and we do not publish customer case studies, because we do not want to invent them. This template lets you produce a real one for your own store. The worked example below uses made-up numbers that we label as illustrations, and every dollar figure is an assumption you should replace.

The 30-second answer (TL;DR)

Run a fair before-and-after test of at least 60 days, with the same measures in both periods.
Track eight numbers: disputes, wins, win rate, average order, minutes per dispute, tool cost, missed deadlines and net benefit.
Net benefit is dollars recovered, minus the value of your time, minus tool fees. Compare it per dispute, not in total, because volume changes.
Time saved is reliable even in a small sample, because you measure it directly. A change in win rate is not, unless you have hundreds of disputes.
To detect a 10 point difference in win rate with reasonable confidence, you need roughly 380 disputes in each group. Most stores will never have that many.
Decide your keep-or-cancel rule before the test starts, so you do not bend it afterwards.

Quick note: A small test can prove that a tool saved you time and cost you a known amount. It usually cannot prove that the tool raised your win rate. Plan the test around what it can prove, and be honest about the rest.

The template at a glance

Part	What it is	Where in this guide
1. Baseline sheet	Your numbers before you change anything	Below, "Part 1"
2. Dispute log	One row per dispute, in both periods	"Part 2"
3. Scorecard	Before and after side by side	"Part 3"
4. Test design	How to split the periods fairly	"Choosing a fair test design"
5. Decision rule	Keep, cancel or extend	"The decision rule"
6. One-page write-up	The case study itself	"The one-page case study"

Why not just read the vendors' case studies?

Case studies are marketing, and that is fine as long as you read them that way. Three things limit what they can tell you.

Selection. A vendor shows its best results. You rarely see the store where the tool made no difference.

Case mix. A store that sells digital goods to a loyal base has different disputes from one that sells fashion to first-time buyers across several countries. A result from one does not carry over to the other.

Shaky numbers. We saw this directly while writing this series. One vendor's comparison page, which was written to make the case for its own product, displayed case study metrics with placeholder values such as +0%, 0% and 0+. That is an extreme example, but it shows how thin the evidence behind a headline can be.

Win-rate claims have a further problem. A win rate is wins divided by disputes fought, so a vendor that fights fewer, easier cases can show a higher percentage while recovering fewer dollars. We work through a numeric example in our Chargeflow vs Disputifier guide. The honest metric is dollars kept per dispute filed, measured on your own store.

Chargeback app vs doing it yourself: what should you measure?

Eight numbers cover almost everything. Keep them simple enough that you will actually record them.

Measure	Definition	How to record it
Disputes	Number of chargebacks opened in the period	Count from the Shopify admin
Wins	Disputes decided in your favour	Count when decisions arrive
Win rate	Wins divided by decided disputes	Compute, but read with care
Average won order	Total value of won orders divided by wins	Compute
Minutes per dispute	Time from opening a case to submitting it	Use a stopwatch for every case
Tool cost	Subscription, share of wins and any extras	From invoices
Missed deadlines	Disputes that reached the due date unanswered	Count
Net benefit	Dollars recovered minus your time value minus tool cost	Compute

Two notes on decisions. First, the bank can take up to 75 days to decide after you submit, so a 60-day test will have undecided cases at the end. Count only decided disputes in win rates, and note how many are pending. Second, a win returns the order value and, in the US, the $15 Shopify fee, so we value a win at the order plus $15. If you prefer a conservative view, value it at the order alone and say so.

Part 1: your baseline sheet

Fill this in from the last 60 to 90 days, before you change anything. If your records are thin, start recording now and run a baseline period first.

Item	Your number
Period covered	from ___ to ___
Disputes opened	___
Disputes fought	___
Disputes accepted or refunded	___
Wins	___
Losses	___
Still undecided	___
Average order value in disputes	$___
Average minutes per dispute, by hand	___
Your hourly value	$___
Missed deadlines	___
Reasons, as a count (fraud, not received, credit not processed, other)	___
Shopify dispute fees paid (US: $15 each, returned on wins)	$___

The reasons row matters more than it looks. If your baseline is mostly fraud claims and your test period is mostly item not received, the comparison is unfair from the start. Our reason code guides explain the main categories.

Part 2: your dispute log

One row per dispute, in both periods. It takes under a minute to fill in, and it is the raw material for everything else.

Date opened	Order value	Reason	Fought, accepted or refunded	Minutes spent	Submitted by (you, tool, other)	Due date	Submitted on	Outcome	Tool fee on this case
___	$___	___	___	___	___	___	___	___	$___

Here is how three rows might look, with made-up values for illustration.

Date opened	Order value	Reason	Decision	Minutes spent	Submitted by	Due	Submitted	Outcome	Tool fee
3 Mar	$95	Item not received	Fought	38	You	17 Mar	11 Mar	Won	$0
6 Mar	$140	Fraudulent	Fought	41	You	20 Mar	18 Mar	Lost	$0
9 Mar	$30	Not as described	Refunded	5	You	n/a	n/a	n/a	$0

Keep the log in a spreadsheet and add a column for "weak, medium or strong evidence" if your tool gives a score, so you can later check whether the scores match the outcomes.

Part 3: the scorecard

At the end of each period, fill in one column per period. We recommend computing net benefit and net benefit per dispute, because totals mislead when volume differs.

Measure	Period A: by hand	Period B: with the app	Change
Disputes opened	___	___	___
Disputes fought	___	___	___
Wins	___	___	___
Win rate on decided cases	___%	___%	___ points
Average won order	$___	$___	$___
Value of wins (orders plus $15 fees)	$___	$___	$___
Minutes per dispute	___	___	___
Hours spent in total	___	___	___
Value of your time	$___	$___	$___
Tool cost	$0	$___	$___
Missed deadlines	___	___	___
Net benefit	___	___	___
Net benefit per dispute	___	___	___

Net benefit = value of wins minus value of your time minus tool cost.

A worked example with made-up numbers

The figures below are illustrative. They do not come from a real store, and they are not a prediction. They show how the arithmetic works.

A store with a $30 an hour value on its time runs 60 days by hand, then 60 days with a $19 plan.

Measure	Period A: by hand	Period B: with the app
Disputes opened	22	24
Wins	7	10
Average won order	$95	$95
Value of wins (orders only)	7 x $95 = $665	10 x $95 = $950
Minutes per dispute	35	9
Total hours	22 x 35 / 60 = 12.83	24 x 9 / 60 = 3.6
Value of your time	12.83 x $30 = $385	3.6 x $30 = $108
Tool cost	$0	2 months x $19 = $38
Net benefit	$665 - $385 = $280	$950 - $108 - $38 = $804
Net benefit per dispute	$280 / 22 = $12.73	$804 / 24 = $33.50

Check the totals: 22 disputes at 35 minutes is 770 minutes, which is 12.83 hours, which at $30 is $385. 24 disputes at 9 minutes is 216 minutes, which is 3.6 hours, which at $30 is $108.

On these made-up numbers, net benefit rose by $524 over 60 days, or about $21 per dispute. That sounds like a clear win for the app. Now look at where it comes from.

Source of the gain	Amount	How reliable it is
Time saved	$385 - $108 = $277	High, because the store measured minutes directly
Tool cost	-$38	Certain
Extra wins	$950 - $665 = $285	Low, because 3 more wins out of about 23 cases is within noise

Of the $524 gain, $277 is time saved, which is solid, and $285 is extra recoveries, which is not. The win rate went from 7 of 22, which is 31.8%, to 10 of 24, which is 41.7%. That is a gain of about 10 points, but the sample is tiny.

Can a small store prove that an app raises its win rate?

Almost never, and it is better to know that now than after you have announced a result.

A win rate measured on about 23 cases has a standard error of roughly 10 points. The standard error of the difference between two such rates is about 14 points. A 10 point gap is therefore well inside the noise. You could see it by chance in either direction.

How many disputes would you need? For a 10 point difference, for example 35% against 45%, at the usual 95% confidence and 80% power, the rough sample is 2 x (1.96 + 0.84) squared x 0.24, divided by 0.10 squared. That works out to about 376, so roughly 380 disputes in each group. Most Shopify stores see a fraction of that in a year.

Disputes per group	Win rate gap you can reliably detect
25	About 39 points
50	About 27 points
100	About 19 points
380	About 10 points
1,500	About 5 points

These are approximate, and they assume win rates near 40%. The lesson is that a single store can usually prove time saved and cost, which are measured directly, and cannot usually prove a moderate change in win rate. If a vendor claims a win-rate gain from a small number of cases, you now know how much weight to give it.

This is not a reason to give up on measuring wins. Log the outcomes anyway, because a large effect would still show, and because over a year the numbers add up. Just do not make a decision that depends on a win-rate gap you cannot detect.

Choosing a fair test design

How you split the periods affects how fair the comparison is. Here are the options.

Design	How it works	Strength	Weakness
Before and after	60 days by hand, then 60 days with the tool	Simple, uses all your disputes	Season, promotions and case mix can differ
Alternating weeks	Odd weeks by hand, even weeks with the tool	Controls for season	Disputes opened in a week may be decided in a different one, so tag by open date
Split by reason or order value	The tool handles one band, you handle the other	Matches like with like if bands are balanced	Bands are rarely equal in difficulty
Split by processor	Tool on one gateway, you on another	Clean separation	Few stores have enough volume on both
Random assignment	Flip a coin per dispute	Best at removing bias	Awkward to run, and one owner per dispute is required

For most small stores, before and after is the practical choice. Make it fairer with these habits.

Pick periods of equal length and similar season. Avoid comparing a quiet February with a holiday December.
Freeze everything else. Keep your shipping, refund policy, descriptor and marketing steady, or note every change.
Tag disputes by open date. A dispute belongs to the period in which it opened, not the one in which it was decided.
Give each dispute one owner. Shopify accepts one evidence submission per dispute and treats it as final, so a second submission cannot add to the first.
Record case mix. Count disputes by reason in each period so you can see if one period was harder.
Do not drop awkward cases. Include the disputes you accepted or refunded in both periods.

What is the decision rule?

Write the rule before you start, so the result cannot talk you into something. Here is a template.

Keep the tool if net benefit per dispute is higher in Period B, time per dispute fell by at least a quarter, missed deadlines did not rise, and you trust the responses you reviewed.
Extend the test if the results are close or the sample is small. Run another 60 days.
Cancel if net benefit per dispute is lower, time did not fall, or you found errors in what the tool prepared.

Add one rule about quality. Read five of the tool's responses as if you were the bank. If you would not have submitted them, say so, and weigh that heavily whatever the numbers say.

For the money side, you can also use our break-even guide to turn a plan price into the number of disputes at which a tool pays for itself.

How do you check whether an evidence score is accurate?

Some tools give an evidence strength score. ChargeFix gives each case a score and a worth-fighting verdict. Whatever tool you use, you can test whether its scores mean anything, and the dispute log is all you need.

Add a score column and, once outcomes arrive, fill in a table like this.

Score band	Cases	Wins	Win rate
Strong	___	___	___%
Medium	___	___	___%
Weak	___	___	___%

If scores are useful, win rates should fall as the band gets weaker. If strong cases win no more often than weak ones, the score is not helping you decide. With small samples, expect noise and look for a trend across several months. A score that is directionally right and helps you skip weak fights is already doing something worthwhile, because it saves you the time of cases you would have lost.

The one-page case study

When the test ends, write it up on a single page. This is for you, a partner or an accountant, and it keeps you honest. Fill in the blanks.

Title: Chargeback app vs doing it ourselves, [store name], [dates].

Question: Does [tool] save us time and money on chargebacks compared with handling them ourselves?

Method: We ran [60] days by hand and [60] days with [tool] on [plan]. We logged every dispute, timed every response and kept shipping, refund policy and descriptor unchanged. Case mix was [describe].

Results:

Disputes: [A] by hand, [B] with the tool.
Minutes per dispute: [A] and [B], a change of [x]%.
Win rate on decided cases: [A]% and [B]%, from [n] decided cases, which is [too few / enough] to be sure.
Tool cost: $[x] over [60] days.
Net benefit per dispute: $[A] and $[B].
Missed deadlines: [A] and [B].

What we are confident about: [Time saved, cost and anything measured directly.]

What we are not confident about: [Win rate difference, if the sample is small.]

Decision: [Keep, extend or cancel] because [rule].

Next review: [Date.]

A page like this is worth more than any vendor case study, because you know how it was made.

A second example: when the test says cancel

A test is only worth running if it can say no. Here is a made-up store whose results point the other way. It has very few disputes, so a plan has little to work on.

Measure	Period A: by hand	Period B: with a $19 plan
Disputes opened	3	3
Wins	1	1
Average won order	$60	$60
Minutes per dispute	30	9
Hours in total	1.5	0.45
Value of your time at $30 an hour	$45.00	$13.50
Tool cost	$0	$38 for 2 months
Net benefit	$60 - $45 = $15.00	$60 - $13.50 - $38 = $8.50

Time saved was $31.50, but the tool cost $38, so the net result is $6.50 worse. At three disputes in two months, a paid plan cannot pay for itself unless it adds wins. The decision rule says cancel, or drop to a free plan. That is a good outcome, because it saved the store $19 a month and told it when to check again. If disputes later rise to 10 or more in two months, rerun the test.

How do you handle seasons and outliers?

Two things distort small tests more than anything else.

Seasonal swings. A quiet month and a holiday month produce different disputes, different order values and different customer behaviour. If you must compare across a season, compare per dispute and not in total, and say in the write-up that the periods differ. If you can, run the second period at the same time of year in the next cycle.

One big dispute. A single $2,000 order can dominate a period. If it was won, net benefit jumps. If it was lost, it collapses. Report your results twice: once with all disputes and once without the largest one. If the conclusion changes, you know it rests on one case.

A simple way to guard against outliers is to use the median instead of the mean for order value and for minutes per dispute. The median is the middle value when you list the cases in order, and one extreme case cannot move it. Your spreadsheet has a MEDIAN function.

Also watch for clusters. A bad batch of shipments can produce ten similar disputes in a week, all with the same cause. That says more about your carrier than about your tool. Tag the cause in your log, and treat a cluster as one event when you compare periods.

What if you want to test on a free plan first?

A free plan is a good way to run a pilot without paying. ChargeFix offers a free plan with 2 filings a month, and a dispute counts as one filing the first time its evidence is saved or filed. That is enough to try the tool on two real disputes and time the review. Paid plans include a 14-day trial. Other tools list free tiers too: DisputeDesk lists 5 evidence packs a month and Chargeback Copilot lists 3 evidence PDFs. Always check what happens at the limit.

For a small pilot, pick two or three disputes of different reasons, time yourself on each with the tool and compare with your baseline minutes. Even a mini test of this size tells you whether the drafts are usable and how long the review takes. It will not tell you about win rates, and that is fine.

How often should you rerun the test?

A result is a snapshot, and your store keeps changing. Rerun the scorecard in three situations.

Every quarter, as a light check. Update the same table with the latest 90 days. It takes ten minutes if the log is current.
When your volume changes by a third or more. More disputes can justify a bigger plan, and fewer can make a paid plan wasteful. Our break-even guide shows where each plan pays.
When anything material changes. A new vendor price, a new product line with higher order values, a new payment method or a change in your refund policy all deserve a fresh look.

Keep the old write-ups. After a year, you will have four snapshots of your own store, and that history is worth more than any vendor claim.

What should you do with the result?

If time fell and net benefit rose: Keep the tool, and re-run the scorecard every quarter.
If time fell but net benefit did not rise: Check the plan. A cheaper plan or a free one may fit. Your volume may be below the break-even.
If neither changed: The tool is not for you at this volume, and that is useful to know.
If you found quality problems: Stay in Manual mode, raise them with the vendor and re-test.
If everything is unclear: Extend the test.

Whatever you decide, keep the log. It becomes your own record of what works, and it makes your next decision easier.

Common mistakes in a chargeback app test

Measuring only wins. Time and cost are easier to measure and just as important.
Reading a win-rate gap from a handful of cases. It is noise until you have hundreds.
Changing other things mid-test. A new refund policy or descriptor muddies the result.
Comparing different seasons. Holiday disputes differ from quiet-month disputes.
Counting by decision date instead of open date. It shifts cases between periods.
Dropping hard cases. Include accepted and refunded disputes too.
Skipping the stopwatch. Remembered minutes are optimistic.
Deciding the rule after seeing the result. Write it first.

What we checked, and what you should check

We read the Shopify Help Center pages on the chargeback process, vendor pricing pages and Shopify App Store listings for the tools named in this series, and ChargeFix's own plan descriptions on 30 October 2026. The worked example is invented for illustration. The sample-size figures use standard statistical approximations, which assume win rates near 40% and are rough guides, not exact thresholds.

What you should check: your own minutes per dispute, your own hourly value, your plan's allowance and how your tool counts a dispute against it. The chargeback cost calculator and chargeback recovery calculator can help you fill in the numbers.

Frequently asked questions

How long should I test a chargeback app?

At least 60 days per period, so that you see a reasonable number of disputes. Remember that banks can take up to 75 days to decide, so some cases will still be pending when the period ends. Count only decided disputes in win rates and report how many are pending.

Can I prove an app improves my win rate?

Rarely, on a small store. A win rate from about 23 cases has a standard error near 10 points, so a 10 point gap is within noise. Detecting a 10 point difference reliably needs roughly 380 disputes in each group. You can reliably prove time saved and cost instead.

What should I measure in a before-and-after test?

Eight things: disputes, wins, win rate, average won order, minutes per dispute, tool cost, missed deadlines and net benefit. Net benefit is the value of wins minus the value of your time minus tool cost, compared per dispute so that differing volume does not distort it.

How do I time how long a dispute takes?

Use a stopwatch from opening the case to submitting it, and include the decision to fight, gathering evidence, writing and uploading. Do it for every dispute in both periods. Remembered times run short, so a stopwatch gives a far more honest baseline than your estimate.

Is a free plan enough for a pilot?

For a mini pilot, yes. ChargeFix has a free plan with 2 filings a month, and DisputeDesk and Chargeback Copilot list free tiers. Use two or three real disputes to time the review and judge the drafts. Check what counts against the allowance and what happens at the limit.

What if my dispute volume is very low?

Then the test may show that no tool pays at your volume, which is a useful answer. Below a few disputes a month, a free plan or Shopify's own form is often enough. Our break-even guide shows the volume at which each plan starts to cover its cost.

Should I include accepted and refunded disputes?

Yes. Leaving them out flatters whichever period had more of them. Include every dispute that opened, record what you decided and compare net benefit per dispute across the whole set. Accepting or refunding a weak case is a legitimate outcome and should count.

What if the app's responses look wrong?

Stay in Manual mode, note the errors, and raise them with the vendor. Evidence is final once submitted, so do not let a tool file anything you would not have filed yourself. Read at least five responses as if you were the bank before deciding.

Can I use this template for another tool or for a VA?

Yes. The measures are the same whoever or whatever does the work: minutes, cost, outcomes and net benefit per dispute. For a VA, put their wage and your management time in the cost line. Our guide to hiring a VA versus a chargeback app has the cost per dispute sums.

Start free trial with ChargeFix

If you want to run this test on your own disputes, ChargeFix has a free plan with 2 filings a month, flat monthly plans as you grow, and a 14-day free trial on paid plans. It never takes a cut of what you win. It checks your evidence item by item, gives each case a strength score and a worth-fighting verdict, and drafts a letter from your real records. Manual mode is the default, so nothing goes to the bank until you say so. It works with Shopify Payments only.

Start free trial

Chargeback app vs doing it yourself: a case study template to test it on your own store