Customer story · Subscription billing
Billing bugs that cost money
A SaaS team started using CodeValid to validate their software. CodeValid found 9 bugs in their subscription and billing features. Each one either quietly lost money or locked out a paying customer.
TL;DR
The discount was removed. The software kept applying it. That was one of 9 billing bugs that 40 checks, written from the team's own requirements, found in about 2 hours. By hand, we estimate the same work at about 54 hours.
Express · Postgres · Stripe · PayPal · 14 features
What CodeValid found
CodeValid tested the things customers and admins do every day. Each one has a clear right answer. Here are four where the app did something else.
An admin removes a customer's discount.
Lost revenue- Business intent (requirement)
- The next bill is full price.
- Application behavior
- The app says the discount is gone, but the customer keeps getting it on every bill. Nobody on the team can see it.
A blocked customer pays what they owe.
Paying customer locked out- Business intent (requirement)
- Their access comes back.
- Application behavior
- They stay locked out until someone on the team turns access back on by hand.
A customer restarts a paused PayPal plan.
Paying customer locked out- Business intent (requirement)
- Their access comes back.
- Application behavior
- The app thinks the plan already ended and keeps them out.
A customer's card payment fails.
Paying customer locked out- Business intent (requirement)
- A grace period with read-only access, as the requirements say.
- Application behavior
- They are locked out the moment the payment fails.
None of these show an error or crash the app. Everything looks fine until a customer complains, or until someone notices the revenue is short.
The setup
The product is a subscription web app. Customers pay monthly or annually through Stripe or PayPal, can start with a trial, and get blocked when they stop paying. Admins can grant access, revoke it and apply discounts. The team connected their requirements, from those requirements, CodeValid generated test scenarios and ran them against the real app.
- 1Removing a discount doesn't remove itStripe · 2 bugs: two endpoints
- 2Paying doesn't unlock a blocked accountPayments · 1 bug
- 3A reactivated plan gets a past billing periodPayPal · 1 bug
- 4No grace period after a failed paymentSubscription gate · 1 bug
- 5An unpaid trial can't be cancelledPayments · 1 bug
- 6Shortening a trial to end now never expires itTrial jobs · 1 bug
- 7Renewal warnings were never builtEmail · 2 bugs: monthly and annual
Why these bugs are hard to reach
Every one of these bugs needs an account in a specific state. A trial that was never paid. A card that failed yesterday. A PayPal plan that was suspended and then reactivated. A coupon that was applied but not yet used.
By hand, getting there means sandbox accounts with both payment providers, waiting for or faking billing cycles, triggering webhooks in the right order, and seeding database rows that have dozens of required columns. That is often an hour of setup before you can check a single rule. So most teams test the happy path and hope.
CodeValid built the entire setup automatically, so every one of these scenarios can be reproduced on demand.
The whole stack starts in about 2 minutes. Each check seeds the state it needs, calls the real API, and checks the result against the requirement.
14
Billing features
40
Checks written
9
Real bugs
The time it saved
This is one run, on one area of the product.
| Work | By hand (est.) | CodeValid |
|---|---|---|
| Test environment and vendor mocksPayment providers, email and background jobs | ~12 h | built at onboarding |
| Putting accounts into the right billing stateTrials, failed payments, suspended and reactivated plans, coupons | ~16 h | 2 min |
| Writing 40 checks | ~20 h | 1.5 hours |
| Running, triage and evidenceRun the suite, find the line of code behind each failure | ~6 h | 0.5 hours |
| Total, one run | ~54 hours | 2 h 02 min |
~52 hrs
More than an engineer-week back, and 9 bugs found before a customer reported them.
And it doesn't stop at one run
Billing code keeps changing. A new plan, a new discount rule, a fix for one of these bugs. By hand, every change means rebuilding the same account states and re-testing the same rules, so most of those hours come back with each release. Teams end up spending their time on test setup instead of testing.
With CodeValid, the environment and the 40 checks already exist. They run again on every code change, and the fixes for these 9 bugs are checked the same way. The time saved grows with every change the team ships.
Assumptions: by-hand hours are our estimate for an experienced engineer building the same coverage. CodeValid time is measured run.
Find your quiet bugs
Billing is where code that looks right costs the most. Connect your repo, bring your requirements, and see what CodeValid finds.