CodeValidCodeValid

Customer story · Subscription billing

Billing bugs that cost money

A SaaS team started using CodeValid to validate their software. CodeValid found 9 bugs in their subscription and billing features. Each one either quietly lost money or locked out a paying customer.

9bugs

TL;DR

The discount was removed. The software kept applying it. That was one of 9 billing bugs that 40 checks, written from the team's own requirements, found in about 2 hours. By hand, we estimate the same work at about 54 hours.

Express · Postgres · Stripe · PayPal · 14 features

What CodeValid found

CodeValid tested the things customers and admins do every day. Each one has a clear right answer. Here are four where the app did something else.

An admin removes a customer's discount.

Lost revenue
Business intent (requirement)
The next bill is full price.
Application behavior
The app says the discount is gone, but the customer keeps getting it on every bill. Nobody on the team can see it.

A blocked customer pays what they owe.

Paying customer locked out
Business intent (requirement)
Their access comes back.
Application behavior
They stay locked out until someone on the team turns access back on by hand.

A customer restarts a paused PayPal plan.

Paying customer locked out
Business intent (requirement)
Their access comes back.
Application behavior
The app thinks the plan already ended and keeps them out.

A customer's card payment fails.

Paying customer locked out
Business intent (requirement)
A grace period with read-only access, as the requirements say.
Application behavior
They are locked out the moment the payment fails.

None of these show an error or crash the app. Everything looks fine until a customer complains, or until someone notices the revenue is short.

The setup

The product is a subscription web app. Customers pay monthly or annually through Stripe or PayPal, can start with a trial, and get blocked when they stop paying. Admins can grant access, revoke it and apply discounts. The team connected their requirements, from those requirements, CodeValid generated test scenarios and ran them against the real app.

CustomerAdminReact web appbilling, plans, trialsEXPRESS API · NODE 20Subscription gatewho gets accessPaymentscheckout, renew, cancelAdmin toolsdiscounts, grant, revokeTrial jobsreminders, expiryPostgres 16StripePayPalRedis job queueEmail1253467
The billing side of the app. Red markers show where each bug was found. The grey boxes are outside services, which CodeValid set up for testing automatically.
  1. 1Removing a discount doesn't remove itStripe · 2 bugs: two endpoints
  2. 2Paying doesn't unlock a blocked accountPayments · 1 bug
  3. 3A reactivated plan gets a past billing periodPayPal · 1 bug
  4. 4No grace period after a failed paymentSubscription gate · 1 bug
  5. 5An unpaid trial can't be cancelledPayments · 1 bug
  6. 6Shortening a trial to end now never expires itTrial jobs · 1 bug
  7. 7Renewal warnings were never builtEmail · 2 bugs: monthly and annual

Why these bugs are hard to reach

Every one of these bugs needs an account in a specific state. A trial that was never paid. A card that failed yesterday. A PayPal plan that was suspended and then reactivated. A coupon that was applied but not yet used.

By hand, getting there means sandbox accounts with both payment providers, waiting for or faking billing cycles, triggering webhooks in the right order, and seeding database rows that have dozens of required columns. That is often an hour of setup before you can check a single rule. So most teams test the happy path and hope.

CodeValid built the entire setup automatically, so every one of these scenarios can be reproduced on demand.

The whole stack starts in about 2 minutes. Each check seeds the state it needs, calls the real API, and checks the result against the requirement.

Subscription & Billing · 2 h 02 min

14

Billing features

40

Checks written

9

Real bugs

Every result went to the team.

The time it saved

This is one run, on one area of the product.

WorkBy hand (est.)CodeValid
Test environment and vendor mocksPayment providers, email and background jobs~12 hbuilt at onboarding
Putting accounts into the right billing stateTrials, failed payments, suspended and reactivated plans, coupons~16 h2 min
Writing 40 checks~20 h1.5 hours
Running, triage and evidenceRun the suite, find the line of code behind each failure~6 h0.5 hours
Total, one run~54 hours2 h 02 min

~52 hrs

More than an engineer-week back, and 9 bugs found before a customer reported them.

And it doesn't stop at one run

Billing code keeps changing. A new plan, a new discount rule, a fix for one of these bugs. By hand, every change means rebuilding the same account states and re-testing the same rules, so most of those hours come back with each release. Teams end up spending their time on test setup instead of testing.

With CodeValid, the environment and the 40 checks already exist. They run again on every code change, and the fixes for these 9 bugs are checked the same way. The time saved grows with every change the team ships.

Assumptions: by-hand hours are our estimate for an experienced engineer building the same coverage. CodeValid time is measured run.

Find your quiet bugs

Billing is where code that looks right costs the most. Connect your repo, bring your requirements, and see what CodeValid finds.