Post

Fern: a Neobank Backend, Load Tested

Fern: a Neobank Backend, Load Tested

Fern is a small neobank backend I built to see what it takes to move money correctly: a double-entry ledger on Postgres, ACH through a real bank sandbox, and the failure modes a money system actually meets. It runs on AWS for a week with a demo UI that shows the trace of every request next to a phone.

Sandbox only. No real money, no real identities.

The five cases

Each case is an everyday flow, chosen for the failure it forces the system to handle.

#CaseFailure it demonstrates
S1Get paid: a payroll ACH credit landsthe bank’s webhook arrives twice
S2Pull $500 from an external bank (ACH debit)the money is held until settlement and the return window
S3Send $50 to a friendconcurrent spends never overdraw
S4Withdraw to an external bank (ACH credit)the bank accepts it, but the response is lost
S5The pulled $500 comes back as unauthorized (R10)the balance goes negative, the account freezes, ops writes the loss off with two approvals

The demo walks through them in order, one button at a time. The panel on the right is the real trace of the request you just made: every SQL statement, every bank, Plaid and Redis call, and the journal entries it posted.

Architecture

1
2
3
4
Browser ─► Cloudflare Worker (waken.dev/apps/fern) ─► Cloudflare Tunnel
             └─► AWS EKS: FastAPI (Python 3.13) ─┬─► RDS Postgres 16, Multi-AZ   (ledger)
                                                 ├─► ElastiCache Redis 7          (rate limits)
                                                 └─► Column (bank, ACH) · Plaid (balances) · OpenAI (ops report)

Ledger. Every accounting effect is one journal with two postings, and each effect has exactly one legal shape: ach_debit.settle must debit bank and credit ach_debit_receivable, nothing else. Any effect that touches cash must point at a bank transaction of the same amount. Customer balances may only go negative through a return, never through a spend. Account rows are locked in id order, so two transfers between the same accounts can’t deadlock.

Idempotency everywhere. Every money-moving request carries an Idempotency-Key; every journal has a dedupe key. Replaying a request, or receiving the same bank webhook twice, posts nothing new. Posting a different effect under an existing key is an error worth alerting on.

The bank is the source of truth. When a call to the bank times out (S4), the money stays held and the transfer is unknown, not failed. Recovery asks the bank what happened and books only what the bank confirms. Reconciliation compares the ledger with the bank’s own records and opens an exception for anything that doesn’t match.

Integrations. Column’s sandbox for accounts and ACH, behind the same interface as an in-process simulator that follows Column’s documented behaviour (outgoing credits leave at initiation, pulled debits arrive at settlement, webhooks repeat and arrive out of order). Plaid’s sandbox checks the linked account’s balance before a pull. OpenAI writes the daily ops report, with a rules-based fallback and a daily budget.

Rate limits. A token bucket per space in Redis, as one Lua script. Redis holds no money state, so if it’s unreachable the limiter lets the request through and says so in the trace.

Infra. EKS on two Graviton t4g.medium nodes, RDS db.t4g.micro Multi-AZ, ElastiCache, provisioned with OpenTofu. Traffic comes in through a Cloudflare Tunnel, so there is no load balancer, no NAT gateway and no inbound port. The estimate for 72 hours was about $18.

Shutting down. At a fixed time the Worker stops forwarding to AWS and serves only static files. The demo page was recorded on the live system with ?record=1; in the archive it replays that recording, so the buttons still work without a backend.

Experiments

Load test method

The load generator runs as a Kubernetes Job inside the cluster, against a separate copy of the API with the in-process bank (the Column sandbox is not load tested). Every run starts from a fresh database.

  • Mix: 40% GET /me, 15% GET /transfers, 15% POST /p2p, 12% POST /withdrawals, 12% POST /pulls, 6% replays of an earlier request with the same idempotency key.
  • Think time: 0–1 s between requests.
  • Step test: 5 stages of 60 s, more load at every stage. A stage is within target when p95 ≤ 500 ms with no errors.
  • Money check after the run: every space is compared against a model of the requests it accepted. Books must balance, balances must match exactly, nothing may be negative, and no replay may have created a second transfer.

Throughput alone says nothing about a money system. The money check is the part that matters.

A deadlock first

The load test found a deadlock before it produced a throughput number. A request held its database connection for the whole response, and saving the request’s trace needed a second connection from the same pool. Under load, every connection belonged to a request waiting for another connection. The fix was a small separate pool for traces, and a test that fires 40 concurrent writes and fails on a timeout.

Run 1: production size

Against the same db.t4g.micro the demo uses, with two API replicas:

StageRequestsreq/sp50 msp95 msp99 msErrors
12,904489.7601410
25,8089710.4571090
39,88216531.96331,1770
49,8761651591,3021,6620
55,9701001,0504,2855,7550

Capacity within target: about 97 req/s. Throughput levels off at 165 req/s and then falls as queries queue for CPU: RDS sat at 85–95% CPU from stage 3, while the API pods were not the limit. 35,133 requests, 13,856 transfers, 0 errors, 0 money errors.

Run 2: a larger database

The second run used its own RDS db.m7g.2xlarge (8 vCPU, Multi-AZ), 12 API pods on two c7g.4xlarge nodes, and a build that writes traces with group commit. It cost about $2.50 an hour and was torn down right after.

StageRequestsreq/sp50 msp95 msp99 msErrors
111,7361965.839420
223,2623886.440430
346,4287748.144500
468,5981,14311.757990
579,5241,32583.22714160

Every stage met the target. 1,325 req/s at p95 271 ms, 1,442 req/s at the best 10 seconds. 230,894 requests in 5 minutes, 90,222 transfers across 800 spaces, 0 errors, 0 money errors.

Per endpoint over the whole run:

EndpointRequestsp50 msp95 msp99 ms
GET /me92,3559.4109190
POST /p2p34,84331.0207335
GET /transfers34,5347.993171
POST /pulls27,70639.8189344
POST /withdrawals27,67352.2300451
replay (same key)13,78313.666197

What the numbers do and don’t say

  • RDS CPU peaked around 57% in run 2, so the database was no longer the limit. The latency rise at the last stage more likely comes from the API processes: 11 of the 12 pods landed on one node. I didn’t measure it directly.
  • Between the two runs both the database class and the trace writes changed, so the 13× gain is not split between them.
  • Both runs use the simulated bank. They measure Fern’s API, Postgres and Redis, not Column.
  • The number I care about most is the same in both runs: zero money errors, including under 6% duplicate replays.

Takeaways

  • Model every money movement as an effect with one legal shape, and make the database reject everything else. Most correctness came from that table of effects, not from careful handlers.
  • Treat a timeout from the bank as “unknown”, hold the money, and ask. Never retry a transfer blindly, and never mark it failed on a guess.
  • A load test for a money system needs a model to check the books against afterwards. Requests per second without that check is just a number.
  • Load tests find concurrency bugs that unit tests don’t. The load test’s first useful result was a deadlock, not a throughput figure.
This post is licensed under CC BY 4.0 by the author.