For the last stretch of work on one of the oldest systems I support, I have been trying to do something that sounds straightforward and turns out not to be straightforward at all: make a legacy payment authorization engine safe to change.

The application itself is not especially fashionable. It is a TCP ISO‑8583 processor written in VB.NET, with a large amount of behavior accreted over roughly fifteen years. It talks to a SQL Server database, owns meaningful balance and hold behavior, and sits directly in the path of real authorization traffic. It is exactly the kind of system where “just add a feature” is a dangerous sentence.

The challenge was not merely that the code was old. The more important constraint was architectural: this is a latency-sensitive, effectively single-threaded engine in the places that matter most. If you add even modest delay in the wrong branch, you are not slowing down a background job. You are slowing down live authorizations on a system that has a five second SLO and a lot less headroom than anyone would like.

That constraint shaped everything.

The goal was never “rewrite the legacy engine in one motion.” The real goal was narrower and more practical: build enough testability, observability, and surrounding infrastructure that pieces of the engine could be offloaded safely, starting with external authorization behavior, without turning the production system into an experiment.

The First Problem Was Not the Code

The first problem was the absence of a real development and validation environment.

This system had a database, of course. It had logs. It had production behavior. What it did not really have was a deterministic local baseline that expressed the application in a way an engineer could work with confidently. There was no single, disciplined test corpus describing seeded cards, balances, holds, distributors, edge-case statuses, and message histories in a way that made strange branches intentionally reachable.

So the first substantial piece of work was building that baseline.

I created a local database setup that did more than stand up schema. It seeded specific cards and stateful artifacts for particular behavioral paths. That distinction matters. A legacy financial system is not meaningfully testable just because the tables exist. It becomes testable when a specific PAN, a specific card ID, a specific balance row, and a specific distributor configuration can be used to reproduce a branch deliberately rather than accidentally.

That seed work was more important than it sounds. It forced a transition from folklore to explicitness. Many long-lived systems survive on institutional memory: “that path should work,” “this distributor behaves differently,” “that reversal logic is weird for historical reasons.” As soon as I had to encode those assumptions into seed SQL and deterministic fixtures, I found out which ones were real, which ones were incomplete, and which ones were simply wrong.

The Application Needed a Test Harness It Never Had

Once a deterministic database baseline existed, the next missing layer was obvious: a real replay and validation harness.

That became auth-engine-test-harness, and it changed the entire shape of the work. Instead of ad hoc socket poking, manual message crafting, and vague confidence in production-adjacent behavior, I could start expressing the system as executable cases:

• a specific payload
• an expected MTI
• an expected action code
• an expected external-auth scenario
• and, increasingly, expected database side effects

That harness grew into more than a simple replay tool. It now includes deterministic payload suites, seeded artifact tracking, SQL assertion support, scenario-driven external-auth mocking, and helper tooling for things that are annoyingly specific to legacy payment systems, like generating encrypted PAN and pseudo-account values for seeded fixtures.

The important part is not that I had tests in the abstract. The important part is that I finally had a corpus.

A corpus means the application is no longer a black box with a few known happy paths. It becomes something I can interrogate. I can ask it about status codes, zero-balance external auth behavior, retransmissions, hold creation, completion matching, reversal interactions, malformed partner responses, and timeout paths. More importantly, I can ask those questions repeatedly, from a fresh baseline, and get answers that are stable enough to reason about.

This work immediately paid for itself.

Along the way I uncovered multiple real bugs. Two came directly from behavioral testing - I had to assume that the existing application was a ‘known good’, by the nature of the fact that it’s the current expression of the ISO-8583 standard for that organization. Except there were two test cases that were causing noisy, catastrophic errors. One of them turned out to be a simple mismatch that in this particular application was easy to fall into - the application assumed that a variable was a 20 place integer (for the card number of a card) - but what the database and stored procedure were expecting was a 50 character string, fully encrypted and hashed value (for looking up the card by the encrypted at rest card number). The second was similar - a stored procedure within a stored procedure, and drift between the two in how they handled particular data types.

That kind of discovery is exactly why I no longer believe testability is a side concern in modernization. In a system like this, testability is the work. You can cargo cult performance, and stare hard at each line of code, and even feed each line into code analyzers or LLMs - but there’s no way to really appreciate what is happening until you have the functional pieces talking to each other in a way that allows data to traverse end to end.

Observability Was Missing, Not Just Incomplete

The next major gap was observability.

The legacy engine did produce logs, but not in a way that gave us trustworthy operational visibility into message shape, timing, routing, or side effects. I needed structured capture, correlation, safe redaction, and the ability to compare what the legacy system was actually doing to what the artifacts of its work looked like.

That effort became flume, a log management utility that gave me a more coherent way to handle capture and inspection around the legacy engine. It’s a deceptively simply (and totally unglamorous!) solution - the legacy application is single-threaded. I/O writes to a local hard drive are cheap and always have been, latency wise. But getting the logs, in a cogent form, TO somewhere is hard - besides network latency itself, data transformations and management aren’t cheap enough to be on the hot path.

So we took them OFF the hot path. Instead of forcing the legacy app to log in a way that was accessible and enabled analysis and reporting - we let it log the way that it always had (locally to lightly formatted text files) - and built an application to parse those files and formats and upload them to Betterstack, where they can be viewed (90 seconds timelate, typically), but more importantly - analyzed - error patterns, latency patterns, even transaction counts.

This may be the least glamorous part of the project (and, hilariously, the easiest), but it was one of the most transformative.

When you are trying to modernize a fragile financial system, you need to answer questions that plain text log files are often very bad at answering:

• What was the actual message received?
• Where there any errors during the processing of that message?
• What’s the actual latency, in production with a production database and a production network and production compute, for every message?
• What patterns are there in terms of behavior? What makes those patterns important or pertinent?
• When we see failures and latency rise, is it because of partner behavior? Application misbehavior? Database configuration? Or sometimes, some combination of all three?

Before this work, a lot of those questions were painful (and sometimes, impossible) to answer. After flume, they became obvious.

With less than a week of ‘live’ logs, and another three weeks ‘backfilled’ by pre-existing logging artifacts through flume for testing, we identified three major bugs. One of the more sobering examples was a thirty second SQL timeout on a system that is supposed to live inside a five second SLO. And instead of being an impossible to reproduce production artifact - we realized that it was, in fact, a resource contention issue that we’d seen throughout development of the test harness itself! There is a special kind of clarity that comes from watching a local repro demonstrate that a “rare production oddity” is actually a concrete, reproducible defect. And even better when you can demonstrate that a fairly trivial code change can ‘make it go away’.

It was the biggest PR of actual production code to change on the legacy application - a whopping +205/-87 change across two files (I suspect it was less than that, even - there’s some line-ending silliness that infected the PR).

The ‘30 second timeout’ bug, that brought our single-threaded application crashing to a halt for 30 seconds at a time, was happening about every other day before the code change. It immediately disappeared post code change.

That is not a cosmetic improvement. It’s not even a performance improvement. That’s an architectural improvement.

The Strangler Needed To Start Outside the Legacy Process

With a deterministic database, a replay harness, and materially better observability, we could finally start shaping the strangler path around the legacy engine.

The first principle was simple: the legacy process should not have to stop being itself all at once. The first real effort, in 15 years, to modernize the legacy codebase in a significant way, could begin not just because ‘newer was better’. It could start from the ground up to make the existing codebase BETTER - to remove processes that were slowing down the existing code, and focus on improving the throughput and performance of what the code was already pretty good at.

To whit - with a whole months worth of data ingested from flume, I could do a quick analysis of p50/p90/p99 performance. And the results were sort of surprising.

I’d been prepared for anything from these numbers. Local testing showed all kinds of weird behavior in terms of latency - but it was hard to take it seriously even directionally, because there were just so many factors that could conflate with actual application and component level performance.

The numbers from production, though, were surprisingly sunny. Some of the p99/long tails were frightening (see 30 second responses above) - but the ‘average’ response, when the application only had to process messages and check the database, were shockingly fast. Less than 50 milliseconds, on average.

Could a modern language + database, optimized, beat that? Of course! But…I wasn’t expecting to find that the existing codebase could handle 20-25 messages per second, even single-threaded. We’d been scrimping and scraping and shaving performance gains everywhere, we thought, because single-threaded 15 year old codebases CAN’T have that kind of performance buffer…can they?

But the numbers were definitive. For the vast majority of transaction types, the existing codebase was…actually doing great!

But there WAS one area where it wasn’t hard to see room for improvement. And it has everything to do with single-threading.

Continued in Part II!