Most people who've heard a COBOL horror story have heard the same one: a government agency, a decades-old mainframe, a sudden spike in demand it was never sized for. That story is real and it happened, but it teaches a narrower lesson than people take from it, because a volume spike is at least an intelligible kind of failure. This is a different one, from a private-sector bank with no pandemic, no political spotlight, and no shortage of qualified staff on the night it went wrong. In June 2012, the Royal Bank of Scotland Group's retail banks locked millions of customers out of their own money for the better part of a month, and when the UK's financial regulator finished investigating, it did not conclude that the underlying banking software was broken. It concluded something more specific and more useful to anyone running COBOL in production today: the bank did not know how to safely undo a change to the system that runs its overnight processing, and nobody had ever actually tested finding out.
What a batch scheduler is doing while everyone else is asleep
Retail banking runs on a nightly rhythm that customers never see. Every deposit, withdrawal, standing order, interbank transfer, and interest calculation that happened during the day gets queued up and applied to account balances overnight, in a specific, dependency-ordered sequence, so that by the time branches open the next morning every balance is final and correct. The software responsible for coordinating that sequence, deciding which job runs next and confirming the previous one actually finished, is a batch scheduler. At RBS Group, the scheduler managing updates to NatWest and Ulster Bank accounts was CA-7, a mainframe workload automation product that, then as now, was a standard piece of infrastructure at large COBOL shops for exactly this purpose: it doesn't run the account-update logic itself, it makes sure the hundreds of individual jobs that do run in the right order, without stepping on each other.
“A software update was applied on 19 June 2012 to RBS's CA-7 software which controls its payment processing system.”
Wikipedia, "2012 RBS Group computer system problems"
It's worth sitting with how unglamorous that description is, because it's the whole point of what follows. Nobody's account-balance logic was rewritten. No COBOL program had a typo in it. The failure sat one layer up, in the orchestration software deciding when those programs were allowed to run, and it is precisely the kind of infrastructure change that gets treated as routine, low-risk maintenance work right up until the moment it isn't.
The upgrade that couldn't be safely undone
On Sunday 17 June 2012, RBS Group's centralised IT function, referred to throughout the regulator's investigation as Technology Services, upgraded the batch scheduler software used for NatWest and Ulster Bank from Version 1 to Version 2A, a modified build of the newer Version 2 release, because the older version could no longer be adequately supported. The following night, Monday 18 June, the team ran the first full overnight batch cycle under the new version and immediately saw problems: the mainframe was consuming an abnormally high share of its processing capacity, and a number of batch terminal jobs failed outright. Internal specialists managed to manually re-run the failed jobs and get that night's processing finished, but the anomalies were serious enough that on Tuesday 19 June, Technology Services made the call to back the upgrade out entirely, reverting Version 2A back to Version 1.
That decision is where the incident actually happened. Rolling back an upgrade sounds like the safe, conservative move; in this case it was the untested one.
“Technology Services had only tested backing out the unmodified version of the upgrade (Version 2). In those tests, Version 2 was compatible with the previous version of the software (Version 1). Technology Services had not tested backing out the modified upgraded version of the software (Version 2A) to the previous version (Version 1).”
Financial Conduct Authority, Final Notice to Royal Bank of Scotland, NatWest and Ulster Bank, 19 November 2014
In other words, the team had validated one rollback path, backing out the plain Version 2, and assumed it told them something about a different, modified build they had never actually run that test against. The FCA's investigation found a second gap sitting right next to the first one: even the rollback testing that had been done wasn't done against realistic conditions. Technology Services had tested the back-out process using thin, unrepresentative queue data instead of a volume and mix of jobs that matched what an actual production night looked like, which meant the test environment had no chance of surfacing the exact failure mode that was about to hit production. Version 2A turned out to be incompatible with Version 1 in a way nobody had checked for, and the moment the back-out ran for real, a significant number of scheduled jobs simply failed to appear in the batch queues at all, while the unprocessed jobs that were still pending began to multiply behind them.
Why a scheduling bug took three weeks instead of three hours
A batch failure on its own is usually a bad night, not a crisis; the standard response is to catch up the backlog before the next cycle starts and move on. What made this one different was a structural decision made long before June 2012: NatWest and Ulster Bank shared a single batch scheduler instance, rather than each running its own. That meant a scheduler-level failure that should have been contained to one bank's processing automatically became two banks' problem simultaneously, and it compounded the recovery math badly. By the morning of 20 June, the NatWest backlog from the 19th was largely cleared, but Ulster Bank's had not been, which meant Ulster Bank started the day already behind. By 21 June, Ulster Bank's batch processing was running more than a full day late, which meant the next day's scheduled batch run started before the current day's had actually finished. Multiple days' worth of files were now sitting in the processing queues at once, interfering with each other and generating fresh recovery problems on top of the original ones. RBS's own systems recovered relatively quickly because RBS ran on a separate scheduler unaffected by the NatWest/Ulster failure directly, but NatWest's core systems weren't stable again until 25 June and Ulster Bank's batch scheduler did not return to full functionality until 10 July 2012, roughly three weeks after the initial back-out.
What "customers couldn't access their money" actually meant
The regulator's findings describe the practical effects in specific, mundane terms, and the specificity is what makes them land. ATMs largely stayed online, but they were showing stale balances built on missing or duplicated transactions, which meant some customers risked overdrawing their accounts by withdrawing cash against a balance that hadn't been updated with a payment that had actually gone out days earlier. Standing orders and direct debits weren't processed on schedule, which meant customers who had done nothing wrong picked up late-payment penalties from third parties who had no idea a bank's batch scheduler was the actual cause. Customers travelling abroad had card purchases declined outright and, in some cases, no way to access cash while away from home. Commercial customers using Bankline, the banks' business internet banking service, couldn't verify cheques, manage payments, or in some cases meet payroll on time. It wasn't limited to the banks' own customers, either: anyone expecting a payment from an RBS Group customer, whether or not they banked there themselves, was potentially left waiting on money that should have already cleared. As a stopgap, the banks arranged a £200 "stand-in" spending limit so debit cards could still authorise small purchases even while account balances behind them were unreliable.
The safeguards that were supposed to catch this, and didn't
The part of the FCA's findings most worth reading closely if you work anywhere near a mainframe isn't the timeline, it's the governance section, because it explains why an organisation the size of RBS Group didn't catch a testing gap this basic before it reached production. The bank's IT risk model relied on what it called Three Lines of Defence: Technology Services itself, responsible for identifying and managing IT risk day to day; Business Services Risk, a second line meant to challenge and independently review Technology Services' own view of its risks; and Group Internal Audit, a third line providing independent assurance over the whole arrangement. On paper that's a reasonable structure. In practice, the regulator found each line under-resourced or badly positioned to actually do its job.
“No one on Technology Services Risk's senior management team had a risk or an IT audit qualification.”
Financial Conduct Authority, Final Notice to Royal Bank of Scotland, NatWest and Ulster Bank, 19 November 2014
That wasn't an isolated gap. The FCA found that more than half of the Technology Services Risk team had joined the RBS Group within the two years immediately before the incident, meaning the function meant to catch exactly this kind of change-management failure had little institutional depth of its own to draw on. The second line, Business Services Risk, was found to have had very limited IT skills at the time and hadn't challenged Technology Services' failure to properly risk-assess the scheduler upgrade in the first place. And the most striking detail sits in the third line: Group Internal Audit had actually reviewed the mainframe batch processes, including the back-out procedures for exactly this kind of scheduler change, back in August 2010, almost two full years before the incident, and its own scoping documents correctly identified the potential for a batch processing failure as a real risk. But that audit's own working papers noted it hadn't been possible to fully test the implementation of the relevant controls, because there was no documented evidence of the steps taken in previous back-outs and changes, and the final report didn't treat that missing evidence as a serious enough gap to flag prominently. The right risk was named on paper nearly two years in advance. Nobody closed the loop on it before the calendar caught up.
What the fine was actually for
That legal basis is worth reading carefully, because it's easy to assume a bank got fined £42 million for a software bug. It didn't. It was fined for the absence of the systems and controls that should have caught the bug before it reached six and a half million people, which is a meaningfully different failure and a much more useful one to actually learn from. Nobody was found to have written bad COBOL, or bad JCL, or a bad batch job. The finding was that the organisation around the code, the change-management discipline, the testing standards, the risk staffing, the audit follow-through, wasn't strong enough to make the surrounding software trustworthy. For a mainframe shop today, that's the uncomfortable and useful reframe: the language your batch jobs are written in was never the risk surface here. The process for changing anything that touches them was.
What actually transfers to a shop running this today
- Test the rollback, not just the upgrade. A back-out procedure for a modified build is a different test than a back-out procedure for the unmodified release it was based on, and passing one tells you nothing reliable about the other.
- Test with production-representative data volumes and job mixes, not a thin sample. RBS's own back-out tests didn't use realistic queue data, and that is specifically why the incompatibility wasn't caught before it hit real batch volumes.
- Don't consolidate a shared point of failure across systems that need to fail independently. NatWest and Ulster Bank running on one scheduler instance is why a NatWest-side problem became an Ulster Bank crisis that took three weeks longer to resolve than NatWest's own recovery.
- An audit that correctly names a risk is worthless if nobody owns closing the specific gap it identifies. RBS's own internal audit flagged the exact failure mode almost two years ahead of time and it still happened.
- Staff the people reviewing IT risk with people who understand what they're reviewing. A risk function with no IT audit qualifications on its senior team, reviewing IT risk, was one of the specific deficiencies named in the regulator's findings.
None of this is a story about COBOL aging badly, and it would be a mistake to read it that way. The banking logic itself, the actual COBOL processing deposits, withdrawals, and interest, appears nowhere in the FCA's list of causes. What failed was everything wrapped around it: the discipline of testing a change the one way it actually mattered, the staffing of the people meant to catch that gap, and the courage of an audit function to insist that a documented risk actually got resolved rather than just filed. That's a harder problem than a syntax error, and it's exactly the kind of problem that reference material focused only on language mechanics will never help a shop avoid.
Try the Job Log Decoder tool →Browse the COBOL reference →