How To Deal With A Payments Outage

What can businesses do when their payment provider fails?

Disclaimer: views expressed here are my own and do not represent any other organisation

Businesses around the world were unable to take payments when Square suffered a total outage earlier this month. The outage started on the 7th of September and took almost 24 hours to resolve. 

The business impact of this outage was significant. Merchants couldn’t accept payments via their Square devices for the duration. As the Wall Street Journal reported:

Jesse Stommel, owner of PlayForge, a toy-and-games store in Littleton, Colo., said they asked customers if they could manually write down their credit-card information. “Some customers just walked away,” he said. Stommel estimated he lost around 30% of usual weekday business.

“If this had happened on a Saturday, we would’ve lost a huge portion of our weekly business,” Stommel said.

Some businesses, such as a Blue Bottle Coffee in downtown San Francisco, taped “cash only” signs on their doors Thursday afternoon. Businesses as far away as Australia said they were affected. 

Asking for customer to pay in cash may have been OK a few years ago. Nowadays, many Americans don’t use cash in a typical week, making it less likely they will carry cash at all. So “cash only” will likely mean customers going elsewhere.

Source: Pew Research Center - More Americans are Joining the Cashless Economy

The share price of Square’s parent company, Block, dropped by 5.6% during afternoon trading on the day of the 7th of September. Such a drop shows how financial markets viewed the situation. Subsequently Square’s root cause analysis identified the issue as a DNS (Domain Name System) Failure:

While making several standard changes to our internal network software, the combination of updates prevented our systems from properly communicating with each other, and ultimately caused the disruption. The issue also affected many of our internal tools for troubleshooting and support, making them temporarily unavailable.

In response, Square is increasing the ability to take payments when connectivity goes down. This will help mitigate any future similar issues. But companies will be concerned about the extent of the outage and how long it took to resolve.

On the 19th of September, Square announced that CEO Alyssa Henry is stepping down. Square’s founder, Jack Dorsey, will now return to be CEO. This change was not directly linked to the outage in official company communication, but some will speculate that this was a factor. 

Square is now facing calls for refunds from many unhappy merchants. There will have been many lost sales, and closer regulatory scrutiny can be expected. Following the outage, Australia’s payments regulator is considering publishing reliability data, so that companies can compare the uptime of various Point of Sale solution providers.

Businesses need a back-up plan

DNS failures, are not uncommon. Online workspace company Notion was hit by a comparable issue in 2021. Later in 2021 Facebook suffered a global outage due to a BGP error (a similar, but slighty different issue to DNS).

But if WhatsApp goes down, most people will have a backup - sending an SMS, using another messenger service such as Signal, or even sending an email. 

If their payments processor goes down, most businesses don’t have an alternative option except reverting to cash only. So, companies big and small should have a backup option in case their payment processor fails.

Businesses should be proactive in getting a backup solution. Outages are rare, but it makes sense to prepare. Not much can happen if an outage occurs without a backup solution in place. Let’s consider that:

• Large merchants often connect to many payments processors. This is for redundancy and to benchmark different providers’ capabilities. 

• The easiest option for a small business would be to buy a second card reader. Square, SumUp and PayPal POS (known as Zettle in Europe) offer similar devices on a pay-as-you-use basis. So if one stops working, swap to the other. 

• Software companies are now building apps which can take payments directly on a smartphone. Download an app and take payments directly from an iPhone or Android device. Use it as a backup or as the primary Point of Sale. 

• MPoC regulation enables tap-to-phone technology to process payments even if the connection is lost. Once the connection is back up, transactions can be completed.