Running without IT: fallback processes and catch-up entry

IT outages are not a rare side issue for companies. The German digital association Bitkom puts the damage that companies suffered from failure, theft or impairment of information and production systems or business processes in connection with data theft, industrial espionage or sabotage at 36.5 billion euros (Bitkom) for 2026, after 73.3 billion euros the year before. The figures are based on a survey of 1,003 companies with ten or more employees. At 25 percent (Bitkom) of the companies surveyed, attackers had encrypted data in the twelve months before the survey and would only release it against a ransom. A company whose systems are encrypted works without them until the data has been restored.

An attack is not the only cause. On 19 July 2024 (BSI and Bitkom 2024) a faulty update to a security software product led to numerous IT outages worldwide. In a joint, non-representative survey of affected companies by the German Federal Office for Information Security (BSI) and Bitkom, 48 percent (BSI and Bitkom 2024) said they had to suspend operations temporarily, for ten hours on average. Among the directly affected companies it took two days (BSI and Bitkom 2024) on average until the disruptions were fully resolved. Ten hours of standstill is a full working day on which orders, deliveries and calls still arrive. Why updates are an outage risk in their own right and how to roll them out in a planned way is covered in the article Applying updates without stopping the business.

Power, by contrast, rarely fails for long. The average unavailability of electricity in 2024 was 11.7 minutes (Federal Network Agency) per end consumer; the figure covers unplanned interruptions of more than three minutes, excluding force majeure events. A fallback mode therefore mainly has to cover the loss of the systems a company uses to run its processes: merchandise management or ERP, tills, file server, telephone system and the connection to the data centre. When they are gone, customer data, stock levels, prices and order history disappear at the same time.

This article deals with what the business does while these systems are down and how it gets back to a complete data set afterwards. What a backup needs in order for the restore to succeed is covered in Backups that hold up in an emergency. How a silent failure of a single interface gets noticed and how data records are preserved until recovery is shown in When an interface fails. Here the focus is on the people in the business and on what they need without a screen.

Not every process needs a form. The selection starts with one question per process: how long can it be down before the damage becomes unacceptable? The BSI standard calls this the maximum tolerable period of disruption; it is determined for each business process from its damage potential (BSI Standard 200-4). In payroll, a delay of a few days may be tolerable; in a wholesaler’s order intake, barely half a morning. Processes with a short tolerance get a fallback route, the rest are deliberately put on hold.

Good candidates are the processes that customers and goods depend on: taking orders, picking in the warehouse, goods out and shipping, cash sales at the till and customer service on the phone. A process map for mid-size companies showing the main processes of the business is a good starting point because it shows which process waits for which upstream one. In wholesale processes, for example, shipping depends on picking and picking on order intake; if the first station fails, all three stop.

The table shows the principle: the fallback tool only covers the details needed to keep working. Whatever the system otherwise does along the way, such as stock postings, credit checks or dispatch notices, is not rebuilt on paper but caught up on after recovery. Anyone who tries to replicate the ERP system on paper ends up with forms that nobody fills in completely when it matters.

In an outage it is not only data that is missing but also the rules the system otherwise enforces. In normal operations the software prevents anyone from accepting an order above the credit limit or granting a discount outside the price list. In fallback mode, people decide. The BSI standard therefore provides for business continuity planning to record the special rights, permissions and duties of staff in an emergency, explicitly including rights within approval processes; they apply from the activation of the plan until de-escalation (BSI Standard 200-4).

Each of these roles needs a named deputy. An outage does not follow the holiday schedule, and the person who knows the system inside out is needed for recovery in an emergency, not for customer calls. How deputies and approval limits are handled digitally in normal operations is described in Digitising approval workflows. For fallback mode one sheet of paper with names, mobile numbers and limits, kept in every emergency box, is enough.

The work is not over once the systems have been recovered. Between the moment the system runs again and the moment it reflects the actual state of the business lies catch-up entry, and deadlines apply to it. Every business transaction must be recorded promptly, as soon as possible after it occurs, in a primary record or journal. Recording non-cash transactions within ten days (Federal Ministry of Finance, GoBD para. 47) is considered unobjectionable, and cash receipts and payments must be recorded daily (Federal Ministry of Finance, GoBD para. 48).

For the period until entry, the GoBD require organisational measures so that records are not lost and name consecutive numbering of incoming and outgoing invoices as an example (Federal Ministry of Finance, GoBD para. 50). That is exactly what numbered form pads provide. If the gap between recording and posting extends beyond the end of the following month (Federal Ministry of Finance, GoBD para. 52), the requirements are only met if the transactions were previously recorded continuously, correctly and completely in primary records. A fallback mode lasting several days is therefore not a problem; a catch-up entry that is left lying around is.

The fourth step deserves particular care. An order that was still created in the system at 8:05 a.m. and again on paper at 8:20 a.m. shows up twice after catch-up entry. The same happens when an interface pushes stalled records through after recovery that have meanwhile been entered by hand. A unique transaction key and a check before writing prevent this; what that looks like technically is shown in Preventing duplicate postings.

After catch-up entry the paper forms remain records. Accounting vouchers must be kept for eight years (German Fiscal Code, Section 147 para. 3), incoming and outgoing business letters for six years. If the forms are scanned, the rules on image capture and on destroying the paper apply, as described in GoBD-compliant document storage. A note in the data record stating where the original is kept saves searching during a later audit.

An emergency plan that nobody has tried out is an assumption. After the IT outage of July 2024, 62 percent (BSI and Bitkom 2024) of the directly or indirectly affected companies had prepared an emergency plan for such IT outages, and in most cases it worked. The president of the BSI concluded that practised IT emergency concepts must be an important part of any crisis preparedness (BSI and Bitkom 2024). Practised means that staff have had the forms in their hands before it gets serious.

A drill does not have to bring the whole business to a halt. An announced afternoon on which one area works without the system is enough for a first round. Three things are measured: how long it takes until the first order has been taken on paper, whether everyone involved knows where the forms and phone list are kept, and how long catch-up entry of the drill transactions takes. The last measurement is decisive for planning because it shows how many hours of backlog a real one-day outage would create.

After the drill, forms with too many or too few fields are adjusted and the findings go into the process documentation so that they do not depend on individual people. An annual repeat and a drill after every major system change keep the plan up to date. New employees get to know the emergency box during onboarding; this can be combined with training and rollout of new processes, where the fallback level that remains in place for emergencies is defined anyway.