Facebook: human error cause of major failure

A human error was the cause of last night‘s major failure on Facebook, whatsapp, and Instagram. The company writes that in a comprehensive statement about the incident.

The malfunction started with a configuration error. A system administrator accidentally gave a command that shut down Facebook’s global backbone network. That‘s a network that connects Facebook servers worldwide. This led to a domino effect, which prevented those servers from being connected to the Internet.

As a

result of the failure, Facebook’s servers were unreachable, even for Facebook‘s employees themselves. As a result, failure crews in Facebook’s data centers needed physical access to the servers to investigate and fix the error. The rigorous security of the servers made it a time-consuming job.

Gently reachable

Once the error was fixed, Facebook made its apps reachable bit by little. This was meant to prevent the systems from hitting up again as billions of phones worldwide tried to reach Facebook, whatsapp or Instagram at the same time.

โ€œThings like this can always happen. People are making mistakes,โ€ says tech editor Joost Schellevis: โ€œAlthough it seems Facebook was very much on the functioning of its own infrastructure. So if things go wrong there, things go wrong right away.โ€

The malfunction started around 5.30 pm Dutch time yesterday and lasted until a while after midnight. Here that led to complaining users, but not major problems. The timing was quite painful for Facebook: today a whistleblower in the US Congress testified about abuses at the company.