ÐÇ¿Õ´«Ã½

Skip to main content

Facebook blames outage on error during routine maintenance

Share
LONDON -

The global outage that knocked Facebook and its other platforms offline for hours was caused by an error during routine maintenance, the company said.

Santosh Janardhan, Facebook's vice president of infrastructure, that Facebook, Instagram and WhatsApp going dark was "caused not by malicious activity, but an error of our own making."

The problem occurred as engineers were carrying out day to day work on Facebook's global backbone network; the computers, routers and software in its data centers around the world along with the fiber-optic cables connecting them.

"During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally," Janardhan said Tuesday.

Facebook's systems are designed to catch such mistakes but in this case a bug in the audit tool prevented it from properly stopping the command, Janardhan said.

That change also triggered a second problem that made things worse by making it impossible to reach Facebook's servers even though they were operational.

Engineers scrambled to fix the problem on site, but this took time because of the extra layers of security, Janardhan said. The data centers are "hard to get into, and once you're inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them."

Once connectivity was restored, services were brought back gradually to avoid traffic surges that could cause more crashes.

It was an "unforeseen anomaly" for a faulty maintenance update to take down Facebook's backbone network, but the company probably could have avoided a scenario in which its servers were completely taken offline, making it impossible to access the tools needed to fix it, said Angelique Medina, of Cisco Systems' ThousandEyes, a firm that monitors internet outages.

"The big question is why so many internal tools and systems could have a single source of failure," Medina said. "Facebook would still have been down because of the network outage, but they could have resolved the outage sooner if they had internal access."

CTVNews.ca ÐÇ¿Õ´«Ã½

Since she was a young girl growing up in Vancouver, Ginny Lam says her mom Yat Hei Law made it very clear she favoured her son William, because he was her male heir.

The search for a missing six-year-old boy in Shamattawa is continuing Friday as RCMP hope recent tips can help lead to a happy conclusion.

BREAKING

BREAKING

The New Brunswick RCMP has issued an alert as officers search for an armed teenager in the Moncton and Shediac areas.

An Ontario man says it is 'unfair' to pay a $1,500 insurance surcharge because his four-year-old SUV is at a higher risk of being stolen.

Local Spotlight

They say a dog is a man’s best friend. In the case of Darren Cropper, from Bonfield, Ont., his three-year-old Siberian husky and golden retriever mix named Bear literally saved his life.

A growing group of brides and wedding photographers from across the province say they have been taken for tens of thousands of dollars by a Barrie, Ont. wedding photographer.

Paleontologists from the Royal B.C. Museum have uncovered "a trove of extraordinary fossils" high in the mountains of northern B.C., the museum announced Thursday.

The search for a missing ancient 28-year-old chocolate donkey ended with a tragic discovery Wednesday.

The Royal Canadian Mounted Police is celebrating an important milestone in the organization's history: 50 years since the first women joined the force.

It's been a whirlwind of joyful events for a northern Ontario couple who just welcomed a baby into their family and won the $70 million Lotto Max jackpot last month.

A Good Samaritan in New Brunswick has replaced a man's stolen bottle cart so he can continue to collect cans and bottles in his Moncton neighbourhood.

David Krumholtz, known for roles like Bernard the Elf in The Santa Clause and physicist Isidor Rabi in Oppenheimer, has spent the latter part of his summer filming horror flick Altar in Winnipeg. He says Winnipeg is the most movie-savvy town he's ever been in.

Edmontonians can count themselves lucky to ever see one tiger salamander, let alone the thousands one local woman says recently descended on her childhood home.