Gmail accounts recovered – from tape
"I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centres?" asked Ben Treynor, Google's site reliability czar, in the firm's official Gmail blog.
"Well, in some rare instances software bugs can affect several copies of the data. That's what happened here," he added.
He said that Google backs up data on offline tapes, which are protected from software bugs.
"But restoring data from them takes longer than transferring your requests to another data centre, which is why it's taken us hours to get the e-mail back instead of milliseconds," he said.