One of Australia’s largest pension funds with USD90 billion (AUD135 Billion) invested on behalf of 647,000 members suffered a Google Cloud outage that crippled it’s customer services for almost two weeks in May.
In Australia pension funds are known as superannuation funds, often abbreviated to “super funds”. The fund hit in this outage “UniSuper” claims to be one of the largest.
The outage occurred when Google Cloud deleted the servers UniSuper was running in the Google Cloud without warning due to a bug in Google’s software. Naturally, being a conservative well prepared financial sector organisation, UniSuper had regular backups, in multiple geographies, but they were also stored in the Google Cloud - no doubt a convenience. But Google also deleted these backups!
The stuff of nightmares!
But UniSuper had an ace up their sleeve as they also “had backups in place with an additional service provider”. They stated that “These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.”
Google’s official response recognised the importance of independent third party backup software in saving the day, stating that “third party backup software, were instrumental in aiding the rapid restoration.”
Google seems to be stretching the language to call one week recovery “rapid restoration” and that the third party backup was just “instrumental” - it is clear that the “third party backup software” was key and saved the pension fund from a much larger and potentially existential total disaster. Although it seems the third party backup may have been incomplete and restoration was slow.
Quotes above are from A joint statement from UniSuper CEO Peter Chun, and Google Cloud CEO, Thomas Kurian and Sharing details on a recent incident impacting one of our customers.
What can we learn from this?
Some commentators are quick to criticise Google Cloud specifically. But the reality is that despite extensive engineering efforts to reduce failures in cloud environments software bugs and system bugs affect all systems and all software. So data loss and interruptions are possible in all cloud environments. Simply moving away from Google Cloud to Amazon, Azure or any of the second tier providers is not a real solution. Enterprises using other hyperscalers exclusively should not sleep soundly either.
In the on-premise days of old, “off site backups” where vital even for small companies that wanted to stay in business. They still are for those still running “on-prem”. This incident shows that the principles of the “3-2-1 rule” for backups still hold and “Off cloud backups” or more practically “Multi cloud backups” are essential in the cloud era. These backups should be as complete as possible and the restoration scenario well documented and tested to match the risk profile of the organisation. Ideally these backups should also be independent of your cloud provider(s) software stack. SEP sesam is one such solution that should be on your short list for evaluation.
SEP sesam is an independent software backup solution that can save data to your choice of storage, including S3 object storage which is ideal for cloud environments. SEP sesam supports essentially all databases & applications running on Linux and Windows operating systems.
What about SaaS Apps?
Are you safe if you use SaaS apps such as Microsoft 365 & Dynamics 365, Google Workspace, Azure AD & Salesforce instead of virtual servers in the cloud? No, you have similar vulnerabilities. Completely independent backup is also necessary for full protection and is available with the solution SEP CAPS. More information is available HERE.
For more information on SEP sesam please see our web site SEP sesam page or SEP CAPS page and contact CustomTech (email info@custom.tech or call) to arrange a FREE evaluation and discuss your needs.
Interested? But got questions? A free initial discussion can help provide the answers, book HERE.
