VIRTUAL BUSINESSES Amazon and Microsoft lost some of their hosted services after a power utility was struck by lightning in Ireland.
Perhaps one particularly important web site operator had been ticked off by his service, or maybe there are just too many metal rods pointing out of facilities over there, we can't say, but what we can say is that the hammer of Thor hit the premises and it took them down.
"What we have is preliminary, but we want to share it with you," reads an explanation on the Amazon web site.
"We understand at this point that a lighting strike hit a transformer from a utility provider to one of our Availability Zones in Dublin, sparking an explosion and fire."
Under usual circumstances, if such circumstances can be considered normal, when power is lost electrical loads should be switched to back up generators. However, the explosion was apparently so large that it took down backup systems as well.
"Normally, upon dropping the utility power provided by the transformer, electrical load would be seamlessly picked up by backup generators. The transient electric deviation caused by the explosion was large enough that it propagated to a portion of the phase control system that synchronizes the backup generator plant, disabling some of them," explained the firm.
"Power sources must be phase-synchronized before they can be brought online to load. Bringing these generators online required manual synchronization. We've now restored power to the Availability Zone and are bringing EC2 instances up. We'll be carefully reviewing the isolation that exists between the control system and other components."
Amazon is still having problems and so are its customers. Its health dashboards are showing a range of connectivity issues in Europe, all of which emanate from the Ireland-based datacentres.
One reason for this delay is the sheer scale of the problems. In one status message, applied in this case to the Elastic Compute Cloud (EC2), it says, "Due to the scale of the power disruption, a large number of EBS servers lost power and require manual operations before volumes can be restored. Restoring these volumes requires that we make an extra copy of all data, which has consumed most spare capacity and slowed our recovery process."
Problems are not limited to that either, and the Amazon Relational Database Service has also been experiencing problems and users were warned of rather lengthy delays to complete restoration of service.
"We continue to make slow progress restoring connectivity to Single-AZ RDS database instances. The rest of the Single-AZ restores are tied to recovery of EBS volumes associated with these RDS database instances," says the latest service update.
"We know many of you are waiting for your database instances to be restored. As we explained in the earlier EC2 post, we expect to make faster progress as were able to bring more EBS capacity online."
Cloudformation stacks were also affected, but there services have been completely restored.
Microsoft's problems, which saw connectivity lost to its Business Process Online systems, have been resolved, according to Twitter posts from the firm.
"#BPOS services are back online for EMEA customers. Please see Service Health Dashboard for updates," reads the latest advice from Microsoft. µ
Sign up for INQbot – a weekly roundup of the best from the INQ