As noted on iTNews.com.au:
http://www.itnews.com.au/News/152531,primus-hit-by-third-dc-outage-this-year.aspx
As many hosting providers know, the Primus Datacentre in Melbourne was yesterday hit by it’s third outage this year, throwing a bucketload of client’s into darkness yet again!
Their excuse? A failure in their diesel generators failing to start along with backup UPS systems.
Our thoughts: Inexcusable!
Whilst we do not have any equipment located in their datacentre, upstream providers such as PIPE and Vocus Connect (2 of Hightek Hosting’s main providers) do, and suffered the impact of the outage.
The end result was in turn our own services being affected.
What did we do?
Well, as per normal, BGP routing done it’s bit and fired all routes over to backup links. But what happens when your trying to squeeze a bucketload of connections down a few smaller backup providers? You get packetloss, timeouts and so much more than that!
Effectively, their outage brought our networks to a real slow down; slower than the speed of molasses being poured at the North Pole in mid winter as noted by one of our clients!
Unfortunately, with an outage of this nature, our hands were tied. We did not have anything in Primus, but dealing with PIPE and Vocus put us as much in the firing path as many others.
We reacted with as much force as we could, but we still, like many others, were powerless to make a difference, so all we could do was scream.
And scream we did!
But one has to ask…when will they get it right?
3 outages in a single 12 month period from an entire datacentre and simply are joke!…actually, they are beyond that point now!
But what can we do?
Nothing infact! We can’t do a thing. We are not going to drop PIPE or Vocus, as they are the ones that brought Hightek out of the woodwork and into the limelight, so why reverse that? It’s not their fault that Primus can’t keep it together. So why punish them?
Of course, we will do everything within our power to sway their decisions for future network locations, but at the end of the day, we simply have to bear with it.
Primary links are good to have, and thanks to BGP, so are secondary, but unfortunately, we copped a double whammy and have our 2 main providers go out at the same time, so the backup links we failed over to were highly congested for quite some time until PIPE came back online, followed by Vocus.
The real question here is how much longer is saga going to go on? How many more outages can we expect this year from Primus?
I don’t know about you, but many other providers in the industry are getting to the point of boiling with this, but the worst part is, we’re all in the same boat.
Anyone who wants to have Melbourne based services and use that such of PIPE or Vocus will have to deal with the matter until someone take’s action!
We’re hoping that Primus get into gear, fix their power issues and pick up the pace a little. Like anything, it would be good to see them recover from this and get it right!….but no-ones getting their hopes up!
We’ll see what unfolds before christmas! 
