announcements

As you may have noticed, my historically stable host of 5+ years, Linode, has been having repeated DoS issues in many of their data centers over the past week or so. They've been posting updates to their status blog pretty frequently as problems arise (bots4 is hosted in their Atlanta data center), so that is a good place to look if you're curious about what's going on. The problems have also been discussed on Hacker News.

These DoS problems have been translating into some periodic downtime for the game as well as perhaps increased latency. You can look at the game's public Pingdom uptime monitoring page to get an idea of how much downtime this is creating. It's short on most days (under 10 minutes), but there was 59 minutes of downtime on 12/27.

Short of switching to a new host, there isn't really a whole lot I can do. I'm not quite ready to leave Linode, so I just wanted to make a quick announcement that I'm aware of and am monitoring the situation. Sorry for the inconvenience and thank you for your patience. :(

Another 41 minutes of downtime just concluded. Here's a more detailed update from Linode about what's been going on:

• "It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business."

• "Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency."

• "Please know that we are dedicating all resources from multiple departments to stopping these attacks. We acknowledge the amount of downtime we’ve been experiencing is completely unacceptable, and we appreciate the understanding and support we have received over the past several days. We will share more information as our investigation continues."

And here's the relevant Hacker News discussion about it.

Today was pretty rough. Nearly 11 hours of downtime. :( Linode is taking this situation very seriously, but I will continue to monitor and consider alternative hosts as the situation unfolds.

Sucks, but it's nice to stay informed. All BBC sites have been knocked down recently too.

My host thinks they've closed off all the attack vectors that have been affecting the game's server's data center. Other than a couple blips, things have been stable for the past 9 hours or so. Let's hope this uptime continues...

Great news, thanks for keeping us updated :)

It indeed looks as though we are out of the woods:

We have seen stability in Atlanta for enough time to consider this outage resolved. We apologize for the immense disruption to customers during this two-day outage, and we will be following up in the next week with a full post-mortem and plans for the future.

Nice.. :-)

I was locked out from home page last night and everything this morning (pacific time)

Yesterday's outage was for a different reason.

(I know you already know this because you posted on that thread - I figured I'd answer here anyway for others that might have missed it and for posterity.)