Hey Chops, my guildies and myself had issues getting to our website this morning. I also couldn't connect to THIS site. Was there a server down or some such?

Nope. I'm not sure what the problem was, but I was able to access it fine, as were a few people I know, and the monitoring service also noticed no downage. I suspect there was some kind of odd routing issue on the internet. I had a friend text me to tell me it was down, and I thought it was odd, since the monitoring service typically picks up on those things before anyone else does and notifies me immediately. None of the servers were offline, and none of them were even exhibiting signs of issue (all webservers online, all database servers online, all with plenty of available slots), and the data center was accessible.

So yeah, I'm stumped as to what that was, but I suspect some kind of routing issue going on somewhere. At least that's the best theory I have.

For research purposes, can I ask what ISP you use, and your general location (state or country if outside of US).

I use Comcast in California, near San Francisco. However I had guildies in other parts of the US that had similar issues. I know one uses Comcast, but not certain of the others.

The cause of the problem yesterday is found!

http://twitter.com/DNSMadeEasy

DNS Made Easy, one of the largest, most reliable DNS services, and the DNS service we use is under a major DDoS from China (estimated at 30Gb/s). Though they seem to have gotten things completely under control by now.

This further explains why my monitoring service, my own browser, and other users didn't lose connectivity.

The monitoring service likely cached the DNS result, and since it knew the IP address, didn't need to look it up every time. Browsers tend to do this as well, to speed up the requests (which is why you might notice that the first time you open a browser in a day, it takes just a tiny bit longer to get to a page on the first load, and much faster thereafter - it's not checking the DNS anymore). Anyone who kept their browser open had a chance to not need to look up the DNS info again.

Fascinating.

If you want to read some of the technical discussion going on at Hacker News (a popular technical new aggregator, here's the link to the current discussion: http://news.ycombinator.com/item?id=1583860 )

Thanks Chops!