Integrated Online Solutions

2001.09.20 Why Things Sucked the Last Few Days...

The connectivity problems we have been experiencing the last few days are the direct result of the Nimda worm which was unleashed Tuesday morning. Our servers were being hit with an unprecedented load, vastly greater than anything I've seen before. Our servers actually, although slowed to a crawl, never did crash; however our upstream bandwidth provider's routing did. When it seemed that our provider was not remedying the situation quickly enough, it was decided to run another cable to a neighboring provider in order to get us back on the net sooner, and improving our protection against such occurrences in the future. You will notice that the IP numbers have changed for Vex.Net (although for the time being the web server still can be reached redundantly via the old IP). We are still connected via ethernet to major internet backbones. Just to give an idea of the strain on our servers (and the internet in general these days, as this attack is not directed at anyone specific) D'Arcy reports that he saw the web server running with a load average of 45 for a while. During the first 12 hours of the worm's life it generated a quarter of million hits on our main web server (which is 3 to 4 times greater than the traffic on most of our busiest hosted sites combined). Over the next half day it generated nearly a million more hits. The worm activity seems to have subsided greatly now, and our connectivity is mostly stabilised, though we continue to be hammered by it and experience slight routing problems from some hosts. Naturally, since we don't run any Microsoft software on Vex.Net, we were not infected by nor propagated the worm in any way: we just were subjected to the suffocating power of all the insecure infected Windows boxes around us. Some info on the Nimda worm, and it's wide reaching effects: http://www.cert.org/advisories/CA-2001-26.html http://www.sarc.com/avcenter/venc/data/w32.nimda.a@mm.html http://www.theregister.co.uk/content/56/21756.html http://www.washingtonpost.com/wp-dyn/articles/A59059-2001Sep19.html