The evil triad: Radio, my old firewall, and my ISP. The past 72 hours have been quite unintentionally Radio free. I began to notice a lot of traffic going over my network when I didn't think I was generating any, so I did a packet capture and saw all the ugly stuff coming from Comcast's network, Nimbda requests, SMB share requests, and even a few infected SQL servers trying to grab ports 1433. I calculated the average number of these "events" every hour. I receive 25 of these potentially harmful requests an hour on the Comcast network. Naturally, I did a "what tha...?" and promptly installed an old Sonicwall Soho2 I had left over from a T1 I had installed before cable Internet access was available in my area. Some times ignorance is bliss. As soon as I installed the Sonicwall, my real nightmares began. All of a sudden Radio wouldn't upstream via FTP. Ok, fine. I checked my Radio settings, checked the "Firewall or NAT" checkbox, and tried again. No dice. I checked my firewall admin console. "What tha ...?" My firewall thought it was being attacked ... FROM THE INSIDE!!! Here is what I saw:
08/08/2003 01:40:03.752 - FTP: PORT bounce attack dropped. -
Source:192.168.1.102, 49187, LAN -
Destination:64.49.219.158, 21, WAN -
Target host: 0.0.0.0, 1090
It looks like Radio is picking up the IP address at system.temp.myIpAddress, which my firewall is interpreting as an attempted redirect to another host. I tried adding rule after rule, static routing, and every trick I could think of. No dice. I backed off and did some research. I come to find out that my Sonicwall only supports passive FTP mode. Great! I run back to Radio and check the PASV option for FTP upstreaming and try again. No dice. "What tha ...?" I contacted my ISP and my FTP server does not support passive FTP mode and they won't enable it. Great!!! What now?
I have spent the last 24 hours contemplating my options. Somewhere in the last 24 hours I decided to see if I could move from Radio to MoveableType. Keep in mind I have two years of weblog entries, photos, audio files, and code stored in my weblog. I downloaded and tried Bill Kearney's Exporter utility. Besides this, have you tried installing Perl packages on OS X? Don't try it. If you have to, use the reference from Apple's site and don't trust what the installation scripts tell you. I spent five hours just trying to install MoveableType on my Powerbook. I finally got MT installed only to find out that the Six Apart crew changed their export/import file format since Bill released Exporter. "Doh!" So Radio remains. What now?
My next option was to change ISP's. I bought myself a Basic account at Digital-Crocus ($30 for a year. Great deal.), since they come so highly recommended by John Robb. Digital-Crocus supports passive FTP mode. Excellent!! I modified my upstream settings and gave it a try. It worked!! Wait a second. What was that? About every minute and a half, my FTP connections were getting dropped. The only explanation I could come up with was that my connection was getting deprioritized trying to cross the pond. Damn. This prompted me to sit down and figure out what it would take to move from my current ISP, BlueDomino, to Digital-Crocus. I run a decent size bug tracking database and project portal for my clients. Moving databases and applications around is never fun. At this point, I had already spent close to 20 hours trying to find a solution. I estimated it would take me at least another 20 hours to plan, test, and move everything. This would also mean a disruption in service for my clients who use these applications while DNS propagates and I move the databases. This isn't good. What now?
My next option was to throw out the Sonicwall and replace it with something. After watching the logs fill up a number of times since Wednesday with port scans, viruses knocking at my door, and general deviant behavior on the Comcast network, I decided I need a firewall that offers stateful inspection, logging, and reporting features. It wouldn't hurt to have VPN functionality since my new employer can set me up as a node in their network via VPN. I did some research and found that Netgear offers a firewall, router, VPN, DHCP server all-in-one device for $150, the FVS318. Ding, ding, ding!! We have a winner, or so I thought.
I ran to Best Buy and purchased a FVS318, ran home, and installed it. At first it looked like everything was working great. When I check the timestamps on my files, nothing was happening. Again, "what tha ...?" I then had the bright idea of check my Radio event log and found many of the follow messages:
Can't upstream because "Can't find a sub-table named "2166"."
After doing some research and finding expert posts from
Paolo Valdmarin and
Rogers Cadenhead about these messages being generated by full disks on a server, I contacted my ISP. Sure enough, the disks on my server were full and they had the situation fixed in about eight hours.
Upon confirming the issue had been resolved by my ISP, I tried upstreaming again. No dice. I was at my wits end. I was still get the sub-table error. I saw nothing in my firewall logs that would lead me to believe that something shouldn't be working, yet I couldn't upstream. The firewall has limited functionality in terms of custom logging. I had to resort to watching tcpdump output via a terminal on my laptop. What I saw was that there was no ACK packets coming back, which leads me to believe that the firewall was dumping the packets thinking that there was a FTP bounce attack going on. Now I was stuck.
I was fresh out of options. Back to the drawing board. I put my original network back in place and fired up the firewalls on each of my machines. My final option is to build my own firewall on a PC that gives me the flexibility to redirect individual packets, define granular rules, and identify specific attacks to automatically warded off. I have learned my lesson with these consumer firewall devices. These device are just fine if you are surfing the web, but as soon as you get into gaming or specialized application they do you no good.
I come away from this ordeal with a lot of wasted time, a new firewall that I will be returning to Best Buy, and a nice development account at Digital-Crocus. It cost me roughly 30 hours and $180. Was it worth it, you might ask? Definitely. I have gathered a lot of data on the Comcast network in New England that I will be analyzing and publishing in the coming weeks. I will also be writing up my experiences installing MoveableType and Perl on OS X. While the exercise was by no means a success, it was a valuable experience and has further motivated me to pursue my security certifications. I'm glad it's over.
