My Server Was Hacked!

This post is a bit outdated since the hack actually occurred about 8 months ago. But I thought it might be helpful to some folks who find themselves in the same boat I was in.

Back in early February I was hacking around my production Linux server when I happened to stumble upon a bizarre command in my "history" file. It was a removal command for a script I had never heard of... My stomach about fell through the floor. I quickly realized my production server had been hacked!

Quickly I checked my last login history and found that someone had logged in as "root" just a few hours before. According to the logs they were on my server for about 9 minutes. During that time they installed a web-based file system browser and then spent the remainder of their time on my server browsing my web site files. It appeared they were looking for passwords or database connectivity information. They also tried to grab my database, but it didn't seem to work according to the log files.

After 9 minutes they logged off. Then back on again about a half an hour later for just 1-2 minutes. No activity that I could tell other than to maybe remove the web script they left behind.

Aside from being scared to death of what I just witnessed in my log files, I came to the realization that I could not even trust what I was reading in the log files. Anyone who obtained root access could have easily spoofed the log files to make me think they didn't get very far...

I quickly started scanning my security log files to see how the heck they got in. For a few months now I had been noticing a brute force attack on my server. Someone was trying to guess a userid/password via SSH. My firewall was configured to block them by IP if they tried too many times, so I was not too worried about it. Well I should have been more worried. They eventually got in via their brute force attack. They successfully guessed my root password and bingo, the doors opened up. Wow. My worst nightmare.

I quickly changed every password on the box and locked it down. I had previously left a small door open (SSH) so I could remote to my box from anywhere if needed. I didn't want to limit SSH incoming requests from my home IP since I travel a lot. Well this came back to bite me.

So I locked SSH down via my firewall (APF) and came up with an alternative solution involving a 3rd party service to allow me to configure APF safely and have a dynamic IP address. That is all fine and dandy, but I still cannot be 100% sure the hacker didn't leave some other back door programs lying around.

I contacted my ISP to see if they could help me sniff out any potential problems. They checked my box out and said they didn't not see anything, but they agreed that there could still be a hidden trojan and we'd never know it until it was too late.

So we made a big decision to build a completely new server from scratch. We took a back up of our websites and data from before the break-in and used it to restore our websites on the new server. We locked down SSH access via a dynamic dns service, and we made our passwords as random, long and secure as we could.

We learned a hard lesson that month. Never ignore a potential security threat. Never think that your defenses are better than they are. Never underestimate.