Yes. Many years ago a server was gotten into by a brute force ssh script. The cracker then used a local exploit to gain root access and run all sorts of unpleasant software, downloaded my databases, etc...
I ended up just getting a new server, as I couldn't track down all the things he'd done. I started learning about server security. I reduced the number of people who had ssh accounts, and installed stronger password rules. I use iptables to block brute force SSH attempts. I use logcheck, chkrootkit, and other tools to constantly monitor my server.
I'd say the the single most important thing that you've mentioned there is 'logcheck'. If you can remove all of the login spam (by moving ssh to a non-default port for example), then watching your logs becomes a reasonable task and will alert you to any specifically targeted attacks.
When I was a young sysadmin at an ISP in the mid-nineties, we had a shell server that would be exploited fairly regularly. This was in the days before automated updates, when the standard Sun delivery method for patches was a mailed CD-ROM. (Online patches were available, but you had to seek them out, and even then they occasionally lagged behind the exploit code being posted to bugtraq by a few days.)
A big part of "securing" this particular machine was to separate it from the rest of the network so that even if compromised, it had no more access than the outside world. I set up an RSA private key for SSH authentication so that I wouldn't have to provide my password to a possibly compromised sshd. I backed up the tripwire database to removable media and checked the integrity of the machine's binaries regularly.
Shell access as a standard feature of dial-up internet access died right around that time, and I don't think many ISPs were sad to see it go. That and Usenet were huge time/resource sinks that I'm not sure provided a lot of value in the end to our average customer.
When I was 17 at my first job developing web sites, I set up a PHPBB forum for a customer. Brazillian script kiddies used a security flaw to gain access to one of our entire servers and ruined a bunch of sites.
Great beginners lesson. Keep your mainstream opensource apps up to date.
I was going to tell this exact story, including PHPBB and a Brazilian script kiddie taking down a ton of sites. I wasn't the one who installed the BB though. I just lost a bunch of data. Was it the LFNetwork server? :)
I had an asterisk server with polycom phones on a network not connected to the internet and because I'm lazy all the phones had 1234 as their SIP password. If you're inside the building, you don't need to hack the server, just use the phone in front of you...
Fast forward a couple of years and now some of us work at home or telecommute. So I had to configure an asterisk server hosted in a datacenter. I used the same config files for the autoprovisionning of some of the phones.
Well, one of the account got exploited(SIP scanner that tried simple password on common extensions) and called thousands of numbers in Colorado(from Québec, so long distance) to fish for credit card numbers. Our termination provider saw that these calls didn't correspond to our normal usage pattern and deactivated our account until I called them.
In all, it only cost us 10$ in long distance fees because of the audit server of our provider(unlimitel.ca). Now all the phones and servers in our business have strong passwords. And I had no excuses, the phone are auto-provisioned so phone users don't even see their SIP password...
Yes, due to the first OpenSSH remote exploit. With some many servers to manage one of them was forgotten and hacked.
Our recovery was a bit complicated, because we didn't trust any of our boxes after that. So, we shut down the hacked box, mounted somewhere else, removed all the data and reinstalled. We reinstalled all of our other boxes too... Lots of work and lesson learned.
To secure our systems now we take multiple approaches:
-Different operating systems across our network. For example, our web server runs on OpenBSD and our database on Linux (RHEL). We also have Debian and FreeBSD.
-Reduce the functionality of each system and who can access it. Plus, never allow SSH from one critical system to another.
Added monitoring. We can't be 100% safe, so we started using:
Yes. I made the mistake of providing accounts for friends, and one of them had a common account name and an easily guessable password - it was bruteforced, the server rootkitted, and used as a 0-day ftp drop. I only noticed when bandwidth usage went through the roof.
I didn't trust the server at all after that, of course, so nuked it and reinstalled from scratch. Sans my idiot friend's account. Luckily it was a standalone and not part of the network or i would have had a lot more work to do; once something gets inside you just can't trust anything. Needless to say I would never have been giving away accounts on a big, fast, well connected work machine to friends though .. coughs .. anyway.
Was a good lesson about not trusting others to know what they're doing, taking steps to minimise bruteforcing (different port, limited tries) and rootkit detection, which is pretty hard. In fact it can be almost impossible to even know you've been rootkitted if you do not take proper precautions in advance. I only knew because they'd blown their cover with the bandwidth; that was a bit of a wakeup call. Nothing like having one of your boxes subverted before your eyes and you don't even notice .. that was a nice feeling, not.
Still, glad it happened. Getting hacked is like catching chickenpox; good to happen nice and early (but not too early), builds resistance for later on. Haven't been hacked since.
I run a filedump as part of all the stuff we do, one day some smart alec figured out that I had forgotten that if somebody uploaded a .php file that it would execute instead of being downloaded... Very very stupid of me.
The next step was to upload a nice little backdoor filemanager package written in php and run it from the download directory (this must have been harsh because the url switches after every download). Eventually they got access to the password hashes, but since I found out about the hack within 24 hours I just changed the passwords on the system after plugging the hole.
Obviously the way to handle that was to refuse files whose exension is .php.
I still can't believe that it was that easy to get in, and I feel totally stupid for not having foreseen that.
It's possible that you've still missed something, unless you're certain that .php is the only extension which your server executes (.php5 often works, and some servers will also run files with .cgi or .py extensions, among others). Blacklists are never fun.
The solution which occurs to me is (assuming you're on Apache) setting the .htaccess file in that directory to cause all files to be downloaded, regardless of their type. I haven't tested it, but I think this is the setting you'd need to make:
It's really hard to harden yourself against code getting interpreted by the browser (and thus allowing someone to steal cookies and impersonate other users). Here's a long article on exactly how complex it is (as known today):
http://tstarling.com/blog/2008/12/secure-web-uploads/
Personally, I just don't trust web uploads on the same domain. Put them on a subdomain or better yet a completely separate domain, preferably served out of something set up to never execute any code whatsoever.
Excellent read Timothy, thank you. I have changed some of the code because of this article. I also checked the history of all uploads to see if any of the holes mentioned had been used, and apparently there was at least one instance of somebody trying the .js route but they got hung up on the fact that the file names are changed after every download.
Yeah, I had someone alter the HTML on a static webpage at some point to make it link out to some sites in China.
It was a simple site, so I just scanned all the pages to ensure the rest weren't tampered with (it was just index.html).
Then I switched webhosts (from iPowerWeb to NSFN+WebFaction) because iPowerWeb was incompetent. (And not just because of this incident; they had 3 hour wait times when I called, weekly unplanned downtime with both their web and email servers (which they wouldn't even admit), lost a valuable business domain which was snapped up by some fellow in India 2 minutes before I found out about it, and billing problems on top of it all...)
Once in 2007 when I was 14,I was running a RealVNC server on a computer with the default port forwarded. Around the same time, unknown to me, a buffer overflow exploit was released for the particular version of RealVNC that I had. Some kid eventually found it, and proceeded to do stupid things on my computer (IM my friends, change my background, etc). Luckily nothing of any importance was lost.
Another time, in 2006, I had a shell account on my friend's Linux server. At the time I was using it to host a Wordpress blog. My friend was running an outdated version of PhpMyAdmin, and it was exploited. The guy who got in dumped all of the databases, and on top of that, e-mailed us screenshots of him doing it. :/
Thankfully, all of the users had recent enough backups to deal with.
Yes. Mine got a SQL Injection attack so the hacker was able to run scripts that changed the database. I thought I plugged the holes to prevent it but apparently, I missed one.
I also did not make a fuss about it since the hacker likely wants some attention. I was backing up my db regularly so I just restored from a backup when it happened.
I set up a sendmail that was hacked to send Chinese spam. Lesson learned: don't set up a sendmail server unless you really know what you're doing. I think I nuked the whole server and started over.
Edit: In fact, we ended up moving those domains over to an Exchange server to let the Microsoft guy run them :(
Back in the day, I ran a Mandrake linux box from home on an old laptop (student days, wahey!) which used to host my site and emails. One fine day, I realised that it was getting a little sluggish in terms of performance.
On a closer look, running "top" showed a process called "z" that was eating the process up. Yup, only called "z". It turns out that someone was running a script looking for awstats vulnerabilities, managed to find my installation and installed "z". "z" opened a back door and a small file management utility. Thankfully, nothing was deleted. I think the intruder couldn't get out of the www-data account. I just reinstalled the whole thing from scratch.
The worst part of it was that the vulnerability (and the solution) had been out in the open for months!
Lessons:
1. Always keep an up to date system
2. Subscribe to all announcement/security mailing lists for sites/packages that are deployed to the public
I got burned about 10 years ago by 'samba' not doing what I thought I'd told it to. I used the 'bind interfaces only' configuration directive, which I thought would prevent it from opening any ports on my Internet interface. I was wrong.
Unfortunately I also had no idea about the concept of security patches and, to this day, I still have no idea how I should have gone about getting security updates for Slackware (I switched to Debian and never looked back). The result was some script kiddie got root and started to use my box to start scanning for more vulnerable samba installations to break.
My response was to unplug all of the network cables and have a poke around to see what he'd been up to. I took a full backup of the box and then re-installed it from scratch as I couldn't trust it.
I learned that you should always look at what ports you have open (`netstat -lpn` is my favourite command for this) and that there are some times when a firewall might be of use (I'm not a fan of firewalls on anything other than gateway boxes).
I've had a dedicated server running my (personal) web site, mail, and a variety of other services, directly connected to the public internet, with no firewall, for five years.
My domain is #95k in alexa, #69k in Compete and #35k in Quantcast. Just to say, while it's no heavy hitting destination site, it's not just a tiny personal site either.
I've never been hacked. I just don't set up insecure services.
Yes, last year my blog got hacked by a script kiddo. The blog software I was using had just been found vulnerable and almost all blogs using that software got hacked that day.
I had one old backup but lost a few months posts. Was terrible but got over it soon.
About 6 years ago I was running a "top list" script that was exploited. They planted a backdoor in an obscure folder, so that even after I removed the script they got in again twice. Took me a week to dig out every dodgy file they'd put in.
Also had my very first PHP script exploited (2004ish). I was threatened by some jerk who told me if I didn't remove it from my server he'd exploit it. I didn't, he did, lost months worth of work.
Yes. It was an up-to-date install of Wordpress, about two months ago. Luckily, the domain was being scanned for vulnerabilities by Duke University OIT, and they let me know about it; it was non-obvious that the hack was there. There were certain URLs that had lists of keywords, mostly pharma scams.
In 2001 I was running a redhat 6.1 server at my house, and the ftp had a known vulnerability in it, but I didn't use ftp so I ignored the update. Then someone was over at my place and trying to retrieve an ancient mail archive from an old account, and asked me to turn it on briefly so they could upload it to me; in the couple of hours it was on, a worm installed a rootkit and a IRC based control thing. None of it worked because it was compiled against different versions of libraries; it hung with a full cpu load, which is how I discovered it. I was using TripWire at the time so I was able to identify all the files that changed. I tracked the attack back to someone else's Redhat 6.1 running on a residential DSL line in S. California, which had a web page advertising an auto salvage place on it, and I got no responses to my friendly emails. I started paying attention to my TripWire reports after that. I cleaned up by deleting individual files ( I saved and examined them with various tools.)
I had a customer in the VoIP / Calling Card area who was using a server provided as surplus by someone who feed him overseas calls, and he kept losing his overseas termination contracts and he suspected this dude had put a backdoor on the server and was finding them and cutting out the middle man, and he was right, a disguised sshd listening on a weird port. The customer decided to leave the backdoor open and pass only selected traffic through that box.
A different customer was traveling and from a university computer cluster ssh'd back into the server I built for him from a machine that had a trojaned sshd. The guy came in and trojaned / rootkitted that box, and followed my cron job of a key-based ssh login that scp'd the backups off that nightly back to one of my personal machines, and somehow upgraded priviledges to root. I discovered it because I happened to be looking at top when he started compiling some hacker tools on my box. He left bundles of php code in various directories hidden by having a name of three spaces and etc, the code was web scrapers of email addresses and phishing pages that looked like the login to various bank sites.
By this time I using integrit instead of tripwire, and on debian. I disconnected all the infected / suspected servers, booted them from various livecds, and scanned them for rootkits; then I wrote a few lines of perl that re-installed every installed debian package, overwiring files, and listed every file on the system that was not from a debian package. That was how I found the directories with 3 spaces for a name and etc.
That guy kept coming back for 6 months or so -- every couple of weeks, there would be a single login attempt on some server I owned, using the distinctive username that was part of the cronjob backup. I would track down the ISP or administrator of the IP he came from and report that the box was probably hacked, I was mostly ignored but I did get a few notes of thanks and I did have a conversation with some college kid running his frat's web site on how to keep his linux secure.
Several times I have helped people who used one of the cheaper web hosting places clean out little snippets of php code out of all their files, that were probably put there for SEO / scamming purposes by an employee of the hosting company or a customer of them that had figured out how to escalate privileges. I think that is actually pretty common, but most people never notice because they only upload files, to their site, they never download and their integrity months later.
There are probably a few that I have forgotten too. I have a usb thumb drive I keep a collection of rootkits, phishing code, etc gleaned from break-ins of my computers or obtained from friends.
I had two or three compromises of client machines that all fell into the catagory of a Redhat 9 box with the original, un-updated, vulnerable samba running; the machines were never intended to be connected to the internet, but were connected briefly to dialup allow remote support. In those cases we wiped the boxes clean.
Yes. Many years ago I ran my own Windows Server network out of my home office -- Exchange, SQL Server, SMS, etc. It was all part of learning how to use the tools and staying on top of the technology. At the time I had a T-1 leased to the office, which gave me tremendous speed (relative to everybody else in town, that is)
The problem with this plan was that Microsoft is a place full of really smart people making really complicated software. Each of these server programs was it's own world of consultants and gotchas -- it was too much to manage while doing other "real" work. Patching and tweaking alone could have took all of my time.
I guess it took about a month for me to figure out that my Exchange Server had be pwned and was sending out spam email for somebody with a Russian IP address. I figured it out when I went to the box for something else and realized the CPU was pinging out at 100%. A little digging around and wham! Guess what? You've been punked.
I played around with cleaning it up for a couple of days but it just wasn't worth the hassle. I learned a lot about Exchange Server databases in the process, but I have no desire to be an email technician. And with Microsoft, everything is tied into everything else: it's never just one complex product, it's fifty.
So I shut the entire thing down. Swore that if I needed server or storage stuff I'd let somebody else worry about the details. It was one of the smartest things I've done as far as network administration -- understanding my own constraints. Now if I want Microsoft Server programs, I just rent them off the web. But I've found that, with the exception of SQL Server and IIS, most of the MS Server products are simply too overblown to make it worthwhile for an advanced hobbiest to play around with.
As a humorous side note, the next New Year's Day I got an email from some Russian guy who wished me the best for the coming year. We exchanged email pleasantries on New Year's for a few years after that. I always wondered what happened to him. I was hoping for jail, but he's probably got a villa and big ship in the Med somewhere.
Are you sure your Exchange server had actually been hacked, and wasn't just relaying? That happened to me too, ages ago, but I wouldn't call it a hack - just a bad misconfiguration, got me blacklisted at the time. In fact it was so long ago I can't even remember if it was called Exchange or something else; this is NT 3.51.
Ah, NT, how I don't miss thee. Although, to be fair, I hear the MS toolchain has improved greatly in recent years.
I ended up just getting a new server, as I couldn't track down all the things he'd done. I started learning about server security. I reduced the number of people who had ssh accounts, and installed stronger password rules. I use iptables to block brute force SSH attempts. I use logcheck, chkrootkit, and other tools to constantly monitor my server.