Server crash, possibly due to hacking attempt
This morning, I couldn’t reach my web server. Dead. Nothing.
I contacted my server company, who very quickly diagnosed that the server had blue-screened; they rebooted it and all is now well. Looking at my web logs, I think it was down for about an hour – not disastrous, but definitely inconvenient and not A Good Thing. (Note to self: set up a monitoring service so I know when it’s down!)
So I started looking at the system logs to see if I could find what caused it to crash. And there was a big clue, right in front of me.
Yesterday, I installed SSH on the server (see my previous post on deploying via Git). And, 40 minutes later, I started getting login attempts (from Germany, Poland, Thailand, etc.) – all trying to login to SSH as “root”. Over the course of 15 hours, I logged over 4200 failed SSH login attempts.
The attempts stopped several hours before the server crashed, but I can only assume the events are related. I think I’ve go it more locked down now…
What I’ve learned…
Of course, these tips apply to most areas of security, but they do bear repeating!
1. Don’t use obvious login names
You log in with a username and password. If you have a default username, such as “root”, the hacker already knows one half of the the combination. (I didn’t have “root” as a username, so all their attempts failed at the first hurdle.)
2. Use non-trivial passwords
If you have a simple password, it’s that much easier to brute-force a login. If you don’t already, get yourself a password manager (I use 1Password) to generate and store unique, complex passwords. If you have a password made up of 16 random characters, it’s going to take a while to crack it.
Update: better still, as recommended in the comments, use key files to authenticate – and turn off pasword authentication completely. Not only is this more secure, it means you can log in without having to remember a password…
3. Don’t run it on the standard port
Hackers will try to access SSH on its standard port (22). Make it that much more obscure by using a custom port – for instance 2587 (you can use anything that doesn’t clash with other services). This is especially important if you are unable to implement tip 4 below.
4. Restrict access to your SSH port
Just because you’ve followed steps 1, 2 and 3, your server is still open to attacks. Hammering away at an SSH login can still affect the running and stability of your server. So, if it’s practical, use your firewall to allow access to the SSH port only to specified IP addresses. If they can’t get a response on a port, hackers will soon move on to somewhere they can.
I hope these tips are useful – and please let me know what I’ve missed or got wrong (I’m by no means a security expert!).
Comments
- Peter Boughton
-
Regarding #4, I've also heard of a technique where you block the SSH port to all IP addresses, and then have a different secret "unlock port" which temporarily lifts the block on the real SSH port.
So for anyone doing port-scanning, all the ports will appear invisible.
However, if you specifically send a message to, for example, port 1234 it will unlock the firewall for the real SSH port (which should still be random), but block it again as soon as an incorrect port is attempted.
So long as the key and SSH ports are NOT sequential (i.e. don't use 1234 and 1235 or whatever) then you've added an extra hurdle that may make it seem SSH is not running at all. (The odds of someone picking two random four digit numbers in the right order is much lower than just starting with 22 and increasing until it's found).
- 22 April 2011, 13:38
- Reply
- Seb Duggan
-
@Peter - that sounds interesting, but kind of complicated!
- 22 April 2011, 13:43
- Reply
- Andy Allan
-
A couple of other things.
1) Disallow remote root logins.
2) Use iptables (or your firewall equivalent) to set rules protecting you from repeated SSH attempts.
3) You can also specify the users that are allowed to SSH to the server.
- 22 April 2011, 13:49
- Reply
- Rob
-
Taking your (good) points in order:
1) Don't just not use root - disable root login - you can do this in your /etc/ssh/sshd.conf config file and it will mean they bounce off the server
2) Passwords are great - key files are even better and you can completely disable password based login. You can backup your private key to your mobile or USB key and passphrase protect it for double security - http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/pka-putty.html
3) This will normally be defeated by even the most basic port scan but will save you being picked up by some more basic bots
4) Yup - lock the port down using IPTables to trusted IPs - only time this causes problems is if you're away from home / work and need to connect.
Also consider running an auth watcher like DenyHosts (http://denyhosts.sourceforge.net/) this detects failed login attempts and blocks the attackers IP in /etc/hosts.deny. You can customise the thresholds and trust certain IPs ... it's great to watch the logs (my webserver gets some 10 - 20 per day!). It also offers synchronization of blocklists with other users to get preemptive.
Finally the technique mentioned by Peter is a basic form of Port Knocking - http://dotancohen.com/howto/portknocking.html - which is great provided you can a) set it up right and b) maintain access to a tool to perform the port knock :o)
- 22 April 2011, 13:55
- Reply
- Tom K
-
I Use IP Whitelists as mentioned above;
i.e block everything except port80 and then allow complete firewall bypassing by approved IPs only.
If i'm away, I can usually RDP into a machine which has permitted access.
Also recommend both monit (for restarting services remotely without SSHing in) and pingdom, which is amazingly good.
- 22 April 2011, 15:25
- Reply
- Adam Tuttle
-
As for monitoring, I've been using uptimerobot.com (free) for about a year now, for several sites, and I've been very pleased with the speed of alerts and the different options available. Not bad for a free service.
- 27 April 2011, 03:45
- Reply
- Seb Duggan
-
@Adam - I'll take a look at that one. I'm currently seeing what Pingdom has to offer...
- 27 April 2011, 08:06
- Reply