Helpdesk Support

Official Supportmonk Company Blog!

Former Hostgator employee arrested, charged with rooting 2,700 servers. How to ensure security of servers when root access is granted to employees?

April 22, 2013 , 5:44 pm

I was reading through the article where 2700 servers of Hostgator were rooted by one of its former employee.

Its relatively easy to secure servers when the intruders are from the outside but what do you do when the hacker happens to be one of your previous employees who had unfettered root access to your servers?
How do you prevent them from installing root-kits and back doors?

How do you keep track of all the commands they run on the server when they have root privileges?

How do you keep the ssh private keys secure and ensure that it does not fall into the wrong hands?

How do you keep the servers root password secure when you have to give out the root password to the employees?

How often do you change the passwords when an employee leaves the company?

All the above are important questions which needs to be addressed if you are hosting company owner or a system administrator and you are dead serious about protecting your servers from a compromise from within. I would say spare no efforts as its your business, your bread and butter.

I am wondering what procedures hosting companies follow and security systems that are used to protect the servers when the attacker happens to be an insider or an former employee?

It would be worse when the hosting company outsources the helpdesk support to some company in Philippines/India or when the employee is allowed to telecommute on work. I am sure most of you would be having some kind of procedures to follow and systems installed.

Here is what we do to mitigate the insider attacks and keep the servers bullet proof.

We have all our techs to log into an intermediate ssh gateway servers that has a software called ezeelogin( installed. Once Ezeelogin is installed, we would create user account for each employee that needs root access to clients servers. This how the ssh gateway interface looks like for the employees accessing servers.

ssh gateway server list

ssh gateway backend interface

SSh access to the interface is allowed from the remote ips that we specifically grant and on the destination servers we make sure that the ssh traffic is allowed only from the gateway server using iptables. The ssh gateway software, logs all ssh sessions just like the way a key logger does and it also saves the output of any commands executed. At the end of week we would be reviewing the logs as it help us improve quality, security and brings in accountability. The private keys stored is encrypted by the software and so even if somebody manages to get their hands on the keys they would still have to decrypt them first which is almost impossible. Also, since the authentication is via keys, we really do not have to give out the root passwords and control panel is also one click access since the software fetches its from the encrypted db and even if we do have to give out the root credentials, it has a feature that lets you reset the root password at the click of a mouse and an automated cron that lets you reset password on a regular interval.

Also, we do ensure that ssh access to the fleet of server is limited from the ips of the ssh gateway server alone. Also, Ezeelogin comes with a smart access control so that we can decide which employee gets access to which servers. It also comes with google 2 factor authentication integrated so as to minimize the impact in the event of an employees desktop being compromised.

two factor authentication in ssh

google two factor authentication in ssh

The battle to ensure that the servers are secure is a never ending ones as new challenges keep cropping up everyday, especially when the threat is internal but software such as ezeelogin should help to minimize such occurrences. The secret in winning the battle would be to never let your guard down by incorporating the latest security software’s that the industry has to offer and keeping abreast with the latest security measures.

Outsourced Customer Support