daily online news

With the help of security experts, we reconstruct a typical hack attack on two large organisations and walk through the steps that the head of IT should follow in such a case.

Monday, 9am
Blackjack, a hacker working from an internet cafe in London, is about to launch an attack on a major government agency. His aim is to cause maximum disruption and embarrassment. And, according to security experts, his job is going to be worryingly easy.

"Most organisations have dozens of vulnerabilities they haven't patched, or aren't even aware of," said Toralv Dirro, a security strategist with McAfee. "Even if a penetration-testing service says you're not vulnerable, that only means they haven't found a vulnerability, not that one doesn't exist."

Blackjack has spent weeks researching his target, identifying names of employees, partners and current projects. He has identified a potential way into the network through People Inc, a staffing agency that provides temporary workers to the public sector and which has direct links to the government agency's website and HR database.

Using basic tools such as Arin and InterNIC, Blackjack is able to identify the IP address of People Inc's web server and database server, and knows what applications are running. Using a range of cracking tools, he also knows how packets move on the company's network, so he is able to establish connections to open ports without being detected. Finally, he uses a simple SQL injection or cross-scripting technique to gain access to the web server.

This is a relatively common and simple hacking technique, explained Rhodri Davies, a technical architect with security specialist Firestorm. "Basically, the attacker uses the existing interface but, rather than entering information, they write a command for the back-end database," he said. "For example, rather than entering a username, you command the database to send back a list of usernames and passwords."

Monday, 12pm
By lunchtime, the hacker has gained access to People Inc's web server, and he is able to access usernames and passwords which will gain him full network access. At this stage, there's a good chance People Inc won't even notice that its systems have been compromised, added Lee Dawson, a penetration tester with ethical hacking company dns. "Most corporates spend money on firewalls and intrusion-detection systems but they don't do anything to prevent web attacks," he said. "Very few have application layer gateways, so attacks on websites are very easy to miss."

With full network access, Blackjack is able to easily identify the connection to the government agency and quickly access the server that runs its payment website. Within minutes, he has identified an open port and used it to access the payment server, where he runs a malicious script to trick the server into revealing the usernames, passwords and payment histories of thousands of users.

Monday, 3pm
The government agency has more sophisticated firewalls and intrusion-detection systems than People Inc, and the IT security team are alerted by a series of odd characters and sequences that something is happening on the web server. However, it's not immediately apparent that...

...the system has been compromised, and no alerts have yet been triggered. "When you go in as an attacker during a penetration test, it's relatively easy to be undetected, providing you don't start changing or deleting data," said Ravid Zavlinsky, of web-security specialist Applicure. "Many clients we speak to think their sites are impenetrable, [while] we find that they're being accessed every one or two minutes in fact."

Monday, 4pm
The hacker's next step is to create a new administrator account on the server — he knows the likelihood of this account being deleted in future is slim, even if staff don't recognise the name associated with the account. With this account, he copies some data from a database table and deletes a series of security profiles — eventually triggering the government agency's IT security systems.

When the IT manager on duty sees the alert from both the firewall and intrusion-detection system, he realises an attack may be in progress. He immediately asks for copies of any logs that are available, including the web server, database, firewall and intrusion-detection system. These logs should reveal what requests were made to the database and web server, and help to locate any attack scripts. This job should be overseen by a security specialist, as it can be hard to locate scripts created by an experienced hacker.

Completing this audit before disconnecting any machine from the network is absolutely vital, according to the experts. "If you panic and simply pull the plug out of the wall, you will lose a good deal of evidence because of volatile memory footprints," said Jose Nazario, senior security researcher with Arbor Networks. "You also lose any track a hacker may have left of what they have accessed or changed, and it makes it much harder to see what they are doing if there are no active processes running."

Moreover, if you don't know what's happened, there is no guarantee the hacker isn't still hiding on the network, observed Brian Chess, chief scientist with security company Fortify Software.

Back at the government agency, the logs reveal that the agency's web server has been compromised and, more seriously, the database server holding web-payment records and user IDs for thousands of citizens has been accessed and some records deleted. The user IDs and passwords of the database administrators have been accessed, and running an anti-rootkit application reveals that a number of rootkits have been installed on the database server.

Monday, 4.30pm
Now that he has a clearer idea of what has been compromised, the head of IT calls the chief information officer, who immediately informs the chief executive of the security breach. The chief information officer also informs the police, who contact the Serious Organised Crime Agency (SOCA), and government security agencies of the attack. "Informing the authorities is often a regulatory requirement, but it's also important to give them the chance to help identify and track down the culprits," said Chess. The chief information officer and the chief executive decide to delay issuing a public statement until there is more information about the security breach itself.

It is important to inform the authorities early on in an attack situation, before all evidence that might trap the hacker...

...is lost, said Chess. "Although it's often a very slight chance that a hacker could be identified and caught, it's important to warn the police before you do anything that tips off the hacker you know about him. It's also important they can come in before the audit trail is lost."

Monday, 5pm
The head of IT meets the chief information officer and IT security specialists to devise a plan of action. The agency has a well-established policy for dealing with security breaches, which means staff are less likely to panic and make poor decisions in a crisis. To ensure that your IT team doesn't panic during the immediate aftermath of an attack, ensure there are written policies detailing the procedure to be followed. "There should be clear rules to guide the team, and rule number one is never unplug the server until you know what you are dealing with," Chess said.

While the action plan is being formulated, the agency uses an automated tool to block the IP address of the hacker. An automated tool is often the best option in this situation, since, as soon as you block a hacker's IP address, he will often switch to another IP address within seconds.

Monday, 5.30pm
One team of IT staff is dispatched to examine the web server, the server is temporarily taken offline, and a "closed for essential maintenance page" is posted on the website. The clean-up team runs a full antivirus and anti-rootkit check on the web server and uses the previous day's backup disk to verify that nothing on the server has been deleted or amended.

The head of IT realises that just rebooting the system isn't an option, since it would compromise evidence but could also prevent the clean-up team from easily seeing what has been done while the machine was running.

Auditing the logs and examining the requests made to the server reveals that the hacker exploited a known vulnerability in the server operating system to gain access to the system, and this weakness is immediately patched.

The clean-up team also ensures that all other patches to the server have been applied, and that there are no other known vulnerabilities that could result in the server being attacked again. It's important not to apply patches or make any system changes until this investigation is complete, said Zavlinsky. "Making untested or ad-hoc changes to an application could just add further vulnerabilities."

This is a real danger if a hacker is determined to target your organisation. "The ultimate goal for many hackers is to take a server offline within minutes of it going back online," added Zavlinsky. "So it is vital not to reboot a server unless you are absolutely confident you know what the problem was and that it has been fixed. Otherwise, you'll be up and down for three days trying to sort it all out."

At the same time, a second team is examining the compromised database server. The database is relatively complex and has links into multiple other agencies and systems. After examining the logs and audits, the investigation team realises the hacker has leveraged a SQL injection and created a false administrator login to the machine. From here, the hacker accessed a number of user IDs and passwords, then moved on to look at financial records and security documents, and deleted some records altogether.

"This isn't a great situation to be in," said Chess. "They've opened the back door but now they've also got a key to the front door as well." The priority is to discover which records have been accessed and to compare the database with the most recent backup to look for changes or deletions. Since it is possible that credit-card and direct-debit details may have been accessed, the chief information officer immediately informs the relevant banks of the potential breach.

Monday, 8pm
A subsequent scan of the server for malware or other scripts shows a number of scripts and programs have been added, including rootkits that could be used to further compromise the network. At this point, the head of IT makes the decision that cleaning up the machine will be too time-consuming, and...

...decides to simply install a new server in place of the affected machine. "Sometimes that's the best decision to make, because you'll spend weeks sorting out the infected machine and you'll never be sure you've got everything," said Chess.

Monday, 10pm
The next task is to update all the organisation's security patches and run a complete check for malware, viruses or other attack scripts. The firewall and intrusion-detection system are updated, and a team of forensic computing experts arrive from a specialist consultancy to begin checking the logs and audits to see if any other systems have been altered or accessed. This team will be able to spot modifications or potential attack scripts far more easily than a general IT specialist.

Tuesday, 8am
After overnight testing and a comprehensive patch update, the web server goes back online, along with the new database server. Although there seem to be no problems, the systems and all logs are closely monitored for any sign of attack or irregular behaviour. The team are also closely monitoring the internet for any sign that the credit-card numbers accessed have been posted online, or offered for sale.

As an added precaution, all payment facilities are unavailable during the following 24 hours, until the team is certain the new servers are functioning properly and are fully secure.

Tuesday, 8.30am
Once the systems are back online, proceedings in the aftermath of the attack begin. First, the chief information officer meets with the chief executive and other senior government officials to discuss what information was taken, who needs to be notified of the attack and how the organisation might be affected. "This is the point in time where non-IT executives should be actively involved," said Chess. "It's not necessary to wait until the clean-up is finished, but you need to be certain that you have information to give them on these vital questions."

In this case, since the information accessed includes payment records (including credit-card numbers), it is decided that the agency should issue a public statement explaining what information has been accessed and what the potential consequences of this might be. Consumers are instructed that they will be issued with new user IDs and passwords for the website, and banks will cancel and reissue credit cards for consumers affected by the breach.

This seems like a lot of hassle, but security experts argue it's better than trying to bury the bad news. "Telling people the bad news is tricky but, if you want to retain their trust, it makes good sense," said Nazario. "It also makes sense because, if people are aware of the danger, it limits the scope of the damage, by putting them on their guard."

And finally...
Once these initial response steps are completed, the experts said anyone recovering from a hacking attack should:

• Review the attack with relevant IT staff: what happened; how can we be sure it won't happen again?
• Ask what vulnerability was exploited and what others might exist that are unknown
• Audit the IT security systems to see whether you need new firewall/intrusion-detection/antivirus technologies, or whether an application layer security device would prove worthwhile
• Ensure that your policies and procedures are kept updated, and incorporate any lessons learned from this attack into a policy for future incidents