Expert insight from our cyber incident responder
Cyber attacks and data breaches are a matter of when, not if.
No single measure is 100% foolproof. A determined attacker will always be able to find their way around your defences, given enough time and resources.
Furthermore, as Vanessa Horton, our cyber incident responder, pointed out in an interview about anti-forensics:
The cyber world is changing all the time, which means we’re playing a bit of a cat-and-mouse game. Basically, as one side improves, so does the other.
In this interview, I pick her brain on cyber incident response more generally, gaining her expert insight into the ‘what’, ‘how’ and ‘why’, along with practical, real-life examples.
About Vanessa Horton
Vanessa holds a degree in computer forensics, as well as a number of cyber security and forensics qualifications.
She’s worked for the police as a digital forensics officer, where she was involved in complex crime cases. Vanessa was also awarded a Diamond Award and an Excellence in Service Delivery Award.
Now, she’s part of our cyber incident response team, helping clients with their cyber security requirements.
In this interview
Cyber incident response misconceptions
What common misconceptions do you see around cyber incident response? Or in your line of work generally?
The big one is the [misplaced] belief that ‘it’s not going to happen to us’. Many organisations believe they won’t suffer a cyber incident, even if other organisations might.
That’s a poor mindset to have because, for one, it’s wrong – everyone will suffer an incident at some point.
But more importantly, this mindset leaves organisations unprepared for when they do suffer that breach, and haven’t got an incident responder on hand to keep the damage to a minimum.
The fact that some big names have ‘been done’ recently shows how nobody’s safe. Not even big cyber threat actors! For example, not so long ago, LockBit [an infamous ransomware gang] got taken down. And various other ransomware groups, including Ragnar Locker and Black Basta, suffered the same fate.
It just goes to show that no one’s above being successfully attacked. That’s just the unfortunate reality.
But if threat actors are being brought down, doesn’t that improve the cyber landscape?
Oh, law enforcement doing its job is definitely a good thing.
However, when you bring one website or gang down, what always happens is that the remaining gang members form a new group.
Law enforcement never catches every single member of the gang when it takes one down. So, the ones who remain are going to take their skills and form new groups.
They might then change their objectives and go after new targets – because they’ll want to establish themselves as that new group.
What do you mean by ‘changing their objectives’?
So, for example, LockBit originally said it wasn’t going to go after hospitals for ethical reasons – perhaps ironic for a criminal gang, but there you go.
Anyway, LockBit changed, and is now going after hospitals.
This adds to my earlier point: you simply don’t know who’s going to be targeted next. And just because one group said it won’t target your industry, doesn’t mean other groups will offer the same courtesy. Plus, you have no way of knowing what any group’s next move will be.
This, by the way, is the type of conversation I often have with clients. It’s how I convince them that they really need that cyber incident response plan, to do tabletop exercises, to train their staff, and so on.
By being open and honest about these things, I can more easily show clients that taking these actions is in their own best interests.
Every organisation should prepare for an incident. They must actively take steps to protect themselves. Because if they don’t, well… We’re all human, and when faced with an unexpected situation like a security incident, you’re going to panic, no matter what.
But by being prepared, you can recover quicker, which makes the company financially better off, as you’ll suffer less disruption. You can also quickly stop the attacker from accessing any further data in your systems, so things don’t get any worse.
Plus, by handling an incident well, stakeholders and the public at large are generally more forgiving, which will obviously serve your reputation well.
Interviewer note: If you did everything you could, people are forgiving about breaches
Louise Brooks, head of consultancy at our sister company DQM GRC, made a similar observation in a recent interview about practical GDPR [General Data Protection Regulation] compliance:
“Organisations must remember that real, living people are behind the vast quantities of information they’re gathering and processing. Those people will be affected if anything goes wrong due to mismanagement of their data.
“However, people are generally open to forgiving organisations when things go wrong if the organisation can demonstrate they treated personal data with the respect that it deserves, and they did the best they can.”
Protection – first steps, simple measures
What are the first steps for organisations to protect themselves?
Before you do anything else, you need to know what you’re defending:
- What are your key assets among your:
- Where are those assets?
- What are the risks to them?
- What controls do you already have in place?
Risk assessment and management are critical starting points, as you need to know what you’re working with before you do anything else. But, as my colleague Andrew Pattison likes to say, it’s important to keep these simple.
The key is to identify your biggest risks, then implement appropriate controls to mitigate them.
What are some simple measures every organisation needs?
Very basic controls can get you a long way:
- Strong passwords and MFA [multifactor authentication]
- Anti-malware software
- Secure configuration
- Regular patching
- Firewalls
Not doing these types of things just makes you a more likely target. I don’t like using the word ‘easy’, but that’s what you’re making yourself if you don’t patch or you use passwords like ‘Password123’: an easy target. You’re leaving the door wide open to threat actors.
This also underlines the importance of prevention and detection. Cyber incident response planning doesn’t start with the response – any response is only triggered if you detect abnormalities.
Detection – security monitoring and what is ‘normal’?
How do you detect an anomaly or a suspicious event?
You need to understand your baseline: what’s normal? Because if you can’t answer that, how will you know what is suspicious?
Is it normal if someone logs in from a Russian IP address, for example? And at 1:00 am?
But you don’t need to have someone sit there and monitor all event logs all the time – a security monitoring solution can do that for you, like:
- An IPS [intrusion prevention system];
- An IDS [intrusion detection system]; and/or
- An EDR solution [endpoint detection and response].
You also want systems that log system activity and forward them to a centralised SIEM solution or SOC [security information and event management, security operations centre].
These types of technological solutions are essential to process huge amounts of information [security events like access logs]. But the human aspect is a vital part of detection, too.
For one, a person will have to ‘teach’ your technological solution what constitutes abnormal behaviour. And two, when your tool detects something suspicious, it must alert a human to follow up on it.
Interviewer note: Automating log analysis with AI
Earlier this year, I spoke to information security manager Adam Seamons about network security. Automated security monitoring tools cropped up, as did the role of AI and machine learning in security:
“AI and machine learning have both been used in detecting anomalies and suspicious patterns for some time, and will only continue to be used more. I expect SOCs to become increasingly reliant on AI.
“Getting more specific, log analysis is a key area for AI to automate. An AI tool could do the heavy lifting, sifting through tons of logs and data to detect and then respond to threats far faster than a human could.”
You gave a 1:00 am login as an example. Does a situation like that require an immediate follow-up?
Good question. Many organisations incorrectly believe that they don’t need 24-hour cover – or at least, don’t need to make someone responsible for responding to an out-of-hours alert from an automated tool.
But things are going to happen overnight. Look at it from the threat actor’s point of view. A smart attacker is going to attempt to breach your systems when they’re at their most vulnerable – i.e. when nobody’s looking.
Organisations are prone to forgetting about out-of-hours protection, but that’s precisely when you most need protection. That’s when you’re more likely to get attacked. Threat actors know that’s when defences tend to be down, and monitoring is slack.
What exactly should organisations be monitoring?
Security events. These are just everyday events on your systems or networks – logins, incoming emails, files received, and so on.
So, for example, if you suddenly get someone logging in from Russia, and they weren’t on a business trip there or something, you need to investigate. This means you’re not just tracking the logins themselves, but also certain information about them – locations and IP addresses, login times, files or services accessed, and so on.
Again, you have to know your baseline. What is expected activity? And if it’s unexpected, someone must quickly investigate. If a threat actor did gain access to a user’s account, you want to prevent them from accessing anything else.
Equally, if it’s just someone who’s on holiday in Russia and decided to log in, you can dismiss the security event. But you must establish that in your initial follow-up; you can’t just assume it’s not a security incident.
[The difference between a security event and a security incident is that an event is an everyday occurrence – like users logging in. But some events also signify a security incident: a breach of confidentiality, integrity and/or availability, also known as the ‘CIA triad’.]
To what extent does seasonality play a role in security monitoring? Like weekdays vs weekends, for example, or Black Friday?
Oh, seasonality plays a role, for sure. Black Friday is the perfect example.
If you’re a retailer, you’re going to see way more web traffic than usual. You must be able to handle that. By having more people on call, for example. By double-checking things. You need to plan ahead and assess the risks.
Again, a smart threat actor will target your weakest link, and when you’re at your most vulnerable. So, if they wanted to take your website down, for example, with a DoS attack [denial of service], a day like Black Friday is a prime time to target, when traffic is up anyway, and it’ll take fewer additional requests to flood your servers.
Black Friday is also a great time [from the attacker’s perspective] to cause most harm. If your website isn’t operating that day, you’ll take a massive financial hit in terms of lost sales.
Threat types and risk assessment
DoS attacks aside, what other cyber threats or attack types must organisations consider?
Take your pick:
- DoS
- Phishing
- Ransomware
- DNS poisoning
- Backdoor attacks
Ultimately, a lot of them involve malware, but threat actors can deliver it in many different ways. The more important thing to think about is where and when you’re most vulnerable. Black Friday is one example. The end of your financial year is another, when you’re dealing with tons of confidential information.
You’re not just looking at who might target you and how, but also when you’d suffer the biggest blow. When would the impact of a security incident be at its worst? [I.e. business impact analysis.]
And OK, not every attacker will think that way. But this question is one every organisation should consider, because you want to be operational during your most critical times. Even if the cause of a disruption wasn’t malicious, you don’t want it to happen – and especially not at a time when disruption would be costly.
How can organisations balance the cost of such assessments and measures against the risk of an incident or a disruption?
That’s always the challenge, especially for smaller organisations. Everyone needs to find that balance and make those difficult choices.
To help make them, ask questions like:
I think of this as a gap assessment, which can be against a standard like ISO 27001 or NIST SP 800-53. I’ve found that many organisations particularly like the NIST incident response guidance because it’s so accessible.
Interviewer note: The core principles behind cyber resilience and defence in depth
Vanessa covered the core ideas behind the three broad layers of cyber defence in depth:
- Prevention
- Detection
- Response
The core idea behind prevention is risk assessment. Identify your threats and weaknesses, and where and when your business would most suffer from a cyber incident, then implement measures to mitigate the risks as best you can. Concentrate on the more basic, cost-effective measures – like firewalls and patching – that prevent most common attacks.
The core idea behind detection is that preventive measures can fail. As Vanessa said to me: “They can only do so much.” You can think of the cyber security world as a ‘cat-and-mouse game’ in which the ‘cat’ (attacker) has the upper hand. Besides, there’ll always be zero-day exploits.
So, you want to become aware – as quickly as possible – of when your prevention failed. The key here is to know what ‘normal’ looks like, so you can identify abnormal behaviour, potentially indicating a cyber security incident.
Let’s get deeper into response: the follow-up to when your tools flag up an anomaly.
Cyber incident response plan
What’s the first step in planning your response?
Your cyber incident response plan. No doubt about it.
Incident response planning is such a vital part of having an effective response overall. As I said earlier, we’re human – panic is natural in unexpected situations. Even if you are prepared!
But preparing and documenting a solid response plan – an incident action plan, if you like – ensures that even when under pressure, people do the right things. They make all the right decisions, because they were made ahead of time, outside the heat of the moment.
A good security incident plan should also ensure that your approach is consistent, even if a key person is unavailable.
How can organisations further improve their cyber incident response plan?
Your incident response plan should include a different incident response playbook for different threats. Dealing with malware on your systems requires a different response than, for example, a DoS attack.
Also, test your plan. Tabletop exercises are important – they tell you whether your plans are working as intended. Plus, they make for valuable training for staff! It’s always much easier to do something if it feels familiar.
[Note: Vanessa gave an example of this below.]
Training – specialist skills and when to outsource
What training do staff with incident response roles or responsibilities need?
That depends on the individual’s role in the team.
Cyber incident response isn’t an IT issue, but a business issue requiring input from a wide range of stakeholders.
At a minimum, training should include:
- What constitutes an incident;
- Responsibilities during an incident response;
- Activities for ensuring compliance with legal requirements;
- How, when and to whom an incident should be escalated; and
- How to handle, store and process evidence in a forensically sound manner.
Is that a common issue? Not recognising something as a potential incident?
We do often see that. Different people in an organisation have different opinions on the definition of an incident.
In many situations, differing views are great. But in the context of incident response in cyber security, the organisation needs to ensure a consistent definition and approach, so staff training can cover it clearly.
You need to teach people how to recognise the abnormal stuff. And to put in a security incident report to someone adequately trained. Whoever triages must know both what to look for and how to handle it securely, to ensure the organisation is taking the right actions from the start.
When is it better to have or develop that technical expertise internally, and when is it better to outsource?
It depends on:
- Your organisational size;
- The complexity of your systems; and
- Your internal capabilities.
If you’re a smaller organisation, it’s unlikely you have the internal capability. More complex organisations would probably also benefit from outsourcing all or some of the capabilities – digital forensics, for example.
Using external expertise also means relying on people for whom responding to an incident is an everyday occurrence. They’ll know exactly what actions to take, and won’t allow their skills to become rusty. In fact, they make a point of keeping up with industry news and trends.
That’s completely different from an internal capacity, whose day to day likely involves other tasks. That makes them less comfortable dealing with an exceptional situation like a security breach.
What skills are specific for incident response? That an internal person is unlikely to use day to day?
Again, keeping up with, and having a deep understanding of, the latest threats. That gives them the ability to quickly detect and respond to these threats.
More specialist capabilities include:
- Digital forensics [discussed below];
- Malware analysis; and
- Threat hunting.
These often require expensive training and significant effort to keep up, as the latest approaches keep evolving.
In fact, due to cost, these are often – unfortunately – not conducted in cyber incident response.
This is a mistake. If you don’t understand what happened, how it happened and when it happened, it makes the incident response procedure much more challenging. It’s not enough to just get your business up and running again.
Do you have a real-world example?
We had a client that had been compromised with ransomware. They decided not to pay – which I think is best in most cases – and restored all their services without investigating the root cause.
They got done a second time just two weeks later, by the same threat actor, wiping everything out again. After which they contacted us.
Anyway, it makes the point: you must understand and remediate the initial vulnerabilities before declaring the incident response complete:
- Investigate how the attackers got into the environment.
- Check for back doors and persistence mechanisms, such as scheduled tasks, new users, new processes, and so on.
Ensure you’ve closed your vulnerabilities, or you’ll get done again. Possibly by the same attacker.
Ending up back where you started is hugely disruptive. Not only that – the financial and reputational impacts are massive, too. Journalists love writing about the ones attacked twice in a short period. Akumin at the end of last year is a good example.
Can you share any other real-world examples?
We had a client that kept getting phishing emails, with staff repeatedly falling for them, clicking the links and typing in their credentials. This suggested a fault with their staff training, and that their email filters weren’t robust enough.
In a scenario like that, you want to stop the threat actor gaining further access to anything, but also prevent this from happening over and over again.
It means figuring out the root cause. Getting an answer to why this keeps happening. Which means asking:
- Why are so many phishing emails coming through? Why aren’t they stopped?
- Why do staff keep falling for them?
Without addressing these, you’re going to keep suffering the same incident. That’s clearly bad for business – financially, operationally and reputationally.
So, a part of incident response resources should go towards resolving this incident, and part towards future prevention. But, of course, had this organisation invested in better training and filters to begin with, they wouldn’t have had to spend all that money after the fact.
Want to get future interviews – and other blogs – like this straight to your inbox? Subscribe to our free weekly newsletter: the Security Spotlight.
Common incident response errors
What’s the first step when responding to a cyber security incident?
Step one is always to confirm whether the incident is real, and if so, to determine the scope and potential impact.
It’s fine if you can’t answer all questions to start with – that’s normal. But as the investigation progresses, you must keep reevaluating your situation and risks.
You may also have to report the incident, often within just 72 hours, under legal obligations like:
- The GDPR [General Data Protection Regulation];
- The PCI DSS [Payment Card Industry Data Security Standard];
- The NIS Regulations [Network and Information Systems Regulations]; and/or
- DORA [Digital Operational Resilience Act].
What else should organisations do during the first stage of their response?
Isolate the compromised device and/or network segment from the rest of the environment to prevent further damage.
Also, if the device is already on, do not power it off.
I remember an exchange during a tabletop exercise that focused on digital forensics:
- Someone said: “I’ll just power off the machine.”
- I exclaimed: “No, no, no! Stop!”
- Everyone just stared at me. Someone asked, completely puzzled: “Why?”
- I explained that I couldn’t reveal the rest of the scenario to them at that stage, but did say that if they turned off that machine, they’d lose vital evidence.
They were about to lose critical evidence only stored in the RAM [the device’s memory]. Once you power off your machine, you permanently wipe that memory.
You see, some malware only runs in the memory, with very few artifacts stored on the hard drive. That limits what we can discover during any forensic investigation. Sometimes, the memory also stores ransomware decryption keys.
[However, if your device is already powered off, don’t switch it on. That makes changes to the system that could overwrite logs or other evidence.]
A lot of people think they’re doing the right thing by turning off their infected machine, when they should only disconnect it from the network. This comes back to training.
You’ve got to train people, particularly on those stages that happen before an external specialist would come on the scene. A key part of that is to not lose evidence. That’s something staff must be trained to do.
Turning off your machine is very instinctive. It’s the classic IT solution: rebooting your PC solves a lot of problems. But doing the right thing in this scenario isn’t difficult – you just have to know what the ‘right’ thing is. What are some other common errors that are easy to correct with basic awareness?
A common one is people simply deleting phishing emails. They just assume that IT already knows about it, or that someone else will report it.
But no: we all have a part to play in security, and clicking that ‘Report Message’ button in Outlook takes seconds – everyone’s got time to do that. And it may just save your organisation a lot of money.
Another thing is to keep your list of contacts and plans up to date. And to keep copies independent of your connected systems.
This might sound too obvious, but I’ve seen it so many times: people suffer an incident, they can’t get to their files, and then realise that they don’t know who to call, and haven’t got a way of easily finding out.
In short, keep a printed copy of your computer incident response plan, or save the contacts on your phone or something – and keep these copies up to date – so you can get in touch with the right people quickly, should you suffer an ICT disruption.
Digital forensics
Let’s come back to digital forensics. What does that involve?
A digital forensics investigation aims to answer questions like:
- How and when did the threat actors gain access?
- What did the threat actors do once they got into the IT environment?
Any digital forensic investigation should be conducted by someone who is trained to do so, because of the evidence aspect again. Any evidence an investigator uncovers might be used in future prosecutions.
The output of this investigation may also lead to follow-up actions such as malware analysis or threat hunting. That’s proactively looking for undetected threats in the network.
Incident response process
Walk me through the next incident response steps. What happens after you isolate the device and conduct a forensic investigation?
I should point out that device isolation isn’t the only thing you might need to do to contain the incident.
Maybe you need to isolate a critical service while the investigation is ongoing. Or you need to engage with third-party providers, such as Cloud service providers, which could give you access to vital logs or isolate the service for you – things like that.
Those initial stages, like the forensic analysis, will inform how you contain the situation and what the next steps should be.
What’s the next step in the incident response process, after containment?
Eradication and recovery. That can involve replacing or rebuilding compromised systems, which takes time and may impact legacy applications.
That process may include, but definitely isn’t limited to:
- Restoring and securing base systems;
- Restoring data from backups [following verification of integrity]; and
- Scanning for known vulnerabilities.
Also, we test any recovered systems for functionality and security before reintroducing them into the IT environment.
Once reintroduced, we monitor them for a period – ideally, at least a week – to make sure the threat actors haven’t returned.
The indicators of compromise discovered during the investigation help inform us what suspicious activity we’re monitoring for. For example:
- Specific network connections from suspicious IP addresses or domains;
- Running applications and related services; and
- Compromised user accounts.
What’s the next step in the incident response life cycle, after recovery?
Once you’re satisfied that you’ve fully remedied the situation, do a post-incident review:
- What happened?
- How did it happen?
- What was the impact?
- What went well in the response?
- What could have gone smoother?
- How can we improve for next time?
In short, establish and implement the lessons learned. This will only improve your ability to protect against, detect and respond to future cyber security incidents.
Should that sort of information also be shared externally?
Yes! Again, we all have a part to play when it comes to cyber security – not just within an organisation, but also in the wider community.
So, it’s important to contribute to threat intelligence. For example, take the ransomware attack on the British Library last year. It released a report to the public afterwards, which talks about what happened, how it happened and what the Library learned from it.
I love this sort of thing, because it shows so clearly that you’re taking cyber security [and cyber security incident response] seriously. You’re holding yourself accountable and being transparent.
Also, by publishing something like this, you’re helping other people. It shows that you’ve done your best.
Most people find it far easier to trust an organisation being open about what happened and what they’ve done about it, than an organisation trying to cover things up.
Don’t be like 23andMe, only admitting to things when forced to and blaming the victims to boot. It just makes you wonder what’s really going on, and question the company’s respect for its customers. PR is a big part of security incident management, too!
Find out how we can help
We take a pragmatic approach to assessing and managing your incident response needs. We align standards and best practices with your operational and business requirements.
Talk to us today to see how we can help you.
We hope you enjoyed this edition of our ‘Expert Insight’ series. We’ll be back soon, chatting to another expert within GRC International Group.
In the meantime, why not check out our previous interview with Vanessa on anti-forensics?
If you’d like to get our latest interviews and resources straight to your inbox, subscribe to our free Security Spotlight newsletter. Alternatively, explore our full index of interviews here.
