Handling a ransomware event
Just the word ransomware can generate fear in many organizations, and understandably so. We have heard the stories of organizations that have lost data forever and have had their operations disrupted for days or weeks. The impacts of a ransomware attack can be devastating.
The best options for handling ransomware remain in preparation and prevention.
- Don’t get it. Improve your protection and monitoring. Easily said, tough in practice.
- Ensure that you have backups of critical files in case you do get hit. This way, you can restore the data instead.
If you are able to follow the above, great! The enemy of the above, though, is typically organizational progress. Being able to protect, monitor and backup an unchanging environment is relatively easy. However, when new systems are turned up, they may be turned up without the proper protection, monitoring or backups. And over time, these unprotected system become critical to the organization, and unprotected.
So, even with the best preparation, ransomware can cause fear. Fear is very real, but fear during a crisis may lead to sub-optimal decisions. But, how can we control fear when faced with a ransomware event?
First, I would like to refer you to this article by
Josh Steimle, from January 4, 2016, posted on Forbes (https://www.forbes.com/sites/joshsteimle/2016/01/04/14-ways-to-conquer-fear/). I personally think it is on target. Allow me to extract 2 key tenants.
“1. Understand fear and embrace it. Fear exists to keep us safe. It is not inherently bad or good but a tool we can use to make better decisions. Fear isn’t designed to keep us inactive, but to help us act in ways that generate the results we need and want. Embrace fear as instruction and let it inform your actions, but not control them.” (bolding is mine)
And this tenant.
“5. Educate yourself. We are afraid of nothing so much as the unknown. If your fear is based on a lack of information, then get the information or knowledge you need to examine the situation based on facts rather than speculation.” (again, bolding mine)
Allow me to walk you through a hypothetical ransomware incident response.
Hypothetical event
One Saturday morning at 6am, I was contacted by the network team for a large retail company, name kept private, based in the northeast. (A typical tactic that I have seen is that ransomware is kicked off on a Friday night or Saturday morning. This is to give the encryption process time to run without people noticing…) This retailer makes their money loading trucks and delivering product. They have sophisticated computer systems to plan driver routes, to ensure the right product gets on the right trucks, and that the product is correctly delivered.
Ah, back to the 6am call. I was told that file encryption was observed on computers in the network. Further, the backups for the organization had been deleted, so no restoration was possible. I was asked to assist on this incident. So, by 7:15am, I was onsite. They wanted to stop the spread, but couldn’t. Every time a computer was turned on and logged into, it would be encrypted.
Important note, by the way. Turning on computers to install a product to deal with the incident at this time could be an AWFUL idea, especially if you aren’t sure how the malware is spreading or activating.
So, the business was concerned with getting operations back on line as soon as possible. Getting the systems on line by Saturday afternoon seemed unrealistic, would Sunday be possible?
In order to get the business back on line, we only needed to do the following:
- Stop the spread.
- Eliminate the malware or pay the ransom.
- Restore the systems.
Pretty simple in theory. Why is stopping the spread the number one task? Well, every system that gets infected is another point where the malware can spread from. This organization had offices in several states. The infection was isolated to the headquarters, it had not yet been observed in other locations. Keeping it that way would reduce the amount of recovery work.
Let’s take a moment to discuss the ransom. In this case, the ransom demand was for 1100 bitcoin. At that time, bitcoin was worth approximate $12,000. The company first thought, oh, $1.3 Million. Not great, but a consideration to get back on line. When I went through the math, explaining that it was actually $13.2 Million, the possibility of paying the ransom was off the table. Personally, I was pleased that the idea of paying was off the table. While I am always against paying, that isn’t my decision in these matters, it is the decision of the leadership of the organization. The reason I was pleased there would be no payment is that there is no guarantee that a payment will yield a working solution. And, the only thing that could make this event worse would be to pay the ransom and still have problems.
To stop the spread, we focused on learning how this ransomware was spreading. Before starting the examination, we shut off all internet connectivity, in case someone was logged into these computers from the Internet. Once the internet access was shut off, we then started to examine infected computers, to learn how were they getting infected. What we found was that the infection was planted in the login scripts. Basically, when a user would log into their computer, they would download and run the ransomware on their computer. Not great, but now we knew how it spread.
To stop the spread, we stood up a new server outside of the administrative domain. We could place the new computers on the same subnet as other systems that were infected, and these new computers did not get infected. This style of ransomware was blind to computers outside of the administrative domain. This was good news, because with this knowledge, the operations team was then able to start restoration activity while we continued to remediate the malware in the rest of the organization.
So, let’s recap.
- Turning computers on to install software would have actually caused the malware to spread faster.
- Shutting off network access is wise until you can determine how the malware is spreading.
- Once the spread of the malware could be controlled, the restoration efforts could begin. The malware did not need to be eliminated to start restoration, but controlled.
So, through this analysis, by Saturday afternoon, we had a plan in place to restore service safely, while working on malware remediation.
The operations systems were moved to a new administrative domain permanently, effectively isolating them from the user and the old infected systems. As for the systems with malware, there were two main options.
- Try to eradicate the malware from the systems that had the login scripts, known as the domain controllers.
- Turn down the old domain, as re-establish a new one.
In this case, option 2 was the option chosen, The reason why was that, during the analysis, we discovered that privileged accounts with weak passwords that were never changed were used in the attack. One of the accounts had a name similar to corpadmin, with an easy password that hadn’t been changed in at least 3 years.
So, remember, the goals were to get the order processing system back online and to get the internet access reestablished. We were on track to getting the order processing system online. However, restoring internet access was judged to be too risky until we knew that no back door accounts could be used to allow remote desktop connections back into this network. Remote desktop connection is a difficult protocol to secure, because enabling secure logins via remote desktop is very difficult.
The operations team did some heavy lifting through Saturday and Sunday, rebuilding the order processing systems. Remember, there were no valid backups. And, by Sunday, the order processing systems were online, and trucks were rolling. Even better, the nearly 100 warehouse employees were able to show to work, they did not lose wages.
Well, great, operations was back. The users were on track to get back. But, how did this happen? And, how could it be prevented from happening again. After experiencing an event like this, the organization leadership was rightfully concerned about wanting to be sure that it didn’t happen again. So, they wanted monitoring in place, monitoring that had been suggested in the past and discarded because it was too expensive. This reaction is quite understandable. But, monitoring on its own won’t prevent this from happening. Why?
For monitoring to be effective, important systems must be monitored. This can be accomplished after an incident, as it will have the focus of the organization to ensure that all systems have proper monitoring. But, recall, the organization got into this situation in the first place even with anti-virus, firewalls, and other protective technologies. If the protection didn’t work, it most likely did not work because it was not maintained. And the same fate awaits a monitoring solution, even adding a monitoring service. Not that the monitoring service would be bad, but the organization will grow. It will grow around the monitoring controls, making the monitoring less effective. And leaving the organization more exposed. And, starting up a monitoring service requires tuning, to eliminate false positives.
So, what is the right solution? I offer there is no “silver bullet”. The proper solution must include the following:
- Protective controls.
- Ensure accounts belong to users and aren’t just generic accounts.
- Protect account access with multi-factor authentication.
- Ensure users only have access to what they need.
- Separate the operations network from the user network. Even better, isolation functional networks wherever possible. Doing this will limit the spread of any potential ransomware.
- Ensure proxies, firewalls and anti-virus are regularly updated, as well as the operating systems.
- Detective controls
- Ensure that you have monitoring in place to respond to reports of computers infected with malware. Whenever a computer is infected with malware, it means that a protective control failed. Look into how this happened. It will help identify gaps in protection.
- Ensure that you monitor login access. If you see suspicious logins, found out if the accounts were compromised.
- Response
- Once a report of an incident is received, it is important to determine whether it is a false positive or real. If it is a false positive, log it. If it is real, then it is critical to under what the event was. For example, the discovery of credential stealing malware is a significant concern. It means account passwords are probably compromised, and it is important to reset the passwords of accounts at risk to restore security.
- Once an event happens, the response is not complete until normal processing has resumed. The goal of a response is to restore normal operations.
I hope that this helps you better understand what happens during a significant malware event. Please note that this is just a typical example for an encryption style ransomware event. The technique used in this story to isolate the ransomware worked because analysis went into understanding how the ransomware spread. Different ransomware attacks will require different methods of containment.