by
Only rarely do companies allow us a look inside their organization while they are recovering from a ransomware attack. Many find it more convenient to keep a low profile or to be secretive.
A positive exception to this is found in the Dutch managed service provider (MSP) VelzArt, one of the many unfortunate victims of Friday’s enormous, cascading supply-chain attack on Kaseya. The attack used a zero-day vulnerability to create a malicious Kasaya VSA update, which spread REvil ransomware to some of the MSPs that use it, and then on to the customers of those MSPs.
Instead of avoiding the limelight, VelzArt has blogged meticulously since Friday about how it and its customers were affected, and the steps it has taken to get them up and running.
VelzArt offers its customers a broad spectrum of ICT solutions, delivered using remote administration tools. One of those tools is Kaseya VSA. The company writes that it was in the process of switching to another remote administration platform at the time of the attack, but Kaseya software was still installed on some customers’ systems. Since Friday it has been working to recover those customers.
Here are five lessons we can all learn about recovering from ransomware, thanks to VelzArt’s admirable transparency.
1. Know when to communicate
Communication is key during times of crisis. VelzArt writes that after learning about the ongoing attacks in the evening of July 2, it immediately informed the customers it managed using Kaseya software by mail, phone, and newsletters. It also started the blog that became the basis for this article.
This open communication allowed it to triage more effectively. A production company that works 24/7 needs their servers more urgently in the weekend than a law firm that needs everything ready by Monday morning.
During the evening and night, VelzArt says it limited its customer contact to email, in order to prioritize actually getting the recovery procedures done. While it is understandable that anxious customers want to be kept informed, there has to be time for actually getting the work done.
2. Backups take time
Recovering from a ransomware attack normally means rebuilding everything from backups. And that makes backups a target for ransomware.
VelzArt writes that on most servers and some of the workstations, it was able to restore from backups without any major problems. However, stopping the attackers getting to the backups is only half the battle. Machines that have been attacked by ransomware may be harbouring other malware, so backups need to be loaded on to a clean machine, and that takes time: Restoring backups is not a quick fix.
VelzArt says that the servers that were taken offline to stop the attack had to be picked up from clients, checked, reinstalled, and then made ready for normal operations. The company writes that it took quite some effort to pull that off, with staff working in teams through the night. Extra power circuits had to be set up to handle the extra demand.
The company expected 70 percent of servers to be restored by the start of Tuesday. On Tuesday, they hoped to get started with gathering all the workstations that had a back-up option, and Wednesday would be the day to get the re-installed workstations back into their operational status, meaning they would get the necessary software installed, and connected to the network.
3. Help can come from unexpected places
Recovering from ransomware doesn’t just take time, it takes people, too. If you are recovering from a ransomware attack there is a good chance that you will need external help.
VelzArt writes that it worked with one customer on a trial for self-remediation. Because of a lack of information from Kaseya it was not sure how much work would be needed for every individual workstation. The company hoped that the trial would produce a method they could use with other customers.
On Sunday afternoon they asked customers to turn on their workstations, without logging in, stating that they found an automated way to restore workstations remotely. The activated workstations gave VelzArt an idea of what the impact of the attack had been. A warning was given to the customers that the reset procedure for the affected workstations would result in a total loss of local data and installed software. Basically, the system would be flattened and reinstalled.
By Monday morning the automated script it had worked out with the help of one willing company was ready to help the others.
VelzArt also noted that friendly competitors had offered to help them resolve the situation. An offer they were happy to tell their customers they had accepted, in order to speed up the recovery.
Insight from another victim
VelzArt’s unusual level of communication provides us with a rare insight in what a company has to go through when they’re recovering from a ransomware attack. Their transparency will help other victims and we wish them luck on a speedy recovery.
Although rare, there are other organizations that have gone public with the details of what it takes to recover from a ransomware attack.
