Search

Mass exploitation of ESXi hosts

What happened

On Friday February 3, 2023, news broke about an attack that targets VMware ESXi hosts.

VMware ESXi is a hypervisor that allows its users to run multiple virtual computers on a single piece of hardware. VMware offers the following definition of a hypervisor: “A hypervisor, also known as a virtual machine monitor or VMM, is software that creates and runs virtual machines (VMs). A hypervisor allows one host computer to support multiple guest VMs by virtually sharing its resources, such as memory and processing.”1.

VMware ESXi has become popular among IT administrators as it offers similar functionality as the enterprise grade ESX vCenter with some features disabled. One contributing factor to why ESXi is so popular is that VMware offer it as a free download.

The apparent objective of the attackers is to encrypt the virtual machines hosted on the impacted ESXi hypervisor as part of a ransomware attack. Initial reports came from the French CERT2 as well as commercial hosting and cloud service provider OVHcloud3. Later OVHcloud published an update to their initial blog post claiming that impacted ESXi hosts are bare metal hosts that are managed directly by clients and that managed ESXi hosts were unaffected. These reports indicated that attackers are targeting the exposed OpenSLP service, on port 427, remotely. The French CERT pointed to two potential vulnerabilities, CVE-2020-3992 and CVE-2021-21974, that are possibly exploited by attackers.

The attack was dubbed ‘ESXiArgs’4 based on the ESXi hosts being targeted and features of the malware that invoked several arguments on the impacted host, as well as appending the ‘.args’ file extension to encrypted files. On Monday February 6, 2023, VMware added that they have not yet seen any evidence that this attack is related to an “unknown vulnerability” or a zero-day attack. Until proven otherwise, the more likely theory is that existing weaknesses are being targeted. By Monday February 6, more than 3200 potentially vulnerable hosts could be identified in search results returned from Censys and Shodan.

VMware have issued several security updates for the OpenSLP service in the past. VMware issued official updates in late 20205 and early 20216 for CVE-2020-3992 and CVE-2021-21974 respectively. VMware has also indicated that this service is disabled in the default ESXi configuration form ESXi version 7.0 U2c and ESXi version 8.0 GA7 onwards. It is unclear if older versions had OpenSLP enabled by default and if this service then could be automatically exposed to the Internet based on normal network configuration found at hosting providers such as OVHcloud and others.

Proof-of-concept (PoC) exploits are also readily available for both vulnerabilities8 9 and have been available within months of the official vulnerability becoming public knowledge.

CISA’s Known Exploited Vulnerabilities10 (KEV) catalog lists CVE-2020-3992 with the entry dated November 3, 2021. Federal Civilian Executive Branch (FCEB) agencies were given until May 3, 2022, to patch vulnerable ESXi hosts. There is no entry for CVE-2021-21974 at the time of writing. There is, however, another critical OpenSLP vulnerability listed for ESXi, CVE-2019-554411, that was entered into the KEV catalog on the same day. No publicly available PoC code could be found that was tied directly to CVE-2019-5544, but since OpenSLP is open source, one could examine the code changes associated with the vulnerability and understand how it might be exploited.

A Rapid7 blog post12 from November 11, 2020, referenced a Kevin Beaumont tweet that alerted the security community about active exploitation of ESXi vulnerabilities CVE-2019-5544 and CVE-2020-3992. The blog post also quoted Kevin Beaumont’s almost prophetic words “[r]ansomware groups …bypass[ing] all Windows OS security, …shutting down VMs and encrypting the VMDK’s directly on hypervisor”. It also seems that the original fix for CVE-2020-3992 released in October 2020 was incomplete. But the November 2020 patch seems to have fully addressed the original vulnerability14.

If CVE-2020-3992, CVE-2021-21974, and possibly CVE-2019-5544 were being exploited, then it seems that it is the first time anyone has exploited these at such a public scale. These attacks affected the availability of virtual machines on the impacted hosts, thus the attackers announced themselves very loudly. These attacks may have gone under the radar if the attackers were stealthier, for example if the attackers only exfiltrated data and did not halt and encrypt virtual machines.

In a World Watch Advisory, the Orange Cyberdefense CERT also highlighted another potential VMware ESXi vulnerability, CVE-2022-3169616, that was fixed along with CVE-2022-31697, CVE-2022-31698, CVE-2022-31699 in December 2022. The former, CVE-2022-31696, could possibly allow an attacker with local access to escape the sandbox feature designed to partition virtual hosts off from the underlying hypervisor. Escaping the sandbox could potentially allow the local attacker to perform privileged actions. It is unclear if the security fix released by VMware in December 2022 was being targeted. None of these vulnerabilities are listed in CISA’s KEV catalog at the time of writing. The OCD CERT noted that to exploit this vulnerability an attacker must already have local access to ESXi. This would require a multi-step exploit chain if the attack would have been launched remotely over the Internet. This makes mass exploitation in this noisy manner rather unlikely, as these types of chained exploits are not the hallmark of typical brutish ransomware crews.

           

Observations from the Vulnerability Operations Center

The Orange Cyberdefense Vulnerability Operations Center discovered 42 ESXi hosts that were vulnerable to CVE-2021-21974. These hosts were discovered between February 2021 and October 2022. This first 6 vulnerable hosts were discovered on February 28, 2021, this is four days after the NVD published date of February 24, 2021. The remaining 36 vulnerable hosts were discovered between November 2021 and October 2022. This means that some hosts were unpatched anywhere between 270 days and 590 days. This is assuming the hosts were patched immediately after discovery. This specific vulnerability impacted 7% of our VOC client base. The Orange Cyberdefense Ethical Hacking Team performed 122 assessments classified as either Internal, External, or Red Team from February 24, 2021, up to and including September 2022. This is the period that vulnerability CVE-2021-21974 was public knowledge and could have been encountered. A detailed blog post about the vulnerability appeared on May 25, 2021, meaning that if a vulnerable ESXi host was encountered and was in scope there existed the possibility that it might have been exploited using the exploit code published on GitHub17 18. At this point, with a full explanation and exploit code, 105 assessments could possibly have yielded a potential finding. Fortunately, none of the clients we assessed had findings with reference to this vulnerability.

As stated earlier, the reported attacks have a noticeable impact on the target system. The attacks resulted in files on the ESXi hypervisor being encrypted after the attacker halted any running virtual hosts. Also, instructions to pay 2 Bitcoins into the designated wallet to obtain the decryption keys were found in files on several exploited ESXi hosts during the first wave of attacks. What was odd was that no ransomware group was named in the instruction file.

One victim shared such a ransom note19 on Bleeping Computer’s forum. The odd thing is that this note ends with what looks like a possible hint: “SSH is turned on” and “Firewall is disabled”. VMware ESXi has an SSH service that can be enabled. Perhaps the attacker brute forced these hosts with known or weak credentials or exploited a weakness in OpenSSH? The reference to the disabled firewall could suggest that the attacker felt that, had a firewall been in place with appropriate configuration, then their attack might have been thwarted. Disabling or limiting remote access to any host exposed to the Internet is crucial as this greatly limits what attackers can do.

Another user of the Bleeping Computer forum posted on the same thread on February 8, 2023, stating that an ESXi host was compromised even though the “SLP service was turned off”20. If this is the case, then the OpenSLP vulnerability theory seems unlikely to be the only attack vector.

There are hints at further extortion in the ransomware note shared on Bleeping Computer’s forum post. In the days after the first wave of attacks, the modus operandi of the attackers changed. They updated their ransom note by removing their Bitcoin wallet address and instead listing a TOR Onion (darkweb) Site. Both versions of the ransom note threaten to share the stolen data if no ransom is paid. The first version lacked a typical negation site. The newer version of the ransom note with the TOR web address might indicate that the attackers are still developing their infrastructure to align with what has become the norm with typical Cyber Extortion negotiations.

There has been mention in some reports of a new ransomware group called ‘Nevada’, but no concrete evidence linking the mass exploitation of ESXi and the group has been presented yet. We do know, however, that Cyber Extortion groups evolve over time. Some fade out, while others morph into new groups.

In Q4 of 2022, we actually saw a relatively low number of threat actor groups extorting victim organizations around the world - a total of 19 threat actor groups victimizing 494 organizations. The last time we counted under 20 active unique threat actor groups was in the beginning of 2021 – almost 2 years ago. Nevertheless, the number of victims increased during Q4, specifically during December 2022. During December, two new threat actor groups ’Play’ and ‘Royal’ were added to our monitoring process, contributing to the sudden increase.

 

Potential Threat Actors involved

The group ‘ViceSociety’ also changed its TOR onion address during this period, which led to 19 victims being noted on the same day, so the data might not represent the actual date and time of the postings of these victims.

In mid-December, multiple sources reported on another new ransomware variant called ‘Royal’, which surfaced in November in our dataset but is said to have been active already in early 2022. It is believed that some of the ex-Conti members are running this Cy-X operation. In November and December, we registered 51 businesses that have fallen victim to this group. Victims originated from countries such as U.S. (59%), Canada (8%), Brazil (6%), Germany (6%) and Austria (4%), showing ‘the usual’ mix of victim countries.

While the total number of victims increased again slightly during Q4, we observe a shift of threat actors contributing to this threat. This is not unusual given the opportunistic nature of this ecosystem While some threat actor groups might cease operations, others are ready to ‘take on their share’ of victims.

Looking at the top 20 countries impacted by this threat in Q4, more than half of all victims are headquartered in the U.S. In Q4 we saw that U.S. based victims rose to 51% of the total number of victims compared from 40% in Q3 2022. The second most impacted country during Q4 was Canada with victims from verticals such as Manufacturing (n=7), Information (n=4) and Wholesale Trade (n=3).

 

The geographic perspective

As can be seen in the chart above, English-speaking countries were the most impacted countries in Q4, closely followed by victims from Germany (n=21), Brazil (n=17) and France (n=17). French victims have decreased by almost half from Q3 to Q4. One reason can be that LockBit, who caused over 80% of the French victims in Q3, has had less activity during Q4. While Brazil is proportionally in the fifth position. In Q4, and especially during December 2022, we registered the highest number of organizations from Brazil falling victim to Cyber Extortion.

Returning to the ESXiArgs malware. On a technical level it seemed at first that the encryption technique used by the attackers is cryptographically sound, but the extent to which the attackers encrypted the files could potentially result in data being recovered. A guide22 was published by Turkish researchers to help with the possible recovery of virtual hosts. The guide does assume a high level of skill and experience with VMware and is not something a novice might be able to have success with. CISA have published resources to help with the recovery of data and should be easier to execute than the guide23 24.

However, it seems that the attackers realized that victims could recover their data and released a new version of the ransomware that encrypts up to 50% of the data if the VMware data file is larger than 128MB25 . The reason for the 50% encryption is that the data files representing the virtual disk of the virtual computes can be gigabytes if not terabytes in size. Encrypting this can take very long. So, encrypting the disks selectively still renders the virtual discs unusable, but also leaves some of the data intact. It’s a matter of luck whether the unencrypted data is usable.

The updated version may render recovery per the guide and CISA tool less likely to be successful.

Conclusion

In conclusion. VMware and others have strongly urged users to install the latest security fixes for their supported ESXi hosts. If this is not possible, then at least disable the OpenSLP service and limit access to ESXi hosts and required services. Some versions of ESXi are no longer supported by VMware and thus no security fixes are available for known exploited vulnerabilities. Make backups of data as this can make restoring systems much easier. Ensure that backups are performed regularly and that backups are complete.

For more information on ransomware and how to approach this looming threat, read our whitepaper we published on the topic26.

Sources

 

 

Incident Response Hotline

Facing cyber incidents right now?

Contact our 24/7/365 world wide service incident response hotline!