For the first time in over twelve months, our CERT has released a ‘Critical’ level World Watch Threat Advisory, regarding a vulnerability in a range of Ivanti products that are being actively exploited on a large scale.
Ivanti is a partner of ours and we sell and support several of their products. We have worked closely with Ivanti to mitigate the impact of these vulnerabilities on our customers and are confident that the risk was quickly addressed, with no impact on our customers. This blog post is being published in an effort to fully inform our clients and market about the details of the issue so that we are all in the best possible position to counter the threat.
Two new vulnerabilities, CVE-2024-21888 and CVE-2024-21893, were disclosed in addition to the previously disclosed vulnerabilities.
Ivanti released patches on January 31, 2024, for a limited number of impacted product versions that address all known vulnerabilities.
New XML mitigations were released to address all known vulnerabilities.
Ivanti released a new external Integrity Checking Tool (ICT) that can be used to detect compromised Ivanti devices.
Active exploitation of vulnerabilities continues.
Orange Cyberdefense has already contacted clients with Orange Cyberdefense managed or co-managed impacted Ivanti products to arrange agreed actions to either patch or mitigate devices.
As of January 31, 2024, Orange Cyberdefense scans show that of the 24,986 exposed responding Ivanti devices, 3,254 devices are vulnerable, 684 have been compromised, 20,674 devices have applied XML mitigations, 451 have XML mitigations but may have been compromised before the mitigations were applied, and 680 devices were leaking sensitive data through the logo/login.gif.
CISA orders federal agencies to disconnect all impacted Ivanti devices and to clean the devices before reinstating the devices.
Ivanti Server-Side Request Forgery vulnerability, CVE-2024-21893, is used to inject previously unknown and interesting backdoor.
Access to this backdoor is controlled through basic ‘API key’ mechanism.
Attacker performs commands injection with high privileges.
Orange Cyberdefense continues to monitor, investigate, and report on activity that is associated with exploitation of the recently disclosed Ivanti vulnerabilities.
A new fix is available for a newly disclosed vulnerability, CVE-2024-22024 for specific versions of Ivanti Connect Secure, Ivanti Policy Secure, and ZTA gateways.
Existing XML mitigations protect against CVE-2024-22024 from being exploited.
The vulnerability is not linked to any known threat actor or current exploitation activity.
We share the research of our CERT in a dedicated blog on the backdoor in DSLog. Please find the details here.
To find out in detail about the discovered backdoor in DSLog you can also download the investigation report as PDF below!
DSLog Backdoor investigation & IoCs PDFOn January 31, 2024, Ivanti released fixes to address four vulnerabilities, CVE-2023-46805, CVE-2024-21887, CVE-2024-21888 and CVE-2024-21893. The last two vulnerabilities were disclosed on the day of the patch release, along with a second set of mitigations that can be applied as an alternative to the fixes.
Details on CVE-2024-21893, a Server-Side Request Forgery (SSRF) vulnerability affecting the embedded SAML module, were quickly shared publicly by security researchers from Rapid7 then AssetNote, with a functional working PoC that anyone could use. Within hours of the release of the PoC, the Orange Cyberdefense CERT identified attacks targeting this SAML vulnerability.
Orange Cyberdefense discovered that attackers injected a backdoor into a component of the Ivanti appliance using this SAML vulnerability, thus providing the attacker with persistent remote access. The attackers also put measures in place to control access to the backdoor.
You can find the vulnerabilities details on two advisories Orange Cyberdefense published (customer account required to access the content):
Vulnerability Intelligence Watch: https://portal.cert.orangecyberdefense.com/vulns/60095
World Watch: https://portal.cert.orangecyberdefense.com/worldwatch/839001
Please apply the latest patch as this fixes the latest vulnerability, CVE-2024-22024, and the previously disclosed four vulnerabilities.
According to Ivanti the existing XML mitigations (mitigation.release.20240126.5.xml) already protects against CVE-2024-22024 exploitation [source].
According to Ivanti, a factory reset is not required to fix CVE-2024-22024 if it was already performed previously. Furthermore, 2 consecutive factory reset operations as we mentioned remain unnecessary. It is quite the opposite as we strongly recommended to apply the January 31 upgrades
Additional investigation can be performed using the network/host-based IOCs (available to our MTD clients through Orange Cyberdefense’s Datalake), detection rules, or analysis of suspicious behavior directly in logs or on the device. Also, pay attention to how appliances are configured as it can be in active-passive and/or involve cluster of devices and may require certain steps to ensure thorough investigation. We can help you assess if your instance was indeed compromised and if so, please contact our Incident Response hotline.
CISA urged Federal agencies to disconnect, and factory reset all their appliances [source]. Ivanti and Mandiant now recommend 2 consecutive upgrades [source], to prevent a possible future rollback that would put the device back in its previous “compromised” state. This ensures that instances are indeed clean. This extreme resolution process may be daunting, but failing that patches should be prioritized, alternatively at least apply the mitigating sooner vs. factory resetting later. Factory reset with full patching is recommended when a device has been compromised or if a target of interest could be in the sights of the Chinese threat actor behind these 0-days.
It is highly recommended to follow Ivanti's process and apply either the new XML mitigation or ideally apply the patches released on January 31, 2024 [source]. The patches are currently limited to a handful of versions so if a version is missing a patch, then apply the XML mitigation and please check back regularly to see if a specific version received a patch in the meantime.
Scans with the Ivanti Integrity Checker Tool (ICT) are not guaranteed to be completely accurate and may miss possible compromises but remain the most practical source of detection in many cases, if:
If these are true, then the device is probably free from compromise. Nevertheless, it is recommended to keep investigating for signs of compromise, by scanning the appliance regularly with the internal / external ICT and using the latest available IOCs.
A system snapshot and extract of relevant memory and log data is recommended before running the ICT external scan, as it reboots the appliance [source]. Consider enabling debug logging as this could improve detection. Debug level logging (including unauthenticated requests) is not enabled by default due to potential performance issues.
An incident response plan must be activated if a compromised device is identified. This plan may include:
Additional investigation can be performed using the network/host-based IOCs below (or the full list available only to our MTD/MTI clients through the Orange Cyberdefense Datalake), detection rules, or analysis of suspicious behavior directly in logs or on the device.
Also, pay attention to how Ivanti appliances are configured as it can be active-passive and/or involve a cluster of devices and may require certain steps to ensure thorough investigation. We can help you assess if your instance was indeed compromised and if so, please contact our Incident Response hotline.
All Orange Cyberdefense Managed Service clients are protected as our dedicated teams have applied the vendor's recommendations. Orange Cyberdefense Managed Threat Defense teams are proactively performing investigations for clients with this service. Impacted clients will be notified if any suspicious behavior is identified.
You can find the vulnerabilities details on two advisories Orange Cyberdefense published (customer account required to access the content):
What we thought was the start of mass exploitation on Thursday evening was false positives generated by the internal ICT checks running by default every two hours. These checks were finding new files from the package loaded onto the appliance when upgrading it. Nevertheless, in the wild exploitation by opportunistic attackers did start late on February 2, right after Rapid7 [source] and AssetNote [source] publicly disclosed how the SAML 0-day vulnerability, CVE-2024-21893, functioned by releasing a working PoC that anyone could use. The Orange Cyberdefense CERT identified attacks just a few hours after these were published. One example included activity involving Mirai-based botnet variants that were being dropped on compromised instances by opportunistic threat actors.
The SAML 0-day is believed to be linked to a third-party open-source library, the "xmltooling" dependency, maintained by Shibboleth [source]. This component was indeed vulnerable to a SSRF flaw fixed in June 2023 and tracked as CVE-2023-36661 (link for our Vulnerability Intelligence Watch clients).
The vulnerable endpoints, which pass to "saml-server" an XML that can be specially crafted, may include the following endpoints:
/dana-ws/saml.ws,
/dana-ws/saml20.ws,
/dana-ws/samlecp.ws,
/dana-na/auth/saml-sso.cgi
/dana-na/auth/saml-logout.cgi
/dana-na/auth/saml-consumer.cgi
Etc.
When processed by xmltooling using the function "XMLToolingFIPS.XMLObject.Signature", the malformed request will give the remote unauthenticated attacker access to the machine. The exploit can be chained with the first command injection 0-day (CVE-2024-21887) directly from within the device to bypass the initial XML mitigation.
Since the start of this crisis, Orange Cyberdefense CERT has been trying to identify vulnerable or compromised instances, using various discrepancies in responses when requesting some files or endpoints (i.e. the 403 "empty" response [source], the GIFTDEVISITOR webshell, the logo/login.gif fake file, etc.). After analyzing a compromised vs. clean instance, CERT found the legitimate module processing logs (DSlog.pm) had been backdoored and non-sensitive data about the appliance was dumped to a newly created .txt file. CERT identified assets displaying this behavior and found several instances possibly infected using this backdoor.
As a reminder, the original "403 - Forbidden: empty response" scan technique described above, which was used to assess whether the initial XML mitigation is applied or not, should not be used anymore since the patches are available. Instances already upgraded do not have the XML mitigation anymore and are not vulnerable.
A researcher on X (Twitter) explained another technique [source] to remotely check if one instance was backdoored using the SAML 0-day with the response from another query. The Orange Cyberdefense CERT could not confirm that this technique worked as described.
The large volume of exploitation attempts targeting the latest SAML vulnerability requires that impacted organizations respond quickly. The US government cybersecurity agency CISA requests all Federal agencies to disconnect appliances by Friday February 2, 2024, to reduce potential exposure [Source]. As part of this request, appliances can only be brought online after the patch has been applied.
The new XML mitigation provided by Ivanti can be applied to all product versions, and the patches just fix the security issues and do not include any new features. Another patch for a new branch (numbered 22.5R2.2) was released on February 1, 2024. The latest updated external ICT doesn't include IOCs or new detection techniques but enables users to check an instance even running on latest (i.e. patched) versions.
The Google's subsidiary released a technical follow-up blogpost about the malware strains used by the main, original threat actor. The incident response provider confirmed they saw highly targeted post-exploitation activity. Mandiant did not name any of the victims, nor their countries or sectors. The threat actor managed in some cases to compromise appliances then revert them to a clean state, to evade detection including from the external ICT scan. Attackers tampered with the internal ICT files to prevent it from running. Mandiant shared some Event IDs to detect such modifications in logs. In addition, Mandiant also shared other IDs that are linked to the clearing of system logs done by this APT group.
New malware strains attributed to the threat actor (UNC5221), are also detailed:
a new Perl webshell called BUSHWALK, embedded into a legitimate file (querymanifest.cgi) that enables read and write actions
a LIGHTWIRE variant found in another legitimate file: "compcheckresult.cgi". Mandiant recommends hunting for this GET request '"/dana-na/auth/url_default/compcheckresult.cgi?comp=comp&compid=%3Cobfuscated%C2%A0command%3E") within web logs, unallocated space, and memory images
another variant of WIREFIRE, dubbed CHAINLINE, found in a newly created health.py file [details] , is requested through the endpoint: "/api/v1/cav/client/health"
FRAMESTING webshell (in file category.py [details]) is accessed by the attacker through requests to "/api/v1/cav/client/categories"
multiple variants of the WARPWIRE credential stealer were also found by the incident responders.
The code of ZIPLINE's strain was further detailed by Mandiant, which linked made use of open-source tool such as Impacket, Iodine, CrackMapExec or Enum4Linux. As stated in previous updates, a .tar archive masquerading as a randomly generated 10-character CSS file within the directory "/home/webserver/htdocs/dana-na/css/" was used to store stolen information (dump of configuration and cache). As we already knew, stolen data was in some cases appended to legitimate "logo.gif" or "login.gif" file, for the same objective.
Two new 0-days have been announced by Ivanti in connection to the attack investigation here1, for CVE-2024-21888 and CVE-2024-21893. CVE-2024-21888 enables an attacker to conduct internal (local) privilege escalation and doesn't change the threat level. CVE-2024-21893 is more impactful and is a server-side request forgery in the SAML component to bypass authentication. The flaw is actively exploited by the Chinese threat actor in targeted attacks against a limited number of clients.
Exploitation of the existing vulnerabilities are continuing, even though the number of exposed vulnerable appliances greatly decreased over time, with only some hundreds to thousands remaining online. However, we confirmed on January 25 that at least 500 instances were exposing online sensitive data through the "fake" logo.gif file created by the threat actor.
Volexity first talked on January 18 about a Rust-based malware strain for Linux that was leveraged by the threat actor, without providing more details at the time. One researcher published a few days later a few IOCs about it here, then further information about this new strain he calls "KrustyLoader" 2 (because coded in Rust). The researcher shared a decryption script for retrieving the URLs used by the loader to drop a Sliver backdoor, a well-known advanced post-exploitation tool that our "Managed Threat Intell-detect" service proactively tracks since a long time.
During our multiple CSIRT engagements, we identified that devices compromised with the Metasploit framework often involve the presence of specific artefacts in the "/imgs/Ivanti.zip/" folder. Furthermore, we believe checks with the (even external) ICT solution may in some cases not identify a compromised appliance. Ivanti already mentioned it previously and the US government agency CISA3 stated on January 30 that attackers managed to subvert the external ICT results to evade detection.
The initial vulnerable endpoints are well-known, with many security vendors adding signatures for requests made to those endpoints (i.e. Cloudflare, Snort/Suricata, etc.4 ). Picus listed many of them at the bottom of their article5. SIEM providers such as Splunk also provided examples of queries to hunt for suspicious requests. However some endpoints tied to the vulnerable path " https:///api/v1/totp/user-backup-code/.." are not well documented by the vendor, and our experts found out that the product does not correctly log some of those requests. Exploitation attempts by attackers before XML mitigation was applied may have left little evidence in logs, and more advanced investigation may be needed.
Ivanti released patches fixing CVE-2023-46805, CVE-2024-21887, CVE-2024-21888 and CVE-2024-21893, in these ICS versions:
Ivanti ZTA version 22.6R1.3 also received a patch for the 4 vulnerabilities listed.
Ivanti recommends doing a factory reset of devices before patching, with few hours needed for each operation. We nevertheless encourage you to patch appliances as soon as security fixes for your versions get released, even if more complexes than applying the XML mitigation.
If unable to patch, Ivanti also announced a new mitigation XML file (mitigation.release.20240126.5.xml) to protect against the two additional 0-days released today. When available (NDLR: the download link does not work at 2pm Paris time), we highly encourage users to deploy at least this new protection until the devices can be fully patched. This updated workaround blocks SAML authentication vulnerability that can be used to exploit in CVE-2024-21893:
"The new mitigation will block all SAML communication and authentication. This will have limited functionality impact on customers who use LDAP for authentication" said Ivanti.
On top of the new XML, Ivanti released a new version of their external ICT:
Any asset exposed on Internet that did not apply one of the mitigations yet should be considered probably hacked. This echoes the latest CISA blog6 that recommends defenders to keep hunting for sign of compromise in any case.
A forensics firm called Volexity first discovered two 0-day vulnerabilities, CVE-2023-46805 and CVE-2024-21887, in Ivanti’s Connect Secure (ICS) products in early December 2023. These vulnerabilities were exploited to insert webshells that Volexity named GLASSTOKEN. Volexity also assigned UTA0178 as the moniker for the unknown threat actor they allege has links with the Chinese government.
The level of sophistication and stealth of the threat actors is high, and initially it was thought that the detected activity could be related to a targeted espionage campaign, due to the initially low number of impacted ICS hosts. Later the number of impacted ICS hosts increased, which diminished the likelihood that we’re dealing with a highly targeted attack. UTA0178 is yet to be defined in terms of other existing Chinese nation-state hackers. Volexity also announced that another threat actor, tracked as UTA0188, is also targeting the ICS flaws.
Following the Volexity report, Mandiant also shared technical details of the malware associated with the recent attacks attributed to the threat actor UTA0178 (tracked as UNC5221 by Mandiant). Post-exploitation activity involves the use of several custom malware, dubbed ZIPLINE, THINSPOOL, LIGHTWIRE, WARPWIRE and WIREFIRE.
On January 10, 2024, Ivanti acknowledged these two 0-days by publishing an advisory naming the impacted products as the Ivanti Connect Secure and Ivanti Policy Secure products. These vulnerabilities impact all versions of Ivanti Connect Secure (a VPN solution formerly known as Pulse Connect Secure), Policy Secure and to some extent Neurons for ZTA gateway.
The two vulnerabilities were chained together to provide unauthenticated Remote Code Execution (RCE). CVE-2023-46805 is an authentication bypass vulnerability, while CVE-2024-21887 is a command-injection security flaw. Combined, they provide attackers unauthenticated access to the ICS host, enabling them to further gain ground on the network and move laterally, in this case using protocols such as RDP, SMB and SSH.
Unfortunately details of the vulnerabilities became public knowledge due to work published by Watchtower Labs and Rapid 7. Watchtower Labs published a blogpost announcing they managed to reproduce the flaws. They explained that by identifying the endpoints blocked by Ivanti in its workaround (i.e. the encrypted XML mitigation file), as differences are visible in responses from vulnerable vs. mitigated instances. Rapid 7 then disclosed details relating to CVE-2023-46805, thus enabling others to craft exploits for this flaw.
Naturally, we expect to see an increase in the number of exploit attempts, as there are still several thousands of devices that are unpatched. Around the same time, GreyNoise reported that they detected an increase in the exploit attempts related to the two ICS vulnerabilities. Volexity in turn indicated that attackers are currently installing malware that facilitates crypto-mining (XMRIG) and other types of payloads.
By January 17, 2023, more than 4,000 ICS hosts were still lacking the mitigation provided by Ivanti. Around the same time Volexity estimated that approximately 2,100 Ivanti Connect Secure VPN devices have been compromised with the GIFTEDVISITOR implant attributed to UTA0178.
Some associated logs that may be investigated would potentially be:
msg="WEB31809: Access for /api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark is blocked due to patch(Patch 2)."
To verify if XML was applied, you may for instance request one of the endpoints located here:
/api/v1/configuration/users/user-roles/user-role/rest-userrole-1/web/web-bookmarks/bookmark
/api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark
This REST API has been introduced in the 8.3R3 Release of Pulse Secure and exists in the documentation for all the 9.XRX releases.
A Nmap script may ease this check for you. The response will include in both cases a 403 Forbidden, but the response content will contain:
an empty page for a vulnerable instance
a full HTML content embedding "Access to the Web site is blocked by your administrator. [...]" in the case the XML fix was applied
Mandiant detailed the GLASSTOKEN webshell mentioned in the Volexity article. Tracked under the name LIGHTWIRE by Mandiant, this malware is embedded in a legitimate Connect Secure file (a CGI script) and allows the threat actor to intercept specific requests and interpret them as Perl code. It is important to note that this LIGHTWIRE only intercepts requests to "compcheckresult.cgi" that contain the 'comp=comp' and 'compid' parameters.
THINSPOOL acts as the Perl script to drop the LIGHTWIRE webshells to a legitimate Ivanti Connect Secure file. This allows the threat actor to persist on compromised devices and manipulate Ivanti Connect Secure files or processes in such a way as to avoid detection. However, according to Mandiant's observations, this attempt failed, as it was detected by Ivanti's Integrity Checker Tool.
WIREFIRE is the Python webshell that functions as trojan horse for a specific component of the Connect Secure appliance (the visits.py file mentioned earlier). This webshell can also download files and execute arbitrary commands on the compromised device. In addition, it decodes, decrypts, and decompresses all raw data existing after a GIF (Graphics Interchange Format) header to run as a sub-process. WIREFIRE undoubtedly uses this header to mask or transport malicious data.
WARPWIRE is the credential collection tool written in Javascript and embedded in a legitimate Connect Secure file, also briefly mentioned by Volexity. Its main purpose is to collect credentials, including usernames and passwords in plain text. Once collected, this information is Base64 encoded and submitted to a command-and-control server using HTTP GET requests.
ZIPLINE is the only newly detailed malware, and more precisely an ELF passive backdoor that hijacks the accept() function in libsecure.so in order to intercept network traffic and take control of the incoming connection discreetly, without awakening suspicion. Once this has been achieved, the malware carries out various malicious actions, such as gathering information, downloading files, setting up proxy servers and tunnelling servers. But no hash of a sample for this strain was provided by Mandiant.
Malicious file location:
/home/perl/DSLogConfig[.]pm
/home/etc/sql/dsserver/sessionserver[.]pl
/home/etc/sql/dsserver/sessionserver[.]sh
/home/webserver/htdocs/dana-na/auth/compcheckresult[.]cgi
/home/webserver/htdocs/dana-na/auth/lastauthserverused[.]js
/tmp/rev
/tmp/s[.]py
/tmp/s[.]jar
/tmp/b
/tmp/kill
Hostname:
gpoaccess[.]com
webb-institute[.]com
symantke[.]com
IPs:
206.189.208[.]156
75.145.243[.]85
47.207.9[.]89
98.160.48[.]170
173.220.106[.]166
73.128.178[.]221
50.243.177[.]161
50.213.208[.]89
64.24.179[.]210
75.145.224[.]109
50.215.39[.]49
71.127.149[.]194
173.53.43[.]7
MD5 hash:
3d97f55a03ceb4f71671aa2ecf5b24e9 (compcheckresult[.]cgi)
677c1aa6e2503b56fe13e1568a814754 (sessionserver[.]sh)
d0c7a334a4d9dcd3c6335ae13bee59ea (lastauthserverused[.]js)
6de651357a15efd01db4e658249d4981 (visits[.]py)