24/7 aid with cyber incidents
Ivanti Connect Secure: Journey to the core of the DSLog backdoor
Ivanti Connect Secure: Journey to the core of the DSLog backdoor
Summary
- Ivanti Server-Side Request Forgery vulnerability, CVE-2024-21893, is used to inject previously unknown and interesting backdoor.
- Access to this backdoor is controlled through basic ‘API key’ mechanism.
- Attacker performs commands injection with high privileges.
- Orange Cyberdefense continues to monitor, investigate, and report on activity that is associated with exploitation of the recently disclosed Ivanti vulnerabilities.
You can download this research as PDF using the button below!
Background
On January 31, 2024, Ivanti released fixes to address four vulnerabilities, CVE-2023-46805, CVE-2024-21887, CVE-2024-21888 and CVE-2024-21893. The last two vulnerabilities were disclosed on the day of the patch release, along with a second set of mitigations that can be applied as an alternative to the fixes.
For the full coverage of the Ivanti case, including the most recent updates you can check our blog CERT alert: Zero-day in Ivanti software!
Details on CVE-2024-21893, a Server-Side Request Forgery (SSRF) vulnerability affecting the embedded SAML module, were quickly shared publicly by security researchers from Rapid7 then AssetNote, with a functional working PoC that anyone could use. Within hours of the release of the PoC, the Orange Cyberdefense CERT identified attacks targeting this SAML vulnerability.
Orange Cyberdefense discovered that attackers injected a backdoor into a component of the Ivanti appliance using this SAML vulnerability, thus providing the attacker with persistent remote access. The attackers also put measures in place to control access to the backdoor.
You can find the vulnerabilities details on two advisories Orange Cyberdefense published (customer account required to access the content):
Vulnerability Intelligence Watch: https://portal.cert.orangecyberdefense.com/vulns/60095
World Watch: https://portal.cert.orangecyberdefense.com/worldwatch/839001
2024-02-03 12:15am UTC | Orange Cyberdefense detects a compromise of a customer exploiting CVE-2024-21893 vulnerability |
2024-02-03 8:35pm UTC | Orange Cyberdefense detects 670 compromised assets; using index.txt and index{1,2} technique |
On the first hour of February 3rd, Orange Cyberdefense analyzed a snapshot and logs of a compromised appliance. This appliance had the initial XML mitigation (API endpoints blocked) in place but not yet the second mitigation (or patch).
After decrypting the snapshot, analysis started seeking to identify any recent pattern that might explain how the appliance has been compromised. Subsequently Orange Cyberdefense discovered a backdoor that was injected into the appliance’s code base.
Initial Request
Evidence of exploitation can be found within the following Ivanti log file:
log.localhostX-DAY-YEAR-XX-XX_XX_XX_XX-(XXX).access
In which, unauthenticated threat actors can be seen issuing a SAML authentication request which contains an encoded command within the ‘RetrievalMethod URI’:
The URL encoded request contains a Base64 encoded command within the requested URI. Removing the URI encoding reveals:
Complete command with previous Base64 text highlighted in orange:
The ‘echo’ command is used to output the results of the ‘uname’ command to the file ‘index2.txt’ in a publicly accessible directory using the bash command interpreter and Base64 decode utility.
The content of the file ‘index2.txt’ contains the output of the ‘uname’ command as follows:
The content of the file reveals the username under which the command was executed, in this case the ‘root’ user (user id 0).
This is almost certainly an internal reconnaissance activity to confirm that the exploit technique will/has given attackers root access to the Ivanti device. The creation time of the file and the timestamp within the file indicates when the ‘uname’ command was executed. This can be useful if access logs have been deleted by attackers.
Backdoor installation
Following a successful attempt, the following SAML request is issued:
The URL encoded request contains a Base64 encoded command within the requested URI:
Complete command with previous Base64 text highlighted in orange:
In this instance, the ‘mount’ command is used to ensure that the attacker has read/write permissions on the file system.
Next the script attempts to detect whether the ‘DSLog.pm’ Perl script, a legitimate command normally used to log events on the Ivanti device, has already been modified by the attacker. The command searches the file for the string 'HTTP_USER_AGENT' and if present, the script responds with ‘OK’. This will essentially tell the attacker if their malicious code is already in place.
The backdoor is injected using the ‘sed’ command at line 102 of the ‘DSLog.pm’ file if the ‘HTTP_USER_AGENT’ string does not exist.
The backdoor installed at line 102 in the DSLog.pm file:
Once this backdoor is in place, the attacker can execute commands on the compromised device.
Backdoor functionality
The backdoor is inserted into an existing Perl file called ‘DSLog.pm’. The DSLog module is responsible to log anything on the device, such as:
- authenticated web requests (username, realm, roles, uid, sessionId, etc.) for user and admin;
- any web request (sourceip, userAgent, browserId, etc.);
- system logs (admin, events, sensors, certHash, etc.):
The attacker specifically targeted the main function of the module, related to the request received by the device: « DSLog::Msg ». This function is related to more than 200 external endpoints (in /dana-na/ & /dana-admin/).
Significant differences between several webshells were observed during the first two campaigns that exploited CVE-2023-46805 and CVE-2024-21887. These are:
- the webshell does not return status/code when trying to contact it. There is no known way to detect it directly.
- the implemented backdoor uses a unique hash per appliance. This hash cannot be used to contact the same backdoor implemented in another device.
This unique SHA256 hash is stored at line 104 in the example shared earlier and is referenced by the ‘$qur’ variable. This hash acts like a simple ‘API’ key. The attacker must supply this hash by populating the HTTP User-Agent header field in an HTTP request to the appliance.
When the SHA256 values match, the content of query parameter named 'cdi' is selected. This content contains the commands to be executed and is a hex encoded ROT47 ciphertext. As shown with the output of the ‘uname’ command, the user that the commands will execute as is the ‘root’ user. At this point the attacker can execute any command on the compromised host.
It was not possible to determine the scheme the attacker used to calculate the SHA256 value for the ‘$qur’ variable. Many combinations based on the IP address, hostname of the device, as well as some additional material created by the attacker such as the content of the ‘index.txt’ and ‘index2.txt’ files was attempted, but none matched the value present in the DSLog.pm file.
Attacker issued commands using the backdoor should be found as hex encoded values in the ‘.access’ logs. However, Orange Cyberdefense CERT noted that ‘.access’ logs were cleared on multiple devices to hide past and future activities done on the device.
After this analysis, we were looking for a way to detect the presence of the backdoor at scale. Following further analysis of the backdoor and the way it has been implemented, we think it cannot be checked in a simple way (such as for some backdoor variants dropped using the CVE-2023-46805 & CVE-2024-21887 vulnerabilities).
We also determine that the best way, at least as a quick-win, to determine if the device has been compromised, was to check the artifacts created by the attacker himself when triggering the SSRF vulnerability: the index{}.txt files,are located in the same directory:
- hxxp://{ip}/dana-na/imgs/index.txt (found in few appliances – containing less than 10 random characters);
- hxxp://{ip}/dana-na/imgs/index1.txt (found in few appliances);
- hxxp://{ip}/dana-na/imgs/index2.txt (found in most compromised appliances).
By implementing this check, we identified almost 700 compromised assets. 20% of these appliances were also compromised during the first campaign. However, the remaining ones had the initial XML mitigation applied (so were not vulnerable to CVE-2023-46805 & CVE-2024-21887) but lacked the second mitigation or patches.
Currently we’re still monitoring assets to identify some variants of this attack.
Backdoor Discovery Timeline
Indicators of Compromise
Host-based IoCs
| Path + filename | Hash | Description |
|---|---|---|
| /root/home/perl/DSLog[.]pm | N/A - varies | Legitimate DSLog Log module embedding the backdoor |
| /root/home/webserver/htdocs/dana-na/imgs/index[.]txt | N/A - varies | Some seem to be random characters. |
| /root/home/webserver/htdocs/dana-na/imgs/index1[.]txt | N/A - varies | Some seem to be random characters. |
| /root/home/webserver/htdocs/dana-na/imgs/index2[.]txt | N/A - varies | Embedding the result of ‘uname -a’ |
| /root/home/webserver/htdocs/dana-na/imgs/index2[.]txt |
Network-based IoC
| Network Indicator | Type | Description |
|---|---|---|
| 159.65.123.122 | IP Address | Massive exploitation activity |
Recommended Actions
CISA urged Federal agencies to disconnect, and factory reset all their appliances [source]. Ivanti and Mandiant now recommend 2 consecutive upgrades [source], to prevent a possible future rollback that would put the device back in its previous “compromised” state. This ensures that instances are indeed clean. This extreme resolution process may be daunting, but failing that patches should be prioritized, alternatively at least apply the mitigating sooner vs. factory resetting later. Factory reset with full patching is recommended when a device has been compromised or if a target of interest could be in the sights of the Chinese threat actor behind these 0-days.
It is highly recommended to follow Ivanti's process and apply either the new XML mitigation or ideally apply the patches released on January 31, 2024 [source]. The patches are currently limited to a handful of versions so if a version is missing a patch, then apply the XML mitigation and please check back regularly to see if a specific version received a patch in the meantime.
Scans with the Ivanti Integrity Checker Tool (ICT) are not guaranteed to be completely accurate and may miss possible compromises but remain the most practical source of detection in many cases, if:
- your appliance was mitigated early on (around January 11th onward)
- no historical ICT nor external ICT scans showed signs of compromise,
- and no other suspicious behavior, i.e. in IOCs, logs, or alerts from security solutions was found in the rest of the infrastructure.
If these are true, then the device is probably free from compromise. Nevertheless, it is recommended to keep investigating for signs of compromise, by scanning the appliance regularly with the internal / external ICT and using the latest available IOCs.
A system snapshot and extract of relevant memory and log data is recommended before running the ICT external scan, as it reboots the appliance [source]. Consider enabling debug logging as this could improve detection. Debug level logging (including unauthenticated requests) is not enabled by default due to potential performance issues.
An incident response plan must be activated if a compromised device is identified. This plan may include:
- isolating the appliance from other systems
- auditing recently created or modified privileged accounts
- revoking credentials including certificates or passwords possibly exposed
- monitoring suspicious authentication attempts
- investigating and finding anomalies in services or devices previously connected to the Ivanti appliance
Additional investigation can be performed using the network/host-based IOCs below (or the full list available only to our MTD/MTI clients through the Orange Cyberdefense Datalake), detection rules, or analysis of suspicious behavior directly in logs or on the device.
Also, pay attention to how Ivanti appliances are configured as it can be active-passive and/or involve a cluster of devices and may require certain steps to ensure thorough investigation. We can help you assess if your instance was indeed compromised and if so, please contact our Incident Response hotline.
What we are doing
All Orange Cyberdefense Managed Service clients are protected as our dedicated teams have applied the vendor's recommendations. Orange Cyberdefense Managed Threat Defense teams are proactively performing investigations for clients with this service. Impacted clients will be notified if any suspicious behavior is identified.
You can find the vulnerabilities details on two advisories Orange Cyberdefense published (customer account required to access the content):
- Vulnerability Intelligence Watch: https://portal.cert.orangecyberdefense.com/vulns/60095
- World Watch: https://portal.cert.orangecyberdefense.com/worldwatch/839001
We can help you assess if your instance was indeed compromised and if so, please contact our Incident Response hotline.
For the full coverage of the Ivanti case, including the most recent updates you can check our blog CERT alert: Zero-day in Ivanti software!
