Log4j Vulnerability

A new Remote Code Execution (RCE) vulnerability (identified as CVE-2021-44228) has been discovered in the Apache Java module, log4j. More vulnerabilities, CVE-2021-45046  and CVE-2021-45105, were discovered that also require critical review. Without wishing to sound alarmist, this may be the most serious vulnerability we’ve faced in the last decade. Once the dust has settled, it may be the most serious vulnerability in the history of modern cyber security.

If you’re reading this, it almost certainly impacts you.

Update: 24 December 2021

• Updated with additional technical detail from our CSIRT, especially regarding attack string obfuscation.

Update: 23 December 2021

• Updated with news about active exploitation by including a ransomware actor and a banking trojan.
• Update with news about a compromise of the Belgian Ministry of Defense
• Updated to reflect the Blumira security team alternative attack vector leveraging a WebSocket JavaScript connection to trigger the exploit
• Updated with reference to a CISA vulnerability scanner at https://github.com/cisagov/log4j-scanner
• Updated with news that China’s Ministry of Industry and Information Technology (MIIT) has sanctioned the cloud computing company Alibaba for disclosing the log4j vulnerability without consulting the government as required by law

Update: 18 December 2021

• Added information on Log4j version 2.17 and CVE-2021-45105.

Update: 17 December 2021

• Updated to mention the first claim that Conti ransomware operators are gaining access through the Log4j vulnerability.
• Updated to include reference to show that CVE-2021-45046 could potentially lead to remote code execution under certain conditions.
• Added that deleting JndiLookup class is dangerous and must only be considered as a final solution.
• Mention that Log4j version 2.17 is planned for release some time.
• Updated timeline graphic to match the SharePoint site’s version.

Update: 16 December 2021

• Updated to reflect that CVE-2021-45046 can “allow for exfiltration of sensitive data in certain circumstances” and reiterate that one should upgrade to 2.16.0 when possible
• Updated to report that more than 300 products are already reported vulnerable on the Dutch government CERT list, and will probably reach 500 before next week.
• Updated to advertise that Orange Cyberdefense is now sharing our own Indicators of Compromise (IoC) publicly via GitHub at https://github.com/Orange-Cyberdefense/log4shell_iocs

Update: 15 December 2021

• Updated to emphasize that we are now recommending an upgrade to 2.16 to mitigate the threat
• Updated to report that a third CVE number was assigned to a vulnerability bypass of the 2.15 version under certain non-default configurations: CVE-2021-45046. Apparently not an RCE at this point. It’s fixed by version 2.16 already.
• Updated to reflect JNDI lookup supports protocols such as LDAP, RMI, DNS, or even IIOP, and we’ve finally noticed exploitation attempts using IIOP today.
• Updated to emphasize that log4j is a library, so we cannot 100% confirm this via a list of “installed” software.
• Updated to report on some newly-confirmed products, including IBM, AWS, and Zoho, and to report new trackers from CISA and NCSC-NL.
• Updated to clarify that if the exploitation is successful, the JDNI string is not actually logged (because it’s interpreted by the Log4j logger).
• Updated to reflect that Web Application Firewalls (WAF) can offer reasonable protection against the majority of know attacks.
• Updated to report that ‘Khonsari’, a new simple ransomware strain, was seen deployed after a log4j attack.

Update: 14 December 2021

• Updated with an example attack string.
• Updated to clarify that the vulnerable platform itself does not have to be internet-facing. Any remotely accessible interface that offers a path to log4j can be exploited as a vector.
• Updated to share the iPhone / iCloud attack example to illustrate the point above.
• Updated to emphasize the Environment Variables attack scenario.
• Updated with an improved image to illustrate the attack surface.
• Updated with a timeline.
• Updated with incident trend data from our Nordic CyberSOC.
• Updated with new patching details.
• Updated with a link to the SwitHalk Blueteam cheatsheet, which we contribute to.
• Updated to include a vulnerability decision tree.
• Added a reference to the work-around formatMsgNoLookups=true.

The attack surface is almost limitless, as long as the attack string finds it way to a vulnerable component.

In one instance, the vulnerability has been exploited on the popular Minecraft gaming platform simply by typing a specially crafted message into the game’s chat window. In another revealing example, a researcher illustrated that the Apple iCloud infrastructure could be attacked by inserting the appropriately formatted string in Device Name field in the iPhone Settings interface. There are also anecdotes about the exploit being triggered via an SMS text message to an automated backend system. In these kinds of examples, the successful exploit is confirmed via a DNS call back, in which a command is issued to resolve a DNS name in a domain the attacker can monitor.

The vulnerability was discovered and reported to the Apache Foundation on November 24^th by a researcher at China’s Alibaba Cloud Security Team. China’s Ministry of Industry and Information Technology (MIIT) subsequently sanctioned the cloud computing company under a new law requiring national companies to first inform the government when they discover a vulnerability. The vulnerability was publicly disclosed on December 9^th 2021.

The exploit for this vulnerability is being referred to as ‘log4shell’ and involves creating a carefully crafted text string that finds its way through to the log4j logging component, where it causes the specified command to be executed on the underlying operating system.

The string that triggers the exploit can be crafted in a variety of different formats – a technique known as obfuscation. This makes detecting an attack via logs or on the network a significant challenge.

The exploit is what’s known as ‘blind’, meaning that the attacker doesn’t receive direct feedback on whether it succeeded. A typical approach would therefore be to attempt to trigger a ‘reverse’ connection from the exploited component to a server the attacker controls on the internet from which remote command and control can be established. Preventing outbound connections from internet-facing servers would help to mitigate this approach, but it is by no means the only option the attacker has.

There are many avenues open to an attacker that don’t require real-time command and control, so this challenge really shouldn’t feature in your risk calculus. One popular vector so far has surprisingly been to deploy crypto coin mining malware.

The researcher Marcus Hutchins has warned that if system environment variables are set, they could also be referenced and fetched inside the Java context. In one example he provides, an AWS_SECRET_ACCESS_KEY is set on the server and fetched via the exploit. Environment variables could also include database connection strings & passwords, and other secrets.

We believe this variation of the attack would be even more difficult to detect, since it doesn’t require any additional network traffic or risk logging on the affected platform.

We also note that the Java Naming and Directory Interface (JNDI – used to trigger the exploit) supports protocols such as LDAP, RMI, DNS, or even ‘Internet Inter-ORB Protocol (IIOP) – sed to facilitate network interaction between distributed programs written in different programming languages. Our CyberSOCs have now observed exploitation attempts using IIOP, and the Blumira security team has released a report on an alternative attack vector leveraging a WebSocket JavaScript connection to trigger the exploit on unpatched systems.

We shouldn’t make the mistake of underestimating the creativity of our adversaries or the level of mayhem that this vulnerability will enable.

News of the vulnerability became publicly known on Friday, December 10^th. However, there is evidence to suggest that there have been attempts to exploit the issue since at least December 1^st, suggesting the exploit has been in the wild for at least 9 days before it was publicly disclosed. We may yet discover that the exploit is even older than that.

Since being publicly disclosed on the 10^th, several exploits have been developed and published, and there are widespread reports of generic scans and targeted attacks.

December 13th Bitdefender reported that they have detected an attempt to use the log4j vulnerability to deploy a new ransomware family they called ‘Khonsari’ (https://businessinsights.bitdefender.com/technical-advisory-zero-day-critical-vulnerability-in-log4j2-exploited-in-the-wild). It later emerged that ‘Kharen Konsari’ is a real person and that the phone number and the email shown in the ransom note probably point to her. The whole incident may simply be an elaborate ruse, therefore. Although we have not yet witnessed it, we do anticipate that ransomware crews start exploiting this vulnerability in anger in the near future. Attacks of this kind are already being attributed to state actors like Phosphorus, Nemesis Kitten and Hafnium.

On December 17 AdvIntel published a blog claiming that the Conti ransomware group has started targeting the Log4j vulnerability, tracked as CVE-2021-44228. AdvIntel specifically mentions that Conti is interested in VMware vCenter Server and vCenter Cloud Gateway that also contains this vulnerability.

As of 23 December, we become aware of two new malicious actors exploiting this vulnerability in the wild – a ransomware operator and a banking trojan.

The new ransomware operator is known as the ‘TellYouThePass’ gang. This ransomware has been dormant since the (European) summer of 2020, but it is regularly used when a new large vulnerability is discovered. This was for example the case with Eternal Blue last year. TellYouThePass does not operate as RaaS (Ransomware-as-a-Service).

The second actor exploiting this vulnerability is Dridex, which is a banking trojan originally developed to steal online banking credentials. Over time, the malware has evolved to become a more general malware loader, used in particular to drive ransomware attacks from operations allegedly linked to the Evil Corp hacking group. Infections include BitPaymer, DoppelPaymer and possibly other ransomware variants.

December 23rd also marks reports of the first high-profile victims. Belgian Ministry of Defense has confirmed a cyberattack on its networks leveraging the Log4j vulnerability, which we feel safe to assume involved ransomware. The reports contain very few other details for now.

The bottom line for you is simple: Firstly, you may already have been targeted using this vulnerability sometime in the past. Secondly, your internet-facing systems will certainly be facing scans for generic manifestations of the vulnerability and will be compromised, if possible, even for something as banal as a botnet or a cryptocurrency miner. Finally, you should anticipate that the vulnerability will be exploited in more sophisticated, targeted attacks in the near future, and for lateral movement across your networks once an initial compromise has been accomplished in some other way.

As we mentioned before, our understanding of the scale of the problem is still in its early stages. Steel yourself for a long, ongoing struggle as the scope continues to expand.

As of December 16^th we added ivanti MobileIron to the list of vulnerable platforms, which is also being actively exploited in the wild. The total list of known vulnerable systems now exceeds 300 and will probably reach 500 by the end of the week

Information about vulnerable, possibly-vulnerable and not-vulnerable products is in fact changing so rapidly that, rather than attempt to keep these resources updated, we have opted to share any new intelligence we collect in collaboration via GitHub here:

• https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
• US Cybersecurity & Infrastructure Security Agency (CISA):
  https://github.com/cisagov/log4j-affected-db
• The Dutch government Nationaal Cyber Security Centrum – NCSC
  https://github.com/NCSC-NL/log4shell

We expect to see these two lists merging over time.

As the vendors are still in the process of investigating this vulnerability, our CERT is engaged to continuously update our Vulnerability Intelligence Database (). We will update via the GitHub page above, via our Managed Vulnerability Intelligence Watch service, or directly via the services you engage us for.

Our multi-disciplinary crisis management team has been engaged with this issue since we published our first World Watch advisory on Friday. We have been meeting regularly to communicate updates and track progress regularly all weekend. Our team includes:

• CERT
• Research
• The Office of the CISO
• Internal cybersecurity
• Internal IT
• Global operations – Managed Detection and Response and Vulnerability Management
• Penetration Testing
• Computer Security Incident Response Team (CSIRT)

Our efforts have focused on the following:

• Tracking the scope and the scale of the issue, variations, attack vectors, affected products, and potential mitigations
• Engaging with our vendors and partners
• Updating our customers via our World Watch advisory service
• Updating our detection capabilities across our worldwide SOCs and CyberSOC
• Updating our vulnerability scanning platform with the most current detections available
• Triaging and patching our own systems to gauge the counter any potential impact to our environment and ensure the safety and reliability of our customer services

We will continue to work tirelessly to ensure the safety of our systems, and the integrity of our services. And to keep you, our customers, informed, updated and supported as we work together to tackle this challenge and ensure there are no serious incidents.

Initially, it was thought that a later version of Java had protection but that was found not to be the case on Tuesday December 14^th.

There are various tools commercially and freely available to ‘scan’ for vulnerable log4j instances. For various reasons, results from such scanners will vary. One the more useful free scanning options has been released as a Python script from the US Cybersecurity and Infrastructure Security Agency (CISA), which can be found here: https://github.com/cisagov/log4j-scanner. As we mention above, however, remote scanning is not sufficient to identify this vulnerability in your environment.

One mitigation that has been discussed is deleting the offending class from the Log4j module. Please try to avoid taking this route. This action must only be considered as a last resort and after you have evaluated the impact. The Apache Log4j team provides instructions on how to delete the JndiLookup class. Manually removing the class could break or destabilize the application. This is an extreme solution to the problem, but it must be treated temporarily. You’ll need to keep track of the modified artifacts to ensure you do not reintroduce the vulnerability later.

Modifying signed Java artifacts such as JAR, WAR, or EAR files will cause the application to fail at start-up as the integrity of the components was impacted. This behavior is as designed and will protect against code modification. When dealing with nested Java code artifacts in WAR and EAR files you’ll need to reclusively go through each file to locate the JndiLookup class. There may also be cached copies of the extracted WAR or EAR files on the system. These will need to be removed. Restarting the affected application is, in most cases, the only way to get the changes to reflect as the Java Classloader will cache this file in memory.

The terrifying scope and scale of this issue is a ‘first’ for the industry. But like other security threat firsts in our space, our collective efforts will allow us to tackle the threat, manage it, and emerge wiser and stronger on the other end.

Please note there will be updates to this blog.