Log4j – 3 Steps to Detect and Patch the Log4Shell Vulnerability Now

It was a long weekend for cybersecurity professionals, thanks to a new vulnerability that impacts almost every organization, dubbed “Log4Shell”.

Working after hours is part of the job for front-line defenders, and that’s what happened this last weekend for security professionals protecting their environments from the new vulnerability Log4j, listed as CVE-2021-44228, impacting customers with Apache Enterprise installed in their environments. Because that includes most internet-facing systems, this is a global threat with new details emerging by the hour.

Take some time to visit Deepwatch labs to get our Threat Intel team’s Significant Cyber Report on this evolving vulnerability, threat outlook, related technical details for more information.

Read more CVE-2021-44228 details: Significant Cyber Event | Log4j Zero-day With Proof-of-Concept Code and Active Scanning Gets Security Fix on Deepwatch Labs

TL;DR – The Latest on Log4j

Deepwatch is actively working on risk mitigation for customers on CVE-2021-44228, the actively exploited vulnerability in Apache Log4j.

• This vulnerability, dubbed “Log4Shell”, affects a popular Java logging library that organizations may have in their environment.
• Log4Shell was first discovered in the Microsoft-owned Minecraft video game, with concurrent reports that Apple iCloud, Twitter, Cloudflare, and more have also been targeted.
• Apache has released an updated version for users to patch their systems, Log4j 2.15.0. You can read the announcement, instructions to patch, and real-time updates on their website here: https://logging.apache.org/log4j/2.x/security.html
• Deepwatch encourages all customers to work closely with their Squads to confirm their internal and third-party usage of Log4j for vulnerable configurations, and take necessary response actions as needed.
• Due to the severity of this vulnerability, Deepwatch has already released and enabled a detection rule to seek any IOCs related to Log4j in customer environments.
• All service and delivery teams at Deepwatch are working closely with Squads and customers to mitigate risks associated with the Log4Shell vulnerability, including Deepwatch Managed Detection and Response, Vulnerability Management, Firewall, and Endpoint Detection and Response teams. Deepwatch’s Threat Operations team is actively updating IOC watchlists with new intel as it comes in, and Threat Hunters are actively threat hunting in customer environments for any potential IOCs.

Log4Shell Vulnerability and the 3 Steps to Detect and Patch

Log4j has a ubiquitous presence in almost all major Java-based enterprise apps and servers. Therefore, literally, every organization with internet-facing assets and systems should be reviewing their potential for this risk to impact their environments now and in the unforeseeable future. Apache has noted that any organization utilizing Apache Struts is “likely vulnerable.”

Deepwatch Security experts have summarized the following recommendations for Apache users who need to detect and respond to Log4j:

1. Identify Devices that may have been impacted by Log4j with authenticated vulnerability scanning. See below for more information on scanning for Log4j.
2. Review Log Sources to Identify and Detect Indicators of Compromise (IOCs).
3. Patch and Implement IDS/IPS rules for all applicable assets within your environment, prioritizing externally facing assets.

Here’s how to know if you need to take any action:

• In order for the Log4Shell vulnerability to be exploited, Log4j needs to be part of a running application or service that’s exposed to the internet or internal network.
  • Many devices can have Log4j installed, but they will not have it running as an active service.
  • While any outdated software can be a vulnerability, this particular vulnerability must be actively running in order to be exploited.
  • Due to the severity of this vulnerability, if there is any doubt, treat any installed instance as dangerous.
  • The vulnerability itself is based on improperly sanitized user-supplied input.
• If your cloud provider is hosting a service that utilizes Log4j, then will be impacted.
  • Both Azure (https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/) and AWS (https://aws.amazon.com/security/security-bulletins/AWS-2021-006/) have detailed impacted services they provide and are in the process of patching. Full details and the latest updates are being provided in real-time on their websites.
• If the application Log4j is not running, but it is being used for processing or back-ups, your systems are vulnerable and could be attacked:
  • If you have built a process or system that sends logs from other systems to the Log4j systems, this process could be exploited to attack the vulnerable host.
  • If you have a system not running Log4j but are using logic or a business layer between it and the back end system, it could process logs from a system not running Log4j and be impacted.

If you believe you have reason to take action on Log4j, here are three initial steps to take to detect, respond, and remediate your environment.

Identify Affected Devices

Remediation efforts should start with scanning your applications with your vulnerability scanner and identifying any internet-facing devices running Log4j. If installed and configured, you can also use your endpoint detection and response (EDR) technology to search for Log4j files in your environment. Next, you should check if you are using versions of Log4j that are vulnerable. Those are versions 2.0 to 2.14.1, inclusive. Version 2.15.0 is the first version with the fix. Any device running the vulnerable versions that are exposed to the internet is at risk of the exploit.

The vulnerable version of Log4j includes java-based applications and services that use the library directly, and a number of frameworks and components that reference it in their infrastructure are also at risk including Apache Struts2, Solr, Druid, Flink, and Swift frameworks.

You can view a list of vulnerable manufacturers and components here: https://github.com/NCSC-NL/log4shell/tree/main/software This page contains an overview of any related software regarding the Log4j vulnerability. On this page, NCSC-NL will maintain a list of all known vulnerable and not vulnerable software. Furthermore, any reference to the software will contain specific information regarding which version contains the security fixes, and which software still requires mitigation. Please note that this vulnerability may also occur in custom software developed within your organization. These occurrences are not registered in this overview. A list of artifacts that use the vulnerable version can be found here: https://mvnrepository.com/artifact/log4j/log4j/usages

Threat Detection, Threat Hunting, and Identification of IOCs

Awareness is critical for a threat like this, and you’ll want to ensure your Security Operations team is working together with your DevOps team and any managed security vendor teams you have on contract to include everyone who is aware of the affected assets. Your SOC should also create and prioritize alerts as “critical” for any devices using Log4j.

• You can check your logs for active scanning IP addresses associated with attackers and add them to your block list. A list can be found on Github here: https://raw.githubusercontent.com/CriticalPathSecurity/Public-Intelligence-Feeds/master/log4j.txt
• A list of addresses that are being called to download the exploit payload can be viewed here: https://raw.githubusercontent.com/CriticalPathSecurity/Public-Intelligence-Feeds/master/log4j.txt
• Microsoft is maintaining a list of IOCs here: https://github.com/Azure/Azure-Sentinel/blob/master/Detections/MultipleDataSources/Log4J_IPIOC_Dec112021.yaml
• A curated list of various IOC sources can be viewed here: https://github.com/curated-intel/Log4Shell-IOCs

Considering activity from CVE-2021-44228 has been identified dating back to December 1st and earlier, you will want to analyze log sources from at least several weeks back. Then, you can expand your threat hunting to scan for exploit activity that may have occurred before these dates.

Patch Systems

Version 2.15.0 of Log4j was recently released in response to the vulnerability and contains a fix, so the best course of action would be to upgrade vulnerable devices to version 2.15.0 immediately. If vendors have mitigations measures ready, work together to ensure you are taking a coordinated approach to incident response.

If you are unable to patch, there are manual ways you can protect your environment:

• If your version is 2.10 or above, you can add Java parameter -Dlog4j2.formatMsgNoLookups=true, which changes the system property log4j2.formatMsgNoLookups to true.
• If you are running an older version, from 2.0-beta9 to 2.10.0, then the immediate fix is to remove JndiLookup class from the classpath

The difficulty in these manual solutions is that they assume there is a complete handle on dependency chains across the board and not a variety of versions across apps. Trying to find every single instance of this library could lead to the error of a rogue Log4j instance being overlooked. If you are going this route, Deepwatch recommends a tool like log4j-jndi-be-gone (https://github.com/nccgroup/log4j-jndi-be-gone) to automatically mitigate the threat regardless of version.

That being said, the best way forward is still to patch to 2.15.0.

Update Firewall Rules and Security Infrastructure Rulesets

Another way to reduce your exposure is to update your next-gen firewall, web application firewall (WAF), and web proxy rules in order to block potentially dangerous requests. The JNDI lookups that are used during exploitation will contain a string similar to ${jndi:abc}, so you could add the prefix to your blocklist. Concurrently, Deepwatch Detection Engineers have observed a number of new user agent string variants being used; see https://gist.github.com/ZephrFish/32249cae56693c1e5484888267d07d39 to ensure you have accounted for all versions.

Unfortunately, there are no ways to obfuscate the appropriate strings; therefore, there’s no 100% foolproof way to detect attacks. Be aware that any filter you put in place could potentially be bypassed.

Keep up with applying any recommended updates from your WAF provider to have the latest rule sets in place. You can expect that rules updates will be released in the near future to account for any newly discovered exploitation techniques. Since there’s no magic bullet solution, we recommend planning ahead for an iterative process to mitigate any risk to your firewall associated with the Log4j vulnerability in the upcoming months.

What Happened with the Log4j Vulnerability?

On Friday, December 10, 2021, the Apache Software Foundation issued an emergency security update to the popular Java library Log4j that provides logging capabilities to address a zero-day vulnerability known as the Log4Shell attack. The vulnerability, tracked as CVE-2021-44228, had proof-of-concept code (PoC) disclosed December 9th on Twitter.

Read Deepwatch’s Significant Cyber Event report here: Log4j Zero-day With Proof-of-Concept Code and Active Scanning Gets Security Fix

Cisco Talos has since reported forensics evidence dating back to December 2nd, and potentially earlier, with researchers at Forrester noting that the vulnerability is trending in a pattern. “Scattered early incidents from unknown actors have been observed only by looking back in telemetry and demonstrate early adoption by coin miners and botnets… many actors with different objectives ranging from financial to espionage will rapidly adopt this exploit in the coming days to secure access either for immediate use, for resale or for long-term footholds,” noted Forrester Research.

As of Monday, December 13, 2021, Deepwatch, in addition to multiple sources in information security, this vulnerability has attracted an onslaught of threat actors actively scanning the internet for systems and applications that may be vulnerable to the Log4Shell attack. Due to the PoC being released and actively scanning for vulnerable servers, server owners will have a limited time to patch before they are actively exploited.

This vulnerability provides a threat actor with remote code execution (RCE) capabilities, and the Cybersecurity and Infrastructure Security Agency states in their Current Activity Advisory that “A remote attacker could exploit this vulnerability to take control of an affected system.” This vulnerability affects all versions from 2.0-beta9 to 2.14.1. Since the flaw is remotely exploitable and requires little technical skill to execute, this is a critical vulnerability with a severity score of 9.8 on the CVSSv3 severity scale.

For on-going lessons in hardening and protection from vulnerabilities like Log4j, the CIS Control 1: Inventory and Control of Enterprise Assets and CIS Control 2: Inventory and Control of Software Assets (https://www.cisecurity.org/) are key considerations for every Security Operations Center. Right now, it may be difficult to address these considerations when an environment is under attack. With a threat like Log4j, critical imperatives in the cybersecurity program, like vulnerability management and asset inventories, demonstrate why investments in cybersecurity risk management and resilience are mission-critical in times like these.

How Has Deepwatch Responded to the Log4j Vulnerability?

Deepwatch has been actively involved with customer support since the original PoC was published on Friday. Because this is an easily exploitable vulnerability, scans are coming from a large number of places, some are malicious, and some are blue-hat. Details are constantly updated since the early days of response to this global threat.

Customers with Managed Detection and Response, Endpoint Detection and Response, Firewall Management, and Vulnerability Management services were notified on Friday, December 10th, that this vulnerability was actively being addressed, by email and Deepwatch MOBILE notifications. On Saturday, December 11th, customers were notified that a custom detection rule had been released and enabled in their environments to mitigate the Log4j risk and find potential IOCs.

Depending upon the customer environment and services, customers were guided through coordinated responses to patch their systems. Meanwhile, with the vulnerability still actively being exploited, Deepwatch squads have been working closely with customers to ensure risks are managed and mitigated to protect and defend against Log4Shell-related compromises.

When the PoC was made publicly available, Deepwatch initiated proactive, advanced threat detection services. Deepwatch’s integrated Threat Operations team performed searches to identify signs of exploitation across all customer environments. This search included looking for IDS signatures related to Log4j as well as running an initial search to look for individuals scanning environments to find systems vulnerable to the log4j vulnerability. Based on the results of these queries, Squad Threat Hunters and Analysts began validating these results in their customers’ environments and expanding the search queries to find logs that initial queries might have missed.

Meanwhile, the Deepwatch Detection Engineering team worked quickly to make sure searches that might help us identify an uptick in scans/exploits such as critical IDS events, increases in IDS events from an external source, and Outbound TOR traffic. The team is actively developing new and emerging Use Cases specifically to detect exploitation attempts against the Log4Shell vulnerability. Customers have their own internal IOC watchlists that are actively monitored and updated in real-time in coordination with their Squads. For example, a Security Analyst reported over the weekend 45[.]155[.]205[.]233, which they saw passing base64 encoded commands to install a crypto-miner. That IOC has now been added to the Deepwatch internal Log4j IOC watchlist.

Deepwatch also pushed out a global detection rule to look for these Log4j exploit attempts with updated logic to account for all the variations of the exploit being observed and recorded. Squad detection engineers worked quickly to verify the detection rule worked within their customer environments. When it didn’t, our security engineers either resolved the underlying issue or created a local version of the rule tailored to their environment. The Deepwatch Endpoint Detection and Response, Vulnerability Management, and Firewall Services teams have been also actively engaged and working with customers. For Endpoint, our EDR experts are helping customers with their EDR consoles to run scans to check for the existence of Log4j in their environments. In one case, a brand new customer still onboarding has been provided additional support and an accelerated timeline to ensure they have coverage through this evolving incident. For more information and a list of IOCs to use for your Security Operations Center, visit https://www.horizon3.ai/cve-2021-44228/

Learn How Vulnerability Management Works with Managed Detection and Response

Tired of working on the weekend? If you have questions about Log4j and how to protect and defend your organization, contact Deepwatch today for a meeting with a Deepwatch Security Solutions Architect. We can help connect you to immediate IR services while helping make proactive plans to get ahead of the next zero-day vulnerability that will undoubtedly happen again.

One of our experts can discuss your Security Operations program and what you can achieve with a Managed Detection and Response provider that integrates Vulnerability Management, Firewall, and EDR services within a full suite of Managed Security Services. With Deepwatch Managed Security Services and an assigned Squad of experts working for you 24/7/365, you never have to go at a zero-day like Log4j alone again. With Deepwatch, we’re here to support you with rapid threat detection and response, and best in class technology and security expertise while giving you measurable results to demonstrate improvements in security maturity year over year.

Log4j Information Security Resources

The below security teams have developed some tools and methodology to assist with scanning for and remediating Log4j. Please use these resources with discretion.

• GitHub – fullhunt/log4j-scan: A fully automated, accurate, and extensive scanner for finding log4j RCE CVE-2021-44228 https://github.com/fullhunt/log4j-scan
• Monitoring for Log4j with Powershell – https://www.cyberdrain.com/monitoring-with-powershell-detecting-log4j-files/
• GitHub – Cybereason/Logout4Shell: Use Log4Shell vulnerability to vaccinate a victim server against Log4Shell https://github.com/Cybereason/Logout4Shell
• Burp Extension used for scanning for the log4shell vulnerability – https://blog.silentsignal.eu/2021/12/12/our-new-tool-for-enumerating-hidden-log4shell-affected-hosts/

↑