Major vulnerability in log4j – Immediate action required by all systems & web administrators and others

Saturday, December 11th, 2021

Posted in General News, Incidents, Security

*See below for updates as of 1/3/2022, 3:00 pm

A critical vulnerability has been discovered in log4j that is actively being exploited. This is an issue both for systems and web administrators on campus, including those who support products with a web interface, as well as requiring the attention of those that manage relationships with Software as a Service (SaaS) vendors. 

This is particularly critical because: 

  • If a system is vulnerable the exploit can grant full system access.   
  • The vulnerability requires a low level of skill to exploit.   
  • The exploit can be sent over https where we may not be able to inspect the encrypted traffic or block the port. 
  • We are blocking this attack pattern at our campus edge.  We are seeing a significant volume of attempts. 
  • Our vulnerability scanner will allow us to detect the presence of vulnerable log4j installations if local account access for the scanner has been configured. We will be sharing the scanner results with systems and application administrators as we get it.
  • The package may be installed and built locally, making detection via package managers like rpm more difficult. 

If you manage the relationship with a SaaS vendor or run a vended product with a web interface, you should contact the vendor for more information about how they are addressing the vulnerability immediately.  If you get notified by a vendor of a data breach, contact Incident Response Team (irt@bu.edu).  An e-mail template for contacting your vendors has been provided below for your convenience. 

It is important that all administrators of web services review their installations for the presence of log4j and update it as needed. 

  • Update to version 2.15.0 that fixes the vulnerability.
  • Older versions, such as 1.x, are not supported by the vendor.  There are some recommendations on the web for things that might work, but we cannot guarantee any of them. Update 12/13: The vulnerability does not impact version 1 of log4j by default, however log4j version 1 is vulnerable to other moderate risks that will never be patched by the original authors.

In addition, if you are running a vulnerable version you should review your web server logs for indication of compromise.  You can search your logs using the egrep command below to find exploit attempts.  If you find entries that appear to have succeeded based on return codes, please reach out to the Incident Response Team (irt@bu.edu). 

  • egrep -i -r ‘\$\{jndi:(ldap[s]?|rmi|dns):/[^\n]+’ {insert your access log file name(s) here} 

Advisories and Technical Discussions 

 

Some vendor references: 

Email Template for Contacting Vendors 

As you may be aware, there is a major exploit in the wild targeting the Apache log4j library. We are requesting a status update from you as to if we are vulnerable or not and if so, what are you doing to address the situation. Can you confirm if our data has been breached and if so, what was the date and time of the incident?  Please forward copies of the logs to me and our incident response team at irt@bu.edu . Please note the date, time, and time zone of the logs when submitting them. 

Update: 12/13/2021, 10:30am 

Based on information gathered since our initial posting, we would like to clarify the following information: 

  • The vulnerability largely impacts software packages, not endpoint operating systems like Windows and MacOS.  We are not yet seeing critical updates for these operating systems.  However, applications you use on these devices may require updates.  It would be a good time to ensure you have automatic updates enabled for your phones and tablets, and to check for updates on your commonly used applications, including games. 
  • While the easiest attack vector today is http(s), anything that uses log4j v2 to log messages that contain variables under the control of the user is vulnerable. 
  • If you have installed an application that provides a user interface over the network, such as a web portal, or are supporting a Software as a Service (SaaS) solution, you need to contact your vendor to determine vulnerability and apply patches as needed. 
  • If you have developed your own application using log4j v2, you need to upgrade.   
  • The vulnerability does not impact version 1 of log4j by default, however log4j version 1 is vulnerable to other moderate risks that will never be patched by the original authors 
  • Some vendors, including RedHat, have patched these older vulnerabilities as part of their distributions.  You will need to obtain an update from the vendor.  
  • If you have an unpatched log4j installation you should still upgrade to log4j v2.15 (please note this requires Java 8) or newer but you may do so at a lower priority.   

We have also received updates to our vulnerability scanner that will now allow us to detect the presence of vulnerable log4j installations if local account access for the scanner has been configured.  We will be sharing the scanner results with systems and application administrators as we get it. 

Additional Resources: 

Update: 12/14/2021, 05:00pm 

Based on information gathered since our last update, we would like to share the following information:

  • Apache announced today that CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations which could result in a denial of service (DOS) attack. It is advised that you update to Apache Log4j 2.16.0 if possible. For more information as well as additional mitigation options, please visit https://logging.apache.org/log4j/2.x/security.html .

Update: 12/15/2021, 04:45 pm 

Based on information gathered since our last update, we would like to share the following information:

  • Windows Server Admins, please note that our vulnerability scanner vendor continues to improve their detection methods for identifying vulnerable hosts so please re-check your scan results at least daily until things stabilize. We will publish another update here when we stop seeing daily updates from the vendor.

Update: 12/16/2021 1:00pm

Based on information gathered since our last update, we would like to share the following information:

  • Apache announced that the previously announced mitigation “In Version 2.10.0 and newer you can set a property called formatMsgNoLookups.” has been discredited since its announcement. It was discovered that these measures only limit exposure while leaving some attack vectors open, and should be discontinued. Please refer to the Apache Security page for the latest information: https://logging.apache.org/log4j/2.x/security.html

Update: 12/17/2021, 4:00pm

  • Additional Remote Exploit found in log4j v2.15 Following the release of version log4j 2.15, a second condition was detected which could result in a Denial of Service Attack prompting the release of version 2.16.  It has subsequently been shown that version 2.15 can be exploited to execute remote code and updating to version 2.16 is now required.Version 1.x remains immune to these issues, though it is vulnerable to other issues as noted in previous versions of this advisory.If you are responsible for maintaining a product or service that uses log4js v2 and you applied updates in the last week to get to version 2.15 you will need to update again to version 2.16.You should reach out to appropriate vendors, monitor advisories, or obtain an updated copy of the code and deploy it to your systems as soon as possible to prevent them from being compromised.Additional Resources:

Update 12/20/21, 10:00am

Update 1/3/2022, 3:00pm

  • On 12/28/2021, Apache issued a 4th patch to fix another remote code execution bug. Upgrade to Log4j 2.3.2 (for Java 6), 2.12.4 (for Java 7), or 2.17.1 (for Java 8 and later).

Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code. This issue is fixed by limiting JNDI data source names to the java protocol in Log4j2 versions 2.17.1, 2.12.4, and 2.3.2.

Reference: https://logging.apache.org/log4j/2.x/security.html