Log4Shell: the weekend we grepped every JAR

The first search came back with two files, both of them where I expected, and for about ten minutes on Friday morning I thought we were nearly done.

Some context for that search. After the Struts jar we’d forgotten about in 2017, we started running a monthly find across every machine for a list of library names we care about, which is crude but has caught a couple of things since. On Friday I added log4j-core to the list and ran it by hand. It found the jar inside Logstash on our log pipeline boxes, and inside Elasticsearch on the search cluster. Two products, both from Elastic, both Java, both obviously candidates. I wrote that in the channel, started reading Elastic’s advisory, and then one of our engineers asked whether the find would see a fat jar. It wouldn’t.

What the bug is

If you’ve somehow missed it: CVE-2021-44228, which people started calling Log4Shell within hours, was disclosed on Thursday with a working proof of concept, and Apache put out Log4j 2.15.0 with a fix. Log4j 2 has a feature called lookups, placeholders in a log message that it expands when it writes the line, for things like environment variables or the Java version. One lookup type uses JNDI, Java’s directory interface, which can go and fetch an object from a remote server, including over LDAP. In a lot of configurations that means it’ll download a Java class from a server the attacker chose and run it.

The part that makes it so bad is that the lookups were expanded in the message, the bit of the log line that contains whatever you passed in. So any Java service that logs a string an attacker controls, a username, a User-Agent header, a search query, a chat message in Minecraft (which is where a lot of the first public demos were), can be made to reach out to the attacker’s server and run their code. Pretty much every Java service logs request data somewhere, and Log4j is in a huge share of them, usually not as a direct dependency but brought in by something else.

I’m not going to go into exploitation beyond that, and you don’t need it to do the defensive work. What you need is to know where Log4j 2 is.

Finding it, properly

A find for log4j-core-*.jar finds the jars that are sitting on disk with their original filename. It doesn’t find Log4j when it’s been shaded, meaning its classes have been copied into another jar as part of a build, which is common for command-line tools and services that ship as one “fat” jar. It doesn’t find a jar nested inside a WAR or an EAR, or inside another jar, which Spring Boot does by default. And it doesn’t find anything inside a container image that isn’t running on that host.

So on Saturday morning an engineer and I wrote a script that walks a filesystem, opens every .jar, .war, .ear and .zip it finds, lists the entries, and recurses into any archive inside an archive. It flags anything containing the class org/apache/logging/log4j/core/lookup/JndiLookup.class, and reads the Log4j version out of the pom.properties file Maven leaves in the jar when it’s there. We pushed it to every instance through AWS Systems Manager, and then ran it against every image in our container registry, extracted to a scratch directory, because the registry is the only complete list of what our jobs actually run.

The result was Log4j 2 in nine places, which came down to four pieces of software:

• Logstash, on the three log pipeline boxes. Vulnerable version.
• Elasticsearch, on the three search nodes. Vulnerable version, but more on that below.
• The Gradle caches on our two CI runners, from building a Kotlin job. That’s build-time only and never handles outside input, so it was the lowest priority, though we cleared the caches anyway.
• The Kotlin job itself.

The last one is the one the find would never have caught, and it’s the one that scared me. Two years ago one of our engineers wrote a data import tool in Kotlin, because the library he needed for one customer’s file format only existed on the JVM. It builds as a single fat jar with Log4j’s classes shaded in under their original package names, and the jar’s filename is the name of the tool. It runs in a container on our job cluster, it processes files our customers upload, and when it hits a column it doesn’t recognize, it logs a warning that includes the column header. Customers write the column headers. It was running Log4j 2.11.

The weekend

Friday afternoon, before we’d found the Kotlin job, we did the things that were cheap and immediate. Our WAF provider pushed a managed rule for the pattern on Friday, and we turned it on, though people were already posting ways to write the string that slip past simple pattern matching, so it was never going to be enough on its own. On Logstash and Elasticsearch we set the log4j2.formatMsgNoLookups system property, which turns off message lookups in Log4j 2.10 and later, and restarted them.

The thing I’d recommend to anyone, if you only do one thing, is the egress rule. The attack needs the vulnerable server to make an outbound connection to the attacker. After SolarWinds last December we’d started locking down egress for the agents on our servers, but those rules were per agent. The log pipeline boxes themselves still had security groups allowing outbound traffic to anywhere, the default, when all Logstash actually needs to reach is the Elasticsearch cluster, S3 and our metrics service. We locked that down on Friday night, and it’s the change that most lets me sleep, because it makes the next bug in this family a lot less useful too.

Elasticsearch was less scary than it looked. Elastic’s advisory said that current 6.x and 7.x versions, running on a supported JDK, use the Java Security Manager in a way that blocks the remote code execution, leaving at most an information leak through a DNS lookup. Logstash had no such protection. We don’t send user-controlled strings to Logstash as log messages on purpose, but Logstash logs a warning when it can’t parse an event, and that warning includes part of the event. Our app logs User-Agent headers. So yes, that was reachable in principle.

The Kotlin job was found at about 2 p.m. on Saturday. We stopped the job queue, rebuilt the jar with the JndiLookup class removed from the shaded classes (the same fix Apache describes for the plain jar, deleting that one class), and had it running again by early evening. Then we went through its logs back to the start of the month, plus the logs of everything else on the list, for any of the lookup patterns. Plenty of attempts against our public endpoints starting Friday afternoon, a lot of them from scanners announcing themselves as researchers. Nothing in any log from the import job, and no outbound connections from the job cluster to anything unexpected in the flow logs. That’s the best I can say. It’s not a proof.

On Monday, Elastic shipped Logstash and Elasticsearch releases with Log4j upgraded, and we rolled those out. Apache released 2.16.0 the same day, which turns off JNDI by default and removes message lookups entirely, and the advice I was reading by Monday evening had started to move away from relying on the flag alone, toward upgrading or removing the class. The Kotlin job is on 2.16.0 as of last night.

The vendors

The part that isn’t done is everything we don’t run ourselves. On Friday we took the agent spreadsheet from December and widened it to every vendor whose software touches our data or runs on our machines: SaaS tools, the monitoring agent, the hosted CI, the support desk, our customer messaging tool, the BI tool from the Struts story, and so on. It has 37 rows. Each one gets an email asking whether they’re affected and what they’ve done, and a column for the answer. Some vendors had a status page up by Friday night with a clear yes-or-no, which is exactly what you want. Some sent a reassuring paragraph that didn’t actually say whether they use Log4j. A couple of the large ones have been updating a long list of products by the hour.

As of this morning, 15 rows still say “no response”.