Malware persistence technology on F5 and Citrix load balancers

Over the past few years, hackers have targeted public-facing network devices such as routers, VPN concentrators, and load balancers to gain a foothold in corporate networks. While it is not uncommon to find remote code execution vulnerabilities in such devices, incidents in which attackers are able to deploy malware on these devices that can survive reboots or firmware upgrades are rare and often attributed to sophisticated APTs organize.

Because they use flash memory, which can degrade over time if written many times, embedded network devices often store their firmware in a read-only file system and load its contents on every reboot into RAM. This means that all changes and files generated by the various running services during normal operation of the device are temporary, as they only occur in RAM and are never saved to the file system, which when the device is restarted will be restored to its original state.

The exceptions are configuration files and scripts generated through the device management interface and stored in a limited storage area called NVRAM (non-volatile RAM). From an attacker’s perspective, this limitation makes it more difficult to compromise network devices in a persistent way, which is why large-scale attacks against home routers, for example, involve automated botnets that periodically rescan and reinfect Start the router.

However, in a targeted attack scenario against an enterprise network, attackers prefer to remain invisible rather than attack the same device multiple times so that they do not trigger any detection that might be implemented if the vulnerability is made public. They also prefer long-term access to such devices and use them as bridges into internal networks, as well as pivot points from which they can perform lateral movement and extend access to other non-public devices.

Persistence opportunities in Citrix, F5 load balancers

Since 2019, three critical vulnerabilities have emerged in Citrix and F5 load balancers (CVE-2019-19781, CVE-2020-5

902 and CVE-2022-1388), these vulnerabilities have been publicly documented and exploited in the wild, prompting warnings from the U.S. Cybersecurity and Infrastructure Security Agency (CISA) and other organizations. Because of this, researchers at firmware security firm Eclypsium recently investigated the persistence opportunities for attackers on such devices. Their findings were published in a report Wednesday.

In May 2022, security firm Mandiant reported that a cyberespionage threat actor—at the time identified as UNC3524, but later associated with the Russian state-run APT29 (Cozy Bear)—compromised enterprise networks and was compromised due to the deployment of a backdoor. The intrusion went undetected for an extended period of time on network devices, including load balancers that did not support detection tools such as Endpoint Detection and Response (EDR) and were running older versions of CentOS and BSD. While Mandiant did not name these devices or their manufacturers, Eclypsium researchers believe they are F5 and Citrix devices because the F5 load balancer runs CentOS and Citrix (formerly Netscaler) runs FreeBSD.

“One characteristic of UNC3524 stands out: their TTP is unreliable, they use modified open source software to build their backdoor, and appear to only know enough about the system to achieve the most basic goals,” Eclypsium researchers wrote in their paper said the report. Report. “Their implants were so unreliable that they installed network shells for the sole purpose of restarting them after they died. It was this feature that became the catalyst for this research, and the open question was: could it be done on a load balancer? Using an off-the-shelf C2 framework? Is the malware resilient enough to persist across reboots or even upgrades? Is it possible to infect a device so deeply that a clean wipe and reinstall is not enough?”

Many attack groups choose to use cracked versions of commercial attack frameworks such as Cobalt Strike or Brute Ratel, but Eclypsium researchers wanted something open source and easily used by less sophisticated attackers, so they chose Sliver, an open source Adversary simulation framework for their test implants. Sliver is written in Go, so it is cross-platform and provides rotation and tunneling capabilities.

To investigate what files the F5 load balancer retains during reboots and firmware upgrades, the researchers looked at the configuration backup feature available through the management interface, which can be used to generate an archive containing all configurations and settings that can later be taken on a fresh installation. Deployment. From an archive containing hundreds of files, the researchers selected three executable scripts and configuration files that could execute scripts on certain events.

“An unexpected finding in this study was vendor documentation; it turns out that these devices contained a wealth of information about undocumented features and functionality over the years,” the researchers said. “Thanks to the vendor, this research would have been much more difficult without documentation. It’s important to understand how the device handles its configuration files.”