Security researchers recently investigated IBM Cloud’s database-as-a-service infrastructure and discovered several security issues that allowed them to access internal servers used to build database images for customer deployments. The demonstrated attack highlights some common security oversights that can lead to supply chain compromise in cloud infrastructure.
The attack, developed by researchers at security firm Wiz, combines a privilege escalation vulnerability in the IBM Cloud Databases for PostgreSQL service, cleartext credentials scattered throughout the environment, and overly lax internal network access controls that allow lateral movement within the infrastructure. 4gdtu
PostgreSQL is an attractive target in cloud environments
Wiz’s audit of IBM Cloud Databases for PostgreSQL is part of a larger research project that analyzes PostgreSQL deployments by major cloud providers that offer the database engine as part of their managed database-as-a-service solutions. Earlier this year, Wiz researchers also discovered and disclosed vulnerabilities in the PostgreSQL implementation of Microsoft Azure and Google Cloud Platform (GCP).
[Advance your career with top security certifications: What they do, how much they cost, and what you need. | Sign up for the CSO newsletter. ]
The open source PostgreSQL relational database engine has been developed for more than 30 years with a focus on stability, high availability, and scalability. However, this complex software was not designed with a permissions model suitable for multi-tenant cloud environments, where database instances need to be isolated from each other and from the underlying infrastructure.
1 second of 28 secondsVolume 0%
PostgreSQL has powerful features that allow administrators to make changes to the server file system and even execute code via database queries, but these operations are insecure and need to be restricted in a shared cloud environment. At the same time, other management operations such as database replication, creating checkpoints, installing extensions, and event triggers need to be available to customers for the service to function properly. That’s why cloud service providers (CSPs) must come up with workarounds and make modifications to PostgreSQL’s permissions model to enable these features, even if customers only operate with limited accounts.
Privilege escalation via SQL injection
While analyzing the IBM Cloud’s PostgreSQL implementation, Wiz researchers looked at the logical replication mechanisms available to users. This functionality is implemented using several database functions, including a function named create_subscription, which is owned and executed by the database superuser named ibm.
When they inspected the code for this function, the researchers noticed a SQL injection vulnerability caused by improper sanitization of the parameters passed to it. This means that they can pass arbitrary SQL queries to the function and the function will execute them as the ibm superuser. Researchers exploited this vulnerability via a PostgreSQL COPY statement to execute arbitrary commands on the underlying virtual machine hosting the database instance and opening a reverse shell.
Through a shell on a Linux system, they start doing some reconnaissance to understand their environment, such as listing running processes, checking active network connections, checking the contents of the /etc/passwd file that lists system users, and running a port scan on Linux . The internal network discovers other servers. The widespread port scanning attracted the attention of IBM’s security team, who contacted the Wiz team to inquire about their activity.
“After discussing our work and sharing our ideas with them, they kindly allowed us to continue our research and further challenge the safety boundaries, which reflects the healthy safety culture of the organization,” the Wiz team said.
Stored credentials lead to supply chain attacks
The information gathered, such as environment variables, told the researchers that they were inside a Kubernetes (K8s) pod container, and after searching the file system, they discovered that the K8s API access token was stored locally in a file named /var/run/secrets/kubernetes middle. io/serviceaccount/token. The API token allows them to gather more information about the K8s cluster, but it turns out that all pods are associated with their account and run under the same namespace. But it’s not a dead end.
K8s is a container orchestration system for software deployment, where containers are typically deployed from images – pre-built packages that contain all the files required to run the container and its pre-configured services. These images are typically stored on a container registry server, which can be public or private. In the case of IBM Cloud, it is a private container registry that requires authentication.
The researchers used API tokens to read the configuration of the pods in their namespace and found access keys to four different internal container registries in these configuration files. A description of the newly discovered key in IBM Cloud’s Identity and Access Management (IAM) API indicates that it has read and write access to the container registry, which would allow researchers to overwrite existing images with malicious ones.
But later I discovered that the key description was inaccurate and I could only download images. This level of access had security implications, but did not pose a direct threat to other IBM Cloud customers, so the researchers moved forward.
Container images can contain a lot of sensitive information that is used during deployment and subsequently deleted, including source code, internal scripts that reference other services in the infrastructure, and the credentials required to access them. Therefore, the researchers decided to download all images from the registry service and use automated tools to scan them for secrets such as credentials and API tokens.
“To comprehensively scan the secrets, we decompressed the images and examined the combination of files that make up each image,” the researchers said. “Container images are based on one or more layers; each of them may contain secrets unintentionally. For example, if a secret exists in one layer but is removed from the next layer, it will be completely invisible inside the container. Therefore, scanning individually Each layer may reveal more secrets.”
The JSON manifest file for a container image has a “history” section that lists the historical commands executed during the build process of each image.In several of these files, researchers found passwords passed to them as command line arguments.
Lessons from other organizations
While all of these issues have been privately reported to and fixed by the IBM Cloud team, they are not unique to IBM. According to the Wiz team, the “dispersed secrets” problem is common in all cloud environments.
Automated build and deployment workflows often leave secrets in various places such as configuration files, Linux bash history, log files, etc., which developers forget to clear after deployment is complete. Additionally, some developers accidentally upload their entire .git and CircleCI configuration files to the production server. Forgotten secrets that the Wiz team often uncovers include cloud access keys, passwords, CI/CD credentials, and API access tokens.
Another common issue that plays a key role in IBM Cloud attacks is the lack of strict access controls between production servers and internal CI/CD systems. This often allows attackers to move laterally and gain a deeper foothold within an organization’s infrastructure.
Finally, private container registries can provide attackers with extensive information beyond credentials. They can reveal information about critical servers inside the infrastructure, or they can contain code that reveals other vulnerabilities. The Wiz team says organizations should ensure their container registry solutions implement appropriate access controls and scoping.