Evacuation in cloud environments or how to go beyond the container

Today, there is an active growth in the popularity of cloud technologies, and with them, virtual services. People appreciate the convenience and prospects of their use in comparison with classic boxed solutions. And one of the indispensable elements present in the entire cloud infrastructure will be containers. They provide a fairly large number of advantages for ordinary users, but here we cannot exclude the possibility of, so to speak, "escape".

Most of the containers that are used in practice today have access to the Internet. And this in itself poses a fairly high threat to confidentiality and security of work. It is possible that some external hacker will be able to access the container through the lowest level privileges. And this will give him a lot of opportunities to exit it. And there are quite a few options here, starting from diagnostics to identify vulnerabilities and up to detecting gaps in configurations.

That is, escapes from the container are a fairly serious risk to the security of both individual users and entire organizations. It is this vulnerability that hackers can use to gain access to personal computers and other devices. In practice, such a situation has already been encountered more than once, which has forced specialists to actively work in this area in order to ensure sufficient user protection indicators. We also suggest studying such a hacker attack as account substitution, understand what it is and how it can be prevented.

As part of today's review, we will dwell in more detail on what a container is as such, why it is needed and what is going beyond. We will briefly consider how this cloud technology works, get acquainted with the capabilities and namespace. We will consider 3 examples of the most frequently used hacker attacks in practice as part of going beyond the container and how they work, methods of their detection using EDR systems. The information presented will allow you to more thoroughly navigate this cloud technology, increase your level of knowledge in the context of countering the actions of Internet intruders. So, first things first.

What is a container: getting to know the concept itself

In this case, a container is a group of processes running as a single application in an isolated user environment, but at the same time capable of sharing the same kernel space. Unlike classic virtual machines, where absolutely the entire host is transferred to a virtual environment, combined solutions can be found here. Yes, in some cases, the container uses visualization. But still, the main emphasis is on using an isolated user space.

One of the most significant advantages of containers is that they allow you to use several separate systems on one server. At the same time, an isolated process tree, file system, network stack and many other elements characteristic of the classic user space will be formed for each of them. This was implemented in practice thanks to the use of Namespaces, that is, a namespace. This is a special mechanism provided by the operating system. We will talk about it in more detail today.

If we talk directly about isolation inside the container, then in this case we mean the presence of an application of its own adapted environment. That is, those softwares that previously a priori could not work together, will now receive a personal environment inside their own container. And there can be a large number of such containers on one server.

Thanks to such a solution, it was possible to implement the ability of each individual block to interact with a personal set of components taken from the user space. These are the products that are abstracted from the host and are in an isolated space. Properly organized work inside each container leads to the fact that applications can work here as if they had their own dedicated server. This aspect made containers so popular when working with microservices.

High portability is another significant advantage of this technology. It is due to the fact that each container will contain the entire set of dependencies that are required to ensure operation. They can be executed without problems in any system. The main thing is that it works under the control of an environment capable of supporting the execution of the container.

But, despite all its high-tech, there are also vulnerabilities. The fact is that when dividing the same kernel into separate components, high levels of isolation from the classic user mode of the host are not always ensured. This aspect can be called the most vulnerable element of the container. It is often through it that various types of cyber attacks are carried out, as a result of which hackers manage to go beyond the isolated container environment. In this case, we are talking about escaping, that is, exiting the container.

To minimize all possible risks and problems when working with containers, it is important to understand how they function.

Features of container operation: what you need to know about it

We would like to immediately draw your attention to the fact that in this case we are considering the use of the Linux operating system. Today, it uses such technological solutions that make it ideal for ensuring the functioning of containers. In particular, in Linux, each process that will be created inherits all the attributes that were characteristic of, so to speak, the parent process. This will also include, among other things:

• not clearly defined environment variables;
• permissions;
• capabilities;
• namespaces.

All of these mechanisms will also be used by containers when forming a separate isolated process tree. And here such a term as container runtime appears. This will be the application that takes on the execution of container orchestration, including starting processes and configuring all the attributes necessary to set restrictions and ensure the isolation of both a single process and all those additional tasks that will be executed in parallel. After this, the process is automatically transformed into init. Now it becomes possible to execute commands that are written in the configuration file of the corresponding container.

In most cases, the container runtime environment is used through additional applications. This can be an orchestration system or a CLI. That is, here they act as an additional link in interaction with the container environment. The most popular of these solutions include Docker Engine, Dockerfile, and Kubernetes. The first two are CLI, and the latter are orchestration systems.

To provide all the conditions necessary to create process isolation, the container environment can use various attributes, such as credentials, Linux security modules, capabilities, secure computing, control groups, and namespaces. Such attributes are typical for most modern container engines.

To better understand how a container works, we will describe the features of two key solutions necessary to create container isolation, as well as privilege restrictions. We are talking about “Capabilities” and “Namespace”.

“Capabilities” attribute

“Capabilities” attribute, that is, Capabilities ” This is a kind of range of actions that a running process can perform. The Linux operating environment is able to divide all privileges related to superusers into separate blocks. This is the Capabilities attribute. You can enable or disable it as needed, and independently of each other. The operating environment needs to limit a number of processes in its work. Various tools are used here, and not just users and groups. The Capabilities attribute is used to perform such tasks. It is able to forcibly limit operations that can also be taken on by all processes endowed with root privileges. Among other things, this also includes such solutions as chown and ptrace - some of the key elements in the array of root operations. Their execution in practice can be easily kept under control thanks to the use of the Capabilities attribute.

The logic of operation here is quite simple: as soon as you remove this attribute, you will be able to perform the corresponding operation. This will be relevant even for those processes that have root privileges. That is, the container engine will be able to absolutely safely implement the code within the block, where even such privileges exist, by strategically removing the corresponding capabilities from all those processes that are necessary for the formation of the container itself. It was possible to implement this possibility in practice due to the presence of the inherited capabilities mechanism in the Linux operating system, which we have already discussed above.

One of the main difficulties here is that administrators are technically unable to remove all privileged capabilities during the process of creating containers. This is exactly what attackers use in practice. All those capabilities that remain active can be used by unscrupulous individuals to implement various options and methods for exiting the container. In this case, they are based on specific capabilities that are available for the implementation of processes located inside the container itself.

The "Namespace" attribute

The previous attribute defined what a particular process can do, but the "Namespace" (Namespaces) already indicates where exactly all these actions can be implemented. That is, in essence, this attribute is what will provide all processes running at the parent and child levels with the ability to interact as if they had their own exclusive instances directly in the global resource.

Today, there are different types of Namespaces. Moreover, each of them will be responsible for its own specific type of global resource in the operating system. One of the simplest solutions here in terms of understanding will be the process ID namespace, that is, PID. It works in the following sequence:

1. While the administrator or software creates a new namespace, the operating system assigns the PID 1 status to the process responsible for it.
2. After that, the PID 2 status will be assigned to the child process, etc.
3. The number of statuses can be different: PID 3, PID 4, PID 5, and so on for each of the child processes.

In reality, everything will work approximately as follows. Let's assume that you have a process that runs with root privileges and is endowed with some ability like CAP_KILL (allows you to bypass checks when sending signals). As a result, we get a solution that will allow this workflow to bypass the corresponding permission checks in almost every process. But if all this is transferred to a new Namespaces PID, then there will already be restrictions on the termination of processes within a certain namespace. In this case, all other operations outside of it will not exist at all for your main process, endowed with CAP_KILL.

That is, in this case, the namespace is a kind of mechanism designed to form high isolation rates, possessing a number of additional functions, which will prevent the transition to other spaces and any unauthorized access.

Escape from the container: getting acquainted with the features with examples

In essence, escapes from the container are something similar to how applications are executed inside this very block on the host system. But practice shows that not all of the exit methods that exist today correspond to this statement. Escape scenarios quite often also include the cybercriminal himself. Attackers commonly use containers to access data from the host or to increase access privileges.

Now we will talk in more detail about the methods of exiting containers that attackers most often use in practice. Let's consider these attacks using the following examples:

1. User mode helpers.
2. Privilege escalation using SUID.
3. Log Mounts.

We will describe each of the options in detail, indicating what it is and how it can be detected.

What is an exit from a container User mode helpers

In this case, a kernel function called call_usermodehelper is used (user mode helper literally sounds like a user mode assistant). It is capable of preparing and directly launching a user mode application from the kernel. As a result, this kernel will be able to implement any program in user mode with elevated privileges. By the way, users will also be able to force a driver or any other element of the kernel mode to perform similar actions, and with the same elevated privileges, but only under certain conditions.

An unpleasant moment is that attackers can make the kernel independently launch the programs they need, endowed with root privileges, in order to create them or make adjustments to the corresponding files directly in user mode. This requires root access. But if an attacker gains full control over a container that already has these same elevated privileges or vulnerabilities, he can use them for his own purposes, then he can easily bypass all these restrictions and get everything he expected.

As part of such an attack, an auxiliary user mode method based on cgroup and the corresponding release_agent file can also be used. This tactic can also help exit the container. In this case, an attacker with root rights can use an auxiliary user mode method and make adjustments to the resources specific to a particular process. As a result of his actions, restrictions on the use of these very resources are triggered. Here, the technique that was presented at Black Hat USA back in 2019 is used. And it was intended to exit the cgroup release_agent. As soon as the attackers enabled the corresponding agent, they had the opportunity to execute the program to clean up the group.

To implement this, the following work is carried out:

• A directory is created and mounted. A control group is assigned to it.
• Another directory is created inside the cgroup. As a result, another new group is created.
• The contents of the notify_on_release file are set to position "1". Thanks to this, a special user mode mechanism is launched. By default, it will already be in each new group.
• The absolute path to the executable files is set directly through the release_agent document. It is stored directly in the root directory within each group, but will be the same as for all cgroups. An attacker will be able to easily obtain the absolute path to this directory by requesting the corresponding file from the container.
• The group is cleaned up. To do this, it is enough to write «0» in the cgroup.procs document. The executable file that was previously specified in release_agent will still be executed even if the group was initially empty.

If in practice hackers use other methods within this technology, then all of them will still work on the same principle to exit the container. Here it is necessary to raise that the main problem is hidden in the ability to make adjustments to related files directly from within the container. And this allows you to execute all programs on the host system that have root rights.

If you compare all the methods that are used to exit the container by popularity and ease of implementation, then this option will be the simplest. Moreover, almost every implementation here will be successful.

How can attacks carried out by the user mode assistant method be detected?

In order not to miss such an attack, it will be necessary to use a systematic approach, including:

1. A complete catalog of all call_usermodehelper calls used by the kernel. You will need to constantly compare them to identify third-party ones.
2. Pay attention to which calls have been manipulated programmatically through user-mode files.
3. Check if it is possible in your case to measure all files contained in the container in order to execute the assigned program.
4. Monitor the files regularly. Only then will it be possible to identify changes in the associated files related to each of the helpers. In this case, special attention should be paid to the changes that are observed in the user mode programs of the container.

In any case, monitoring should be constant and comprehensive. This is the only way to reduce potential risks and ensure decent security indicators, including those associated with the use of usermodehelper in containers.

What is a container exit by escalating privileges using SUID?

This technology assumes that you can work directly on your hosts with root privileges. That is, an attacker who already has limited permission on the host will be able to implement a program with root rights directly from the container. This will not allow a full exit from the node, since the hacker must initially have access to the host.

The main problem here is that the SUID/GUID bit, which is required for a file located in the container, will be able to retain its rights and permissions even outside of it. The main condition here is that the container itself operates in the same user namespace as the host itself. Unfortunately, this setup is quite standard for most modern container environments.

This means that to implement it, you need to have a container that functions the same way as root in a similar user namespace with the host. There must also be a directory here that can be accessed directly from the host and from the container. The presence of a shell on the host and a shell in the container will also be mandatory.

In practice, this technology is implemented as follows:

• the attacker creates an executable file in the directory that already exists, directly from the container or through the host;
• the SUID bit is added from within the container;
• the SUID binary file is launched for execution from the external environment.

Once these actions are completed, the hacker will be able to launch the executable file on the host, where root privileges are provided. If the prerequisites we discussed above are met, it will be quite easy to implement such an attack. There is no technical complexity here, even for a person who is not particularly well versed in programming. What can we say about hackers then?

How can an attack using SUID be detected?

In order to detect such an attack variant, which involves a partial exit from the container, in a timely manner, it is necessary:

1. Create a monitored file that can be executed.
2. Modify the SUID/GUID bit in order to be able to identify chmod actions that will occur inside the container.
3. Execute the executable file, but outside the container in order to identify in case of execution on the host of a file with the SUID/GUID bit set. Relevant for users without root rights.

It is necessary to conduct such checks on a regular basis. This is the only way to avoid serious problems when working in a cloud environment.

What is a container exit using a Log Mounts attack?

The first thing we want to draw your attention to is that this attack can only be implemented within a Kubernetes pod. With its help, a hacker gains access to read absolutely any directory or document stored on a host with root privileges. To use this technology in practice, you need to:

• have access to the pod where /var/log is added to the host directory;
• have the ability to read logs from both regular users and directly through the Pod service account using the Kubernetes interface;
• in most cases, you can access logs directly from within the pod by adding the /var/log host.

This technology is based on the fact that Kubernetes is able to access Pod logs. Each of them will have a log file in /var/log that is directly linked to the log file located in var/lib/docker/containers, that is, directly in the container directory. It turns out that Kubelet will read the contents of the Symlink without even checking its purpose. This is exactly what the hacker uses to gain access to the etc/shadow host file. It simply manipulates Symlink values from the log files stored in /etc/shadow.

But the attack does not end there. Then, using tools present directly in the Kubernetes command line, an HTTP POST request is generated to gain access to the logs. As a result, the hacker can gain access to the entire file system where root rights are present.

This idea is quite complex in implementation, requiring a comprehensive and consistent approach. But in practice, it can be implemented, which is what unscrupulous individuals with sufficient knowledge and skills take advantage of.

How can you detect an attack carried out using the Log Mount method?

Today, there are 2 effective ways to detect this attack:

1. Continuous monitoring of HTTP requests. Here you need to track all those requests that involve gaining access to read logs. Next, you need to set up a filter system to understand where they come from. This will help you identify unusual ways of receiving requests. If this approach observes changes to the symbolic legal link of the log, this method of detecting attacks will not work.
2. Identifying the process of changing or creating a Symlink. This is done based on data from the host directory /var/log. Initially, you should make sure that you are working directly with the host, and not with files inside the container.

In practice, the highest results in detecting hacker attacks are achieved by combining both methods in practice.

Summing Up

In today's review, we got acquainted with the principle of containerization, explained what an exit from a container is and provided possible escape options. All this allows us to confidently state that the risk of such attacks is quite high today. And if we take into account the fact that cloud technologies are increasingly used in practice, all these tools will improve and develop, which will entail an expansion of threats. To minimize potential risks, it is necessary to perform comprehensive monitoring of the operation of your cloud structures and more.

We would also like to draw your attention to a useful and easy-to-use tool for ensuring high security and privacy of work on the network - mobile proxies from the MobileProxy.Space service. Follow the link https://mobileproxy.space/en/user.html?buyproxy to study in detail the functionality of this solution, evaluate current tariffs, get acquainted with the variety of available geolocations to bypass regional bans established at the legislative level. Also, with their help, you can use programs that automate actions, including when working with social networks without the risk of running into a bam and other restrictions from the system.

We would also like to draw your attention to the fact that on the MobileProxy.Space website you can use additional services to find out your IP-address, check your Internet speed, port availability, etc. Don't forget to take advantage of the free 2-hour testing to see the benefits of this solution, its wide functionality and ease of use. If you encounter technical difficulties, please contact the support service, which operates around the clock.

Share this article: