Guide to exotic techniques
When developing a software, in order to extend the software life cycle, a matching software is needed to monitor the released software. With the maturity of container technology, it becomes easier and easier to customize the system and package the software. At the same time, monitoring containers has become a necessary skill for container users. Next, the author will show you the resource management tool Cgroups for containers.
When we talk about container monitoring, we first think of using Cadvisor, Docker stats and other methods to obtain the monitoring data of the container, and at the same time we think that the container uses Cgroups to limit the resources in the container. But where do these data come from and how are they calculated? The answer is Cgroups. Recently, I was writing the docker container monitoring component, and found that the data in depth Cadvisor and Docker stats source code came from Cgroups. Besides understanding, I took notes of Cgroups.
Introduction to Cgroups
CGROPS is the abbreviation of control groups, and is a mechanism provided by Linux kernel to limit, record and isolate the physical resources used by process groups. At first google engineers proposed it and later it was integrated into Linux kernel. Therefore, Cgroups provides the basic guarantee for the virtualization of containers and is the cornerstone for building a series of virtualization management tools such as Docker and LXC.
02. Cgroups Effect
- Resource limiting:
- CGROPS can limit the total amount of resources used by process groups. If memory usage limit is imposed on a specific process, OOM will be triggered when the limit is exceeded.
- Priority allocation: Through the number of CPU time slices allocated and the IO bandwidth of the hard disk, it is actually equivalent to controlling the Prioritization the process.
- Accounting): Cgroups can count the system’s resource usage, such as CPU usage time, memory usage, etc. This function is very suitable for billing.
- ControlCgroups: You can suspend, resume, and other operations on process groups.
03. Cgroups Composition
Cgroups is mainly composed of Task, CGROP, Subsystem and hierarchy. The following are the respective concepts.
- Task: In Cgroups, task is a process of the system.
- Cgroup: Resource control in CGROUP is implemented in CGROUP units. Cgroup refers to a task group that is divided according to a certain resource contr ol standard and includes one or more subsystems. A task can join a cgroup or migrate from one cgroup to another.
- Subsystem: subsystem: Cgroups is a Resource Controller. For example, the CPU subsystem can control CPU time allocation and the memory subsystem can limit cgroup memory usage.
- Hierarchy: hierarchy is formed by a series of cgroup arranged in a tree structure, and each hierarchy schedules resources by binding the corresponding subsystem. Cgroup nodes in hierarchy can contain zero or more child nodes, which inherit the attributes of the parent node. The whole system can have multiple hierarchy.
Relationships between components
Subsystems, hierarchies, controlgroup and Tasks have many rules, which are described below:
1. One or more subsystem can be attached to the same hierarchy.
As shown in the following figure, the cpu and memory subsystems (or any number of subsystems) are attached to the same hierarchy.
2. A subsystem can only be attached to a hierarchy.
As shown in the following figure, the cpusystem has been attached to the hierarchy A, and the memory subsystem has been attached to the hierarchy b. Therefore, cpusubsystem cannot be attached to hierarchy B.
3. Every time the system creates a new hierarchy, all task on the system constitute the initialization cgroup of the new hierarchy by default. This cgroup is also called root cgroup. For each hierarchy you create, a task can only exist in one cgroup, that is, a task cannot exist in different CGroups of the same hierarchy, but a task can exist in multiple CGroups in different hierarchies. If a task is added to another cgroup in the same hierarchy during the operation, it will be removed from the first cgroup.
As shown in the following figure, cpu and memory are attached to the hierarchy of cpu_mem_cg. Net_cls is attached to net_cls hierarchy. And httpd process is added to both cg1 cgroup of cpu_mem_cg hierarchy and cg3 cgroup of net hierarchy. The httpd process is limited in cpu,memory and network bandwidth by two hierarchy subsystem.
4. When any task (process in Linux) fork in the system creates a child task (sub-process), the child task will automatically inherit the relationship of the parent task cgroup, in the same cgroup, but the child task can be moved to other different CGROUs as required. The parent-child task is independent of each other.
As shown in the following figure, the httpd process is in /cg1 cgroup of cpu_and_mem hierarchy and writes PID 4537 to tasks of the cgroup. After that, httpd(PID=4537) Process fork A child process httpd(PID=4840) and its parent process are in a unified cgroup of the same hierarchy, but since the relationship between the parent task and the child task is independent, the child task can be moved to other CGroups.
04, Cgroups use
We directly use shell commands to directly operate hierarchy and set cgroup parameters. The tools provided by libcgroup can also be directly used on centos6 to simplify the use of cgroup.
Create a Hierarchy
Use the shell command to create a hierarchy and attach subsystems to the hierarchy. Create a mount point for hierarchy as root. And the mount point contains the name of cgrou.
Next, use the mount command to mount the hierarchy and attach one or more subsystem to the hierarchy.
If you want to attch or detach subsystem on the existing hierarchy, you can use remount operation, for example, we want to detach memory subsystem.
Unmounting a Hierarchy
You can unmount an existing Hierarchy directly by using the umount command:
Creating Control Groups
Use the shell command mkdir directly to create a child cgroup:
Setting Control Cgroup Parameters
Use echo command to insert 0-1 to cpuset.cpus in group1 to restrict tasks in the cgroup to run only on cpu core of 0 and 1. As follows:
Moving a Process to a Control Group
Just add the PID of the process you want to restrict to the tasks file of the desired cgroup. For example, put the process with PID=1701 into the cgroup of “/cgroup/cpu_and_mem/group1/”.
05, Subsystem Introduction
- The blkio: blkio subsystem controls and monitors task’s I/O access to block devices in cgroup. For example, restrict access and bandwidth.
- Cpu: Mainly limits the cpu utilization rate of the process.
- Cpuacct: cpu usage reports of processes in cgroup can be counted. Cpuset: You can allocate independent cpu and memory nodes for processes in cgroup.
- Memory: automatically generates a report on the memory resources used by the task in the cgroup, and restricts the memory use of the task of the cgroup. devices:
- You can control whether a process can access certain devices. net_cls:
- Marking network packets with a Class Identifier (clssid) allows the Linux Traffic Control Program (tc) to identify packets generated from a specific cgroup. freezer:
- You can suspend or resume processes in cgroup. Ns: Processes in different cgroup can use different namespace.
06. The container uses Cgroups for resource restriction
Whether you use docker run to create containers directly or use various container orchestration tools (e.g. Kubernetes) to create containers, the nature of container restriction is Cgroups. We use these two methods to create containers and observe cgroups:
Use docker run to create containers
1. Limiting CPU share and creating two containers will create two sub-cgroups under /sys/fs/cgroup/cpu/docker/, the host running the container, respectively, in the following format.
2. Create a container and set the –cpu-shares parameter to 1024*10.
See the contents of the cpu.shares file for this container cgroup as follows.
3. Create a container and set the –cpu-shares parameter to 1024*14.
See the contents of the cpu.shares file for this container cgroup as follows.
4. The two containers use the stats of cpu. The relative cpu calculation time of one container to 14 cores and the relative cpu calculation time of the other container to 10 cores are as follows:
Limit container memory usage
1. Create a container and limit the maximum amount of memory the container can use to 1024M m.
2. Check the stats of the container memory, and the memory usage rate is 100%.
3. When the amount of memory used by the container exceeds 1024M, the container will be killed-9.
Use Kubenetes Container Orchestration Tool to Create Containers
For containers created by kubernetes orchestration tool, cgroup associated with the container is under /sys/fs/cgroup/cpu/kubepods/ running the host of the container, with the specific format as follows:
Use Pod to create a container, and the corresponding yaml file content is as follows:
Looking at the cgroup information of the container on the host computer running the container, it will be observed that cpu.shares is 1 core and memory.limit_in_bytes is 2G.
About 360 Technology
360 technology is a technology sharing public number created by 360 technology team, pushing technology dry goods every day.
For more technical information, please pay attention to “360 technology” WeChat public number