Gracefully Close nginx in kubernetes

  docker, nginx


All three are to end/terminate the process.

1.SIGINT SIGTERM difference

The former is associated with the character ctrl+c, while the latter has no control character association.
The former can only end the foreground process, while the latter is not.

2. the difference between sigtermsigkill

The former can be blocked, handled and ignored, but the latter cannot. The default signal sent without parameters for the KILL command is SIGTERM. let the program exit well. Because it can be blocked, when some processes cannot be ended, use kill to send the latter signal. That is: kill-9 process number.


Docker stop and dockkill

docker stop

When we use the docker stop command to stop the container, docker will allow the application in the container 10 seconds by default to stop running.

When the docker stop command is executed, the process with PID 1 in the container (main process) Send the system signal SIGTERM, and then wait for the application program in the container to terminate execution. If the waiting time reaches the set timeout, or the default 10 seconds, the system signal SIGKILL will continue to be sent to kill the process forcibly. The application program in the container can choose to ignore and not process the SIGTERM signal, but once the timeout period is reached, the program kill be forcibly killed by the system, because the SIGKILL signal is directly sent to the system kernel, and the application program has no chance to process it.

docker kill

By default, the docker kill command does not give the application in the container any chance to gracefully shutdown. It will directly issue a SIGKILL system signal to forcibly terminate the operation of the program in the container.

The difference with the docker stop command is that docker kill does not have any timeout setting. It will directly send SIGKILL signals and other signals specified by the user through the signal parameter.

Docker stop command is more similar to kill command in Linux system, both of which send system signal SIGTERM. The docker kill command is more like kill -9 or kill -SIGKILL command in Linux system, which is used to send SIGKILL signal and forcibly terminate the process.

On Processes with pid 1

Processes with process ID of 1, usually UNIX init processes, play an important role in the operating system: whenever a process exits, if it still has child processes, the child processes become orphan processes and the init processes take over. Unix is designed in such a way that the parent process must explicitly “wait” for the child process to terminate in order to collect its exit status.

After the child process is finished, the kernel will still maintain a basic structure for it, saving its pid, exit reason and status information, which can be obtained by the parent process through waitpid system call. If the parent process does not call waitpid, the status information will remain and become a so-called zombie process. If the child process ends after the parent process, in general, the init process will be responsible for these orphan processes.

According to the general principle that a container only runs one process, for a web service, the pid when it runs in the container is 1. suppose it calls bash’s cgi script, and this script calls grep. after a period of time, the cgi script times out for some reason, and the web service starts to try to kill it, but grep is not affected, so it becomes an orphan process. At this time, the web service with pid=1 should be able to take over, but the vast majority of web services do not have the capability of init, so grep becomes a zombie process.

Pod shutdown mechanism of kubernetes

Pod represents processes running on nodes in a cluster, so that these processes are no longer needed. It is important to exit gracefully (and roughly end with a KILL signal, leaving applications no chance to clean up). The user should be able to request deletion and know it when the room process is terminated, and also ensure the deletion is finally completed. When a user requests to delete pod, the system records the desired graceful exit period. Pod is not allowed to be forcibly killed before this, and the TERM signal will be sent to the main process of the container. Once the deadline for graceful exit has passed, the KILL signal will be sent to these processes and pod will be deleted from the API server. If Kubelet or container manager restarts while waiting for the process to end, the end process will be retried with a complete graceful exit period.

An example process:

  • 1. The user sends a command to delete Pod. The default graceful exit time is 30 seconds.

  • 2. Pod update time in 2.API server, after which Pod is considered dead

  • 3. Within the client command, Pod is displayed as “Terminating”

  • 4. (at the same time as step 3) when Kubelet saw that Pod was marked as exiting, because the time in step 2 had been set, it started the process of pod shutdown.

    • 4.1 If the Pod defines a hook before stopping, it will be called inside the pod. If the hook is still running when the graceful exit time expires, the second step will be called with a very small graceful break.

    • 4.2 The process is signaled as TERM

  • 5. (Concurrent with Step 3) Pod is removed from the service list and is not considered part of the running pod. The slowly shut-down pod can continue to serve the outside world when the load balancer removes them in turn.

  • 6. When graceful exit time expires, any process running in pod will be sent SIGKILL signal and killed.

  • 7.Kubelet will complete the deletion of pod and set the graceful exit time to 0 (indicating immediate deletion). Pod is removed from API and is no longer visible to clients.

By default, the graceful exit time for all delete operations is within 30 seconds. The kubectl delete command supports the option of –grace-period= = to run the user to modify the default value. 0 means that deletion is executed immediately, and a new pod such as deleting pod from API immediately will be created at the same time. On the node, pod, which is set to end immediately, will still give a very short graceful exit period before being forced to kill.

Nginx and SIGTERM

There are two ways to signal the running Nginx process to fully manage the operation: use the -s option of Nginx process or directly use the system command kill to signal the master process. When the -s option is used, nginx will automatically find the running master process ID(master process is responsible for receiving and processing signals, and completes different management operations for all working processes according to different signals).

SIGINT, like SIGTERM, is used to force the Nginx process to exit. When the master process receives the forced exit signal, it will send the forced exit signal to all working processes. if the working process fails to exit in time, master will use the timer to repeatedly send the forced signal, and SIGALRM signal will be sent when the timer is triggered. SIGIO signal is explicitly ignored by Nginx; SIGCHLD signal tells the master process that the working process has exited and needs to complete resource recovery or restart the working process.

Nginx process exits in two ways

  • When master process receives SIGQUIT signal
    Forward this signal to the worker process. The worker process then closes the listening port so as not to receive new connection requests, closes idle connections, and calls ngx_worker_process_exit to exit after waiting for the active connections to reach full normal junction speed. The master process calls ngx_master_process_exit function to exit after all working processes exit.

  • When master process receives SIGTERM or SIGINT signal
    Forward the signal to the worker process. The work process directly calls ngx_worker_process_exit function to exit. The master process calls the ngx_master_process_exit function to exit after all worker processes exit. In addition, if the worker process fails to exit normally, the master process will wait for 1 second and then send a SIGKILL signal to force the worker process to terminate.

Nginx’s graceful exit

 -s signal      Send signal to the master process. The argument signal can be
                one of: stop, quit, reopen, reload.

                The following table shows the corresponding system signals.

                stop    SIGTERM
                quit    SIGQUIT
                reopen  SIGUSR1
                reload  SIGHUP

among them
Stop-quick shutdown
Quit-graceful exit, exit after executing current request
Reload-Reload the configuration file
Reopen—Reopen the log file

Use kubernetes Lifecycle Hooks to gracefully exit nginx

kind: Deployment
  name: nginx-demo
  namespace: scm
    app: nginx-demo
  replicas: 1
        app: nginx-demo
      - name: nginx-demo
        image: library/nginx-demo
        imagePullPolicy: IfNotPresent
              # nginx -s quit gracefully terminate while SIGTERM triggers a quick exit
              command: ["/usr/local/openresty/nginx/sbin/nginx","-s","quit"]
          - name: PROFILE
            value: "test"
          - name: http
            containerPort: 8080

Off topic

How to gracefully close java applications

command: ["/bin/bash", "-c", "PID=`pidof java` && kill -SIGTERM $PID && while ps -p $PID > /dev/null; do sleep 1; done;"]