Tomcat’s acceptCount and maxConnections

  network, tomcat

Order

Regarding tomcat’s parameters, it is easy to confuse the following parameters: acceptCount, maxConnections, maxThreads, minSpareThreads. Let’s make some clarification here.

Different levels

  • MaxThreads and minSpareThreads are the configuration parameters of tomcat worker thread pool. maxThreads is equivalent to maxPoolSize of jdk thread poo l, while minSpareThreads is equivalent to corePoolSize of jdk thread pool.

  • AcceptCount, maxConnections are tcp layer related parameters.

图片描述

Tomcat has an acceptor thread to accept socket connections, and then a worker thread to handle business. For a request from the client, the flow is as follows: tcp’s three-way handshake establishes a connection. During the process of establishing the connection, the OS maintains a semi-connection queue (syn queue) and a full connection queue (accept queue). After three-way handshake, the server receives an ack from the client, enters the establish state, and then the connection is moved from syn queue to accept queue. Tomcat’s acceptor thread is responsible for fetching the connection from the accept queue, accepting the connection, and then handing it over to the worker thread for processing (Read request parameters, processing logic, return response, etc.; If the connection is not kept alive, the connection is closed, and then the worker thread is released back to the thread pool; if it is kept alive, the next data packet is waited until keepAliveTimeout, and then the connection is closed and released back to the thread pool.), and then go to the accept queue to fetch connection (When the current socket connection exceeds maxConnections, the acceptorance thread itself will block waiting until the connection drops before processing the next connection in the accept queue.)。 AcceptCount refers to the size of this accept queue.

maxConnections(NioEndpoint$Acceptor)

This value indicates the maximum number of socket that can be connected to tomcat. NIO mode defaults to 10000.

// --------------------------------------------------- Acceptor Inner Class
    /**
     * The background thread that listens for incoming TCP/IP connections and
     * hands them off to an appropriate processor.
     */
    protected class Acceptor extends AbstractEndpoint.Acceptor {

        @Override
        public void run() {

            int errorDelay = 0;

            // Loop until we receive a shutdown command
            while (running) {

                // Loop if endpoint is paused
                while (paused && running) {
                    state = AcceptorState.PAUSED;
                    try {
                        Thread.sleep(50);
                    } catch (InterruptedException e) {
                        // Ignore
                    }
                }

                if (!running) {
                    break;
                }
                state = AcceptorState.RUNNING;

                try {
                    //if we have reached max connections, wait
                    countUpOrAwaitConnection();

                    SocketChannel socket = null;
                    try {
                        // Accept the next incoming connection from the server
                        // socket
                        socket = serverSock.accept();
                    } catch (IOException ioe) {
                        //we didn't get a socket
                        countDownConnection();
                        // Introduce delay if necessary
                        errorDelay = handleExceptionWithDelay(errorDelay);
                        // re-throw
                        throw ioe;
                    }
                    // Successful accept, reset the error delay
                    errorDelay = 0;

                    // setSocketOptions() will add channel to the poller
                    // if successful
                    if (running && !paused) {
                        if (!setSocketOptions(socket)) {
                            countDownConnection();
                            closeSocket(socket);
                        }
                    } else {
                        countDownConnection();
                        closeSocket(socket);
                    }
                } catch (SocketTimeoutException sx) {
                    // Ignore: Normal condition
                } catch (IOException x) {
                    if (running) {
                        log.error(sm.getString("endpoint.accept.fail"), x);
                    }
                } catch (Throwable t) {
                    ExceptionUtils.handleThrowable(t);
                    log.error(sm.getString("endpoint.accept.fail"), t);
                }
            }
            state = AcceptorState.ENDED;
        }
    }

CountUpOrAwaitConnection () here judges whether the current number of connections exceeds maxConnections.

acceptCount(backlog)

In the source code is the backlog parameter, the default value is 100. This parameter refers to the number of connections that can be accepted when the current number of connections exceeds maxConnections, that is, the size of tcp’s accept queue.

The backlog parameter prompts the kernel to listen to the maximum length of the queue. If the length of the listening queue exceeds the backlog, the server will not accept new client connections and the client will also receive an ECONNREFUSED error message. In Linux before kernel version 2.2, the backlog parameter refers to the upper limit of all socket in the semi-connected state (SYN_RCVD) and the fully connected state (ESTABLISHED) . However, since kernel version 2.2, it only represents the upper limit of a fully connected socket, while the upper limit of a semi-connected socket is defined by the/proc/sys/net/IPv4/TCP _ max _ syn _ backlog kernel parameter.

TCP three-way handshake queue

  • Socket wait queue on client side:
    When shaking hands for the first time and establishing a semi-connection state: when the client sends a SYN packet to the server through connect, the client will maintain a socket queue. if the socket waiting queue is full, the client will return to connection time out. as long as the client does not receive the SYN+ACK for the second handshake, the client will send it again 3 seconds later, and if it still does not receive it, it will continue to send it 9 seconds later.

  • The server-side semi-connection queue (Syn queue):
    At this time, the server will maintain a syn queue, and the length of the semi-connected SYN queue is MAX (64,/proc/sys/net/IPv4/TCP _ max _ SYN _ backlog). The tcp_max_syn_backlog value of the machine is configured under/proc/sys/net/IPv4/TCP _ MAX _ SYN _ BACKLOG. When the server receives the SYN packet from the client, it will perform a second handshake to send the SYN+ACK packet for confirmation. The client’s TCP protocol stack will wake up the socket waiting queue and issue a connect call.

  • The server-side full connection queue (Accpet queue):
    When the first three-way handshake, when the server receives the ACK, it will enter a new queue called accept. The length of the queue is min(backlog, somaxconn). By default, somaxconn has a value of 128, indicating that there are 129 ESTAB connections waiting for Accept (), The value of the backlog should be specified by the second parameter in intlisten (intsockfd, intbacklog). the backlog in list can be defined by our application.

NioEndpoint’s bind method

@Override
    public void bind() throws Exception {

        serverSock = ServerSocketChannel.open();
        socketProperties.setProperties(serverSock.socket());
        InetSocketAddress addr = (getAddress()!=null?new InetSocketAddress(getAddress(),getPort()):new InetSocketAddress(getPort()));
        serverSock.socket().bind(addr,getBacklog());
        serverSock.configureBlocking(true); //mimic APR behavior
        serverSock.socket().setSoTimeout(getSocketProperties().getSoTimeout());

        // Initialize thread count defaults for acceptor, poller
        if (acceptorThreadCount == 0) {
            // FIXME: Doesn't seem to work that well with multiple accept threads
            acceptorThreadCount = 1;
        }
        if (pollerThreadCount <= 0) {
            //minimum one poller thread
            pollerThreadCount = 1;
        }
        stopLatch = new CountDownLatch(pollerThreadCount);

        // Initialize SSL if needed
        initialiseSsl();

        selectorPool.open();
    }

Socket (). bind (addr, getbacklog ()); The backlog of is the acceptCount parameter value.

Tcp’s Semi-connected and Full-connected Queues

When the accept queue is full, even if the client continues to send ACK packets to the server, they will not be responded to. At this time, the server decides how to return by/proc/sys/net/IPv4/TCP _ abort _ on _ overflow, 0 means to directly drop and discard the ACK, 1 means to send RST notification to the client; Accordingly, client will return read timeout or connectionletby peer respectively.

图片描述

In general, we can see that in the whole TCP connection, our Server side has the following two queue:

  • One is the semi-connection queue: (SYN QUEUE) QUEUE (MAX (TCP _ MAX _ SYN _ BACKLOG, 64)), which is used to store SYN_SENT and SYN_RECV information.

  • The other is a full connection queue: accept queue (min (SOMAXCONN, BACKLOG)), which saves the state of ESTAB, so after the connection is established, the thread of our application service can accept () to handle the business requirements.

SYN attack

In the process of three-way handshake, after the Server sends SYN-ACK, the TCP connection before receiving the ACK from the Client is called half-open connect. At this time, the Server is in SYN_RCVD state. After receiving the ACK, the Server goes into ESTABLISHED state. SYN attack is that the Client forges a large number of nonexistent IP addresses in a short period of time, and sends SYN packets to the Server continuously. The Server replies to the acknowledgement packets and waits for the Client’s acknowledgement. Since the source address does not exist, the Server needs to retransmit the packets until it times out. These forged SYN packets will occupy the unconnected queue, causing normal SYN requests to be discarded because the queue is full, thus causing network congestion and even system paralysis. SYN attack is a typical DDOS attack. The way to detect SYN attack is very simple, that is, when there are a large number of semi-connected states on the Server and the source IP address is random, it can be concluded that SYN attack has occurred. Use the following command to make it current:

netstat -nap | grep SYN_RECV

About maxConnections and maxThreads

MaxConnections indicates how many socket are connected to tomcat. NIO mode defaults to 10000. MaxThreads is the maximum number of concurrent requests processed by the woker thread. In other words, although the client’s socket is connected, it may all be in tomcat’s task queue, waiting for the worker thread to process the return response.

Summary

Tomcat server has added a layer of protection to the request connection based on the size setting of tcp’s accept queue, that is, the size limit of maxConnections.

When a large number of requests from the client end come in, firstly, the accept queue of tcp in OS layer helps to block them. if the accept queue is full, the subsequent connection cannot enter the accept queue and cannot be handled by the working thread. the client will get an error of read timeout or connection reset.

The second layer of protection is to buffer in the acceptor thread. When the socket connected exceeds maxConnections, it will block and wait to control the speed of the connection from acceptor to the worker thread, and wait for the worker thread to process the response client.

doc