[Solved-4 Solutions] How to Increase the maximum number of tcp/ip connections in linux - Linux Tutorial
Problem:
How to increase the maximum number of tcp/ip connections in linux ?
Solution 1:
Maximum number of connections are impacted by certain limits on both client & server sides.
Client side: Increase the ephermal port range, and decrease the tcp_fin_timeout
Default values to find out:
sysctl net.ipv4.ip_local_port_range
sysctl net.ipv4.tcp_fin_timeout
- The ephermal port range is defines as the maximum number of outbound sockets a host can create from a particular I.P. address.
The
fin_timeout
defines the minimum time these sockets will stay inTIME_WAIT
state. Usual system defaults are: - net.ipv4.ip_local_port_range = 32768 61000
- net.ipv4.tcp_fin_timeout = 60
- Basically the system cannot consistently to guarantee more than (61000 - 32768) / 60 = 470 sockets per second.
- To increase the availability by decreasing the
fin_timeout
. Suppose we do both, we should see over 1500 outbound connections per second, more readily.
To change the values:
sysctl net.ipv4.ip_local_port_range="15000 61000"
sysctl net.ipv4.tcp_fin_timeout=30
The above should not be interpreted as the factors impacting system capability for making outbound connections per second. But rather these factors affect system's ability to handle concurrent connections in a sustainable manner for large periods of "activity."
Default Sysctl
values on a typical linux box for tcp_tw_recycle
& tcp_tw_reuse
would be
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_tw_reuse=0
Don't allow a connection from a "used" socket (in wait state) and force the sockets to last the complete time_wait cycle.
sysctl net.ipv4.tcp_tw_recycle=1
sysctl net.ipv4.tcp_tw_reuse=1
- To allows fast cycling of sockets in time_wait state and re-using them.
- This change make sure that this does not conflict with the protocols to use for the application that needs these sockets.
- On the Server Side: The net.core.somaxconn value has an important role. It limits the maximum number of requests queued to a listen socket.
- If you are sure of your server application's capability, bump it up from default 128 to something like 128 to 1024.
- Now you can take advantage of this increase by modifying the listen backlog variable in your application's listen call, to an equal or higher integer.
sysctl net.core.somaxconn=1024
txqueuelen
parameter an ethernet cards also have a role to play. Default values are 1000, so bump them up to 5000 or even more the system can handle it.
ifconfig eth0 txqueuelen 5000
echo "/sbin/ifconfig eth0 txqueuelen 5000" >> /etc/rc.local
Similarly bump up the values for net.core.netdev_max_backlog and net.ipv4.tcp_max_syn_backlog. Their default values are 1000 and 1024 respectively.
sysctl net.core.netdev_max_backlog=2000
sysctl net.ipv4.tcp_max_syn_backlog=2048
Now remember to start both your client and server side applications by increasing the FD ulimts, in the shell.
Solution 2:
There are a couple of variables to set the max number of connections. Running out of file numbers first. To check ulimit -n. After that, there are settings in /proc, but those default to the tens of thousands.A single TCP connection ought to be able to use all of the bandwidth between two parties:
- To check TCP window setting is large enough. Linux defaults are good for everything except really fast inet link (hundreds of mbps) or fast satellite links and also find your bandwidth*delay product.
- Check for packet loss using ping to large packets (ping -s 1472 ...)
- Check for rate limiting. On Linux, this is configured with tc
- Confirm that the bandwidth is exists actually exists using e.g., iperf
- Confirm the protocol is sane.
- Connections is actually using (try netstat or lsof). If that number is substantial:
- Have a lot of bandwidth, e.g., 100mbps+. In this case, actually wants to up the ulimit -n. Still, ~1000 connections (default on my system) is quite a few.
- Some network problems to slowing down in connections (e.g., packet loss)
- Have something else slow down, e.g., IO bandwidth, especially in seeking. Have you checked iostat -x?
Additionally, using a consumer-grade NAT router (Linksys, Netgear, DLink, etc.), beware that may exceed its abilities with thousands of connections.
Solution 3:
To determine OS connection limit is by catting nf_conntrack_max.For example: cat /proc/sys/net/netfilter/nf_conntrack_max
Use the following script to count the number of tcp connections to a given range of tcp ports. By default 1-65535.
#!/bin/bash
OS=$(uname)
case "$OS" in
'SunOS')
AWK=/usr/bin/nawk
;;
'Linux')
AWK=/bin/awk
;;
'AIX')
AWK=/usr/bin/awk
;;
esac
netstat -an | $AWK -v start=1 -v end=65535 ' $NF ~ /TIME_WAIT|ESTABLISHED/ && $4 !~ /127\.0\.0\.1/ {
if ($1 ~ /\./)
{sip=$1}
else {sip=$4}
if ( sip ~ /:/ )
{d=2}
else {d=5}
split( sip, a, /:|\./ )
if ( a[d] >= start && a[d] <= end ) {
++connections;
}
}
END {print connections}'
Solution 4:
From server side:
- To check if load balancer works correctly.
- TCP timeouts turn slow into 503 Fast Immediate response, load balancer work correctly working resource to serve.
Using node server, you can use toobusy from npm. To implementation:
var toobusy = require('toobusy');
app.use(function(req, res, next) {
if (toobusy()) res.send(503, "I'm busy right now, sorry.");
else next();
});
From Client side:
- To group calls in batch, reduce the traffic and total requests number between client and server.
- Unnecessary duplicates requests handled by build a cache mid-layer.