TCP Stack Tuning

Recommandations to improve TCP performance on your nPerf server

Kernel

We recommend to use an up-to-date kernel. Versions 4.x or above are recommended.
If you are using CentOS or RHEL, you should consider to switch to Debian or Ubuntu in order to get a newer kernel.

TCP Illinois

Using TCP Illinois module can improve speed by managing congestion diffrently.
More information on Wikipedia's page : https://en.wikipedia.org/wiki/TCP-Illinois

Check if the module is available on your system :

$ ls /lib/modules/$(uname -r)/kernel/net/ipv4/tcp_illinois.ko
/lib/modules/4.4.0-135-generic/kernel/net/ipv4/tcp_illinois.ko

If the previous command returns the file path, it's available, we can load it :

echo "tcp_illinois" > /etc/modules-load.d/tuning-nperf.conf

Tell kernel to use tcp_illinois as default congestion control algorithm :

echo "net.ipv4.tcp_congestion_control=illinois" >> /etc/sysctl.d/tuning-nperf.conf
echo "net.core.default_qdisc=fq" >> /etc/sysctl.d/tuning-nperf.conf

TCP Buffers

Increase TCP buffers

echo "net.ipv4.tcp_rmem=4096 87380 33554432" >> /etc/sysctl.d/tuning-nperf.conf
echo "net.ipv4.tcp_wmem=4096 65536 33554432" >> /etc/sysctl.d/tuning-nperf.conf
  • The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always allocated to a TCP socket, even under high pressure on the system.
  • The second value specified tells the kernel the default receive buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols.
  • The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket.
echo "net.core.rmem_max=67108864" >> /etc/sysctl.d/tuning-nperf.conf
echo "net.core.wmem_max=67108864" >> /etc/sysctl.d/tuning-nperf.conf

This sets the max OS receive and send buffers sizes for all types of connections.

Increase backlog size

echo "net.core.netdev_max_backlog=10000" >> /etc/sysctl.d/tuning-nperf.conf

Disable TCP timestamps

echo "net.ipv4.tcp_timestamps=0" >> /etc/sysctl.d/tuning-nperf.conf