Under heavy network load (high-throughput or low-latency services, databases, storage, video streaming, routing), default Linux network stack settings often lead to invisible packet drops (packet drops), jitter, and reduced throughput.

In this article, we will examine how packets travel through Linux buffers, how to diagnose bottlenecks, and how to correctly optimize parameters at all levels: from the network card to application sockets.

Network Stack Buffering Architecture

A packet passes through several buffering tiers:

[ Physical Cable / Network Environment ]
                │
                ▼
┌────────────────────────────────────────┐
│  1. NIC Hardware & RX Ring Buffer      │  (Managed via ethtool)
└────────────────────────────────────────┘
                │  DMA -> sk_buff, NAPI poll / SoftIRQ
                ▼
┌────────────────────────────────────────┐
│  2. Kernel Backlog Queue (netdev)      │  (net.core.netdev_max_backlog)
└────────────────────────────────────────┘
                │  Routing, Netfilter, IP stack
                ▼
┌────────────────────────────────────────┐
│  3. TCP/UDP Socket Receive Buffer      │  (net.core.rmem_*, tcp_rmem)
└────────────────────────────────────────┘
                │  recv() / read()
                ▼
┌────────────────────────────────────────┐
│  4. User Space Application             │  (nginx, postgres, custom app)
└────────────────────────────────────────┘

If at any of these stages the buffer overflows before the next layer can process the data, the packet is dropped by the kernel or the card.

1. Diagnostics: How to Identify Buffer Issues

Before changing configurations, collect baseline metrics.

Drops at the Network Card Level (NIC / Ring Buffer)

# View general interface statistics
ip -s link show eth0

# Detailed driver statistics (look for rx_dropped, rx_missed_errors, rx_fifo_errors, rx_no_buffer_count)
ethtool -S eth0 | grep -E -i "drop|discard|miss|fifo|overrun|error"

Drops at the Kernel and TCP Stack Level

# Drops in the netdev_max_backlog queue (second column /proc/net/softnet_stat)
cat /proc/net/softnet_stat

# Socket buffer overflow errors and TCP listen backlog
nstat -az | grep -E "TcpExtListenOverflows|TcpExtListenDrops|TcpExtTCPRcvQDrop|TcpExtTCPWmemDrop"

# Real-time socket monitoring
ss -ntip

2. Network Card Level: Ring Buffers (RX/TX Rings)

A Ring Buffer is a circular memory structure area (DMA descriptors) where the network adapter places incoming packets before CPU processing.

Checking Current and Maximum Sizes

ethtool -g eth0

Example output:

Ring parameters for eth0:
Pre-set maximums:
RX:             4096
TX:             4096
Current hardware settings:
RX:             512
TX:             512

Increasing Buffer Size

If Current is less than Pre-set maximums, under high loads it is recommended to increase the size to the maximum or a reasonable value (e.g., 2048/4096):

sudo ethtool -G eth0 rx 4096 tx 4096

Persistence: To ensure settings persist after reboot:

  • systemd-networkd: add a [Link] section with ReceiveBufferSize=4096 to your .network file.
  • Netplan: receive-buffer-size: 4096 parameter under the match block.
  • udev rule: /etc/udev/rules.d/10-ring-buffer.rules
    ACTION=="add", SUBSYSTEM=="net", NAME=="eth0", RUN+="/sbin/ethtool -G eth0 rx 4096 tx 4096"

3. Kernel Queue: netdev_max_backlog and somaxconn

When the network card generates a SoftIRQ, packets are placed into the kernel input queue if the driver does not use NAPI mode or packets arrive faster than the CPU can process them.

  • net.core.netdev_max_backlog — maximum number of packets in the kernel processing queue.
  • net.core.somaxconn — length of the pending connection queue (listen() backlog).
# Recommended values for high-load servers (10G/40G/100G)
sudo sysctl -w net.core.netdev_max_backlog=16384
sudo sysctl -w net.core.somaxconn=65535

4. Kernel Socket Buffers: rmem and wmem

BDP (Bandwidth-Delay Product) Calculation

For TCP, the transmission window size and socket buffer directly determine the maximum throughput on a link with a specific round-trip time (RTT):

$$\text{BDP} = \text{Bandwidth (bps)} \times \text{RTT (s)}$$

For example, for a 10 Gbps link with a 20 ms ping ($0.02\text{ s}$): $$\text{BDP} = 10 \times 10^9 \times 0.02 = 200,000,000\text{ bits} \approx 25\text{ MB}$$

If the maximum TCP buffer size (tcp_rmem / tcp_wmem) is less than 25 MB, the connection will physically be unable to utilize the full 10G bandwidth.

sysctl Configuration

Basic Network Kernel Limits (net.core)

# Maximum socket receive/send buffer sizes (in bytes)
net.core.rmem_max = 67108864     # 64 MB
net.core.wmem_max = 67108864     # 64 MB

# Default buffers
net.core.rmem_default = 262144   # 256 KB
net.core.wmem_default = 262144   # 256 KB

TCP Auto-tuning (net.ipv4.tcp_rmem and tcp_wmem)

Parameters accept three values: [min] [default] [max] (in bytes):

# min, default, max for incoming TCP buffers
net.ipv4.tcp_rmem = 4096 87380 67108864

# min, default, max for outgoing TCP buffers
net.ipv4.tcp_wmem = 4096 65536 67108864

# Enable Window Scaling (RFC 1323) — mandatory for buffers > 64KB!
net.ipv4.tcp_window_scaling = 1

Global TCP Memory Pool (net.ipv4.tcp_mem)

Specified in memory pages (typically 1 page = 4096 bytes): [low] [pressure] [high]. By default, the kernel calculates this based on RAM size, but on memory-constrained machines it’s worth checking:

sysctl net.ipv4.tcp_mem

5. Mitigating Bufferbloat: Qdisc and BBR

Excessive buffer increases lead to Bufferbloat: packets are not lost, but accumulate in huge queues, causing massive latencies (RTT jumps from milliseconds to seconds).

To prevent this:

  1. Use modern queue management algorithms: Fair Queueing (fq) or fq_codel / cake.
  2. Use BBR (Bottleneck Bandwidth and RTT) as the congestion control algorithm:
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Ready Configuration File

Create file /etc/sysctl.d/99-network-performance.conf:

# ====================================================================
# Linux Network Buffer and Stack Optimization for High-Load
# ====================================================================

# Maximum queue lengths
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384

# Maximum and default socket buffer sizes (bytes)
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.core.rmem_default = 262144
net.core.wmem_default = 262144

# TCP buffer auto-tuning: min default max (bytes)
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864

# Enable TCP Window Scaling and Timestamps
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

# Qdisc and congestion control (Bufferbloat mitigation)
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

# TCP SYN flood protection
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384

Apply parameters without rebooting:

sudo sysctl --system

Verification Checklist

  • Check ethtool -S for rx_dropped / rx_fifo_errors.
  • Increase network card Ring Buffer (ethtool -G rx <max> tx <max>).
  • Calculate BDP for primary server network routes.
  • Configure sysctl socket buffers and TCP window scaling.
  • Enable fq + bbr to reduce latency and eliminate Bufferbloat.
  • Perform load testing using iperf3 -P <threads> or wrk before and after changes.