Advanced Linux kernel optimization with sysctl

Informatec Digital » Resources » Advanced Linux kernel optimization with sysctl

If you've been managing Linux servers for a while, you'll eventually discover that the difference between a mediocre system and one that flies under heavy load lies in how well you've tuned the kernel. It's not just about installing more RAM or a more powerful CPU: many bottlenecks are resolved by adjusting a few carefully chosen parameters.

The kernel exposes hundreds of network, memory, file system, and process scheduling options that can be modified on the fly. With the tool sysctl and some adjustments in /procIt is possible to get the most out of the hardware and achieve lower latency, higher throughput, and better stability This applies to web servers and databases, as well as real-time environments and Kubernetes. However, it's essential to know exactly what you're doing and how to measure it.

What is sysctl and how does it organize kernel parameters?

Utility sysctl It is the gateway to changing kernel parameters without needing to recompile or reboot, allowing read and modify values ​​at runtime to test configurations instantly. These parameters are organized in a low hierarchical tree /proc/sys/and each one controls a specific behavior of the system.

The settings are grouped into distinct categories, making it easy to pinpoint what you need to adjust when you're experiencing a specific performance or stability issue. This structure allows you to focus, for example, solely on network (net.*), memory (vm.*), or file system (fs.*) without getting lost in the rest of the kernel options.

The main categories you will find in /proc/sys/ These include, among others:

• kernel.*: general kernel options (scheduler, PID, messages, NUMA, etc.).
• vm.*: virtual memory management, swap, caches and overcommit policies.
• net.*: IPv4/IPv6 network stack, TCP/UDP buffers, queues, congestion control.
• fs.*: limits of descriptors, inodes, inotify, AIO and VFS behavior.
• dev.*: specific parameters of certain devices and drivers.

With sysctl -a You can list all available parameters, and with commands like sysctl vm.swappiness o sysctl net.ipv4.tcp_tw_reuse reading specific values, which is key to audit the current configuration before changing anything. To locate something specific, it's common to filter with grep, For example: sysctl -a | grep tcp.

Modifying a parameter on the fly is as simple as using sysctl -w clave=valorHowever, it's important to understand that these changes are temporary. To save them after a reboot, it's essential to back up the configuration to /etc/sysctl.conf or in files of /etc/sysctl.d/, usually using numbered files (for example 10-network.conf, 20-memory.conf, 99-custom.conf) that determine the order in which they are applied.

Measure before you touch: benchmarks and system baseline

Before you start fiddling with dials, it's wise to establish an objective performance baseline . Without prior data, it's impossible to know if the changes have improved or worsened the system, and you'll fall into the classic "it seems better to me" trap, which is useless for making sound decisions.

At a system-wide level, it's a good idea to collect CPU, load, memory, disk, and network statistics for a few minutes under a representative load. Tools such as uptime, mpstat, vmstat, iostat -x o sar -n DEV They allow you to see how the stock kernel behaves, how much swap it uses, how disk I/O responds, and whether the network is saturated or wasting bandwidth.

In addition to that overview, it's crucial to run specific benchmarks for the application you want to optimize. On web servers, you can use ab (Apache Bench) or similar tools for measuring requests per second, average latency, and errorsIn databases, utilities such as mysqlslap o sysbench They help evaluate transactions per second and query times.

For example, in a scenario without tuning, it's common to find figures like 500 HTTP requests/s with a 200 ms average response time, around 250 SQL queries/s with 40 ms latency, a network speed of approximately 500 Mbps, and a system load nearing its limit under stress. These numbers will be used to verify, after applying kernel changes, whether a measurable increase in throughput and a reduction in latency are actually achieved.

Finally, document the commands and results before making any changes. Having a history will allow you to accurately compare different batches of adjustments and will also help you detect regressions after a kernel update or after introducing a new sysctl profile in production.

Advanced memory settings: vm.*

Memory is one of the subsystems where kernel changes are most noticeable. A server with plenty of RAM, but poorly configured, can end up unnecessarily using swap space, causing... I/O spikes and drastic performance dropsThat's why parameters like vm.swappinessDirty page ratios or VFS cache behavior are just as important.

Parameter vm.swappiness This controls how much memory pages the kernel sends to swap. In many distributions, it's set to 60, a value intended for mixed desktop environments. For database servers or low-latency applications, it's usually preferable to lower it to 10 or even less, resulting in a more efficient system. Prioritize keeping data in RAM and reduce swappingIt's not uncommon to see disk traffic plummet and applications respond with much less jitter after this adjustment.

Another fundamental building block is vm.dirty_ratio y vm.dirty_background_ratioThese values ​​determine what percentage of memory can be filled with dirty pages before the kernel starts flushing them to disk. Excessively high values ​​can lead to large bursts of writes, with huge latencies for all applications when the flush occurs. Reducing these percentages to figures like 15 and 5, respectively, typically results in a more consistent and predictable I/O pattern.

The setting of vm.vfs_cache_pressure This defines how aggressively the kernel purges inode and directory caches. A default value of 100 tends to purge the cache quickly, while settings around 50 allow metadata to be retained longer, increasing the hit rate for file operations and improving performance. the perceived performance of the file system, something key in servers that handle millions of small files.

For its part, vm.min_free_kbytes Reserve a minimum amount of free memory to ensure the system can respond to bursts of allocation without experiencing sudden out-of-box (OOB) situations. Sizing this at approximately 0,5% to 1% of total RAM is usually an effective safety measure to prevent the kernel from running out of space under extreme load and starting to kill critical processes, thus improving performance. overall host stability.

Finally, the overcommit policy deserves attention: parameters such as vm.overcommit_memory y vm.overcommit_ratio They determine how much virtual memory is allocated to processes relative to available RAM and swap. In some cases (for example, with Redis or certain databases) aggressive overcommitment is allowed, while in others a more conservative policy is preferred to limit the risk of OOM and situations of excessively compromised memoryFor techniques for analyzing and diagnosing complex cases, consult memory debugging in Linux.

Advanced network optimization: net.* and TCP/IP

The network stack is perhaps the area where tuning is most noticeable in production environments. A simple change in connection queues or buffers can make the difference between a server that chokes at 500 Mbps and one that It easily saturates the gigabit.The parameters under net.core.* y net.ipv4.* In this area, they are your best allies.

To begin, socket buffer sizes directly influence the maximum throughput of a connection, especially when there are high latencies or high-capacity links. Adjust net.core.rmem_max y net.core.wmem_max to generous values ​​(tens or hundreds of megabytes) and define correctly net.ipv4.tcp_rmem y net.ipv4.tcp_wmem (minimum, default value and maximum) allows each socket to have the space needed to cushion traffic bursts without falling into artificial restrictions imposed by very conservative factory values.

Another key point is the size of the connection queues. Parameters such as net.core.somaxconn, net.core.netdev_max_backlog y net.ipv4.tcp_max_syn_backlog These limits define how many pending connections the system can handle before dropping packets or rejecting handshakes. On high-traffic web servers, increasing these limits can prevent saturated queues and drastically reduce load. connection errors during peak loads.

The behavior of TCP connections can also be fine-tuned with a multitude of options: enable net.ipv4.tcp_window_scaling To support large windows on high-bandwidth links, enable net.ipv4.tcp_sack y net.ipv4.tcp_timestamps to improve loss and retransmission management, or adjust net.ipv4.tcp_fin_timeout and the management of TIME_WAIT (net.ipv4.tcp_tw_reuse, net.ipv4.tcp_max_tw_buckets) in order to control resource consumption on servers with millions of short connections per minute.

The choice of congestion control algorithm also matters. Although cubic It remains the default value in many distributions; changing to bbr in modern kernels (4.9 and later) by net.ipv4.tcp_congestion_control y net.core.default_qdisc=fq It can multiply TCP throughput in environments with complex losses or latencies, achieving improvements of between 2 and 25 times compared to classic configurations in certain scenarios, at the cost of somewhat different behavior in congested networks.

We must not forget safety and reliability parameters such as net.ipv4.tcp_syncookies, the handling of redirects (net.ipv4.conf.all.accept_redirects, send_redirects) or filtering in bridges (net.bridge.bridge-nf-call-iptables in Kubernetes environments). Properly configured, they allow you to protect the system from common attacks (SYN flood, spoofing, etc.) without sacrificing a solid network performanceFor a practical guide on rules and detection, check out the Netfilter and Suricata implementation.

File system, descriptors, and global limits: fs.* and ulimit

On modern servers, a low file descriptor limit is the fastest way to break a system during peak hours. When a web server, reverse proxy, or database encounters the classic "too many open files" error, it's usually because the kernel parameters and user limits weren't scaled for the actual load.

Globally, fs.file-max This specifies how many descriptors the kernel can have open in total. For applications with thousands of simultaneous connections, it's common to increase this value to several million, along with an increase of fs.nr_openwhich establishes the hard ceiling for descriptors per process. Additionally, adjustments are needed. /etc/security/limits.conf so that users (or services managed by systemd) have soft and hard values ​​consistent with the kernel configuration.

The inotify subsystem, widely used by IDEs, web development tools, and file monitoring systems, is controlled by parameters such as fs.inotify.max_user_watches y fs.inotify.max_user_instancesIf you encounter errors related to watchers or processes that stop monitoring disk changes, you probably need Increase these limits to avoid silent bottlenecks.

Another relevant feature is asynchronous I/O (AIO), whose global maximum is defined with fs.aio-max-nrIn applications that rely heavily on asynchronous I/O (certain database engines, queuing systems, etc.), increasing it prevents the kernel from running out of slots for AIO operations under a massive request load.

In addition to these parameters, the performance of the disk subsystem is affected by the I/O scheduler configured on each block device. In systems with SSDs or NVMe, it is usually recommended to use lightweight schedulers such as none o mq-deadline, while on mechanical discs other options might make sense, such as bfq depending on the type of load. Adjust the scheduler through /sys/block/<disco>/queue/scheduler and fine-tune options such as queue depth (nr_requests) or the read_ahead_kb can make a noticeable difference in read/write latency and throughputIt is also important to consider which file systems and drivers are used; for example, articles on NTFSplus on Linux and other systems can help when designing heterogeneous loads.

Finally, it should not be forgotten that these kernel limits must go hand in hand with system configurations such as those of ulimit and systemd units, to ensure that services can actually use the defined additional resources and are not blocked by restrictions in upper layers.

Kernel general, NUMA, CPU affinity, and huge pages

Beyond networking, memory, and file systems, the kernel itself offers a range of tuning options that influence how processes are scheduled, how workloads are distributed across cores and NUMA nodes, and how memory is managed at scale. On modern machines with many cores and even multiple sockets, these details often make the difference between a balanced server and one with some cores saturated while others are underutilized . It's also worth exploring alternative kernels like the Liquorix kernel in environments where a different latency profile is desired.

Parameters like kernel.pid_max They establish the maximum range of process IDs that the system can assign, which is relevant for nodes running a large number of containers or ephemeral processes. Adjusting kernel logging options (kernel.printk) helps reduce excessive noise in dmesg while still recording critical events, which also indirectly influences the diagnostic capability and stability.

Regarding memory topology, the parameter kernel.numa_balancing It controls the extent to which the kernel automatically moves pages between NUMA nodes. Disabling it in certain environments allows for more explicit management of process and memory affinity, using tools such as numactl to ensure that CPU and RAM-intensive processes remain on the same node, reducing the latency resulting from remote accessTo better understand the interaction between hardware and core scheduling, it is also helpful to review technologies such as Intel Thread Director.

The huge pages (vm.nr_hugepagesThese are another key component in database systems or other workloads that handle large regions of contiguous memory. Reserving an appropriate number of giant pages (2 MB or even 1 GB, depending on the architecture) can reduce TLB overhead and significantly improve the performance of applications that run many operations on large blocks of memoryFor this to be effective, the kernel configuration must be coordinated with the application itself, so that it uses that memory explicitly.

In the field of latency and scheduling, there are scheduler parameters such as kernel.sched_min_granularity_ns, kernel.sched_wakeup_granularity_ns o kernel.sched_migration_cost_nsThese settings fine-tune the size of the CPU "chunks" allocated to each task and the aggressiveness with which the kernel moves processes between cores. Smaller values ​​represent a more preemptible system, which tends to offer best answer for interactive tasks at the cost, sometimes, of a little raw throughput.

In systems where memory is a sensitive resource, some administrators choose to activate panic mode in the event of an out-of-box error (vm.panic_on_oom) and establish kernel.panic with an automatic restart time. This strategy, while somewhat drastic, can be useful in high-availability environments where A quick and controlled reboot is preferable to a hung and unstable host. during hours.

Specialized profiles: web, databases, low latency and Kubernetes

Instead of applying a single magic bullet, it's more effective to create sysctl profiles tailored to different types of workloads: web servers, databases, container platforms, or low-latency systems. Distributing these settings in files such as 99-webserver.conf, 99-database.conf o 90-kubernetes.conf helps maintain an organized and easy-to-version configurationYou can also consider performance-oriented distributions and editions such as CachyOS Server Edition for very specific environments.

For web servers (Nginx, Apache, proxies, etc.), the essential thing is to handle large volumes of simultaneous connections with large queues and response times. FIN_WAIT With fine-tuning, an expanded ephemeral port range, and very high descriptor limits, it's common to go from a few hundred HTTP requests per second to several thousand, reducing average latency and eliminating errors. saturated queues or full ports.

On the database side (MySQL, PostgreSQL, and similar), priority is given to minimal swap usage, fine-tuning dirty page writing, and appropriate shared memory and semaphore settings for the engine. Parameters such as kernel.shmmax, kernel.shmall, kernel.sem and the huge pages configuration makes a difference in terms of transactions per second and response time, multiplying performance by two or three compared to default values ​​in many scenarios.

For extremely low latency applications (trading, online gaming, real-time processing), even more specific settings are added: net.ipv4.tcp_low_latency, deactivation of slow start after inactivity, use of tcp_fastopen, busy poll parameters (net.core.busy_poll, busy_read) and, in many cases, the deactivation of transparent huge pages by /sys/kernel/mm/transparent_hugepage/enabledAll of this aims to reduce the processing queue and latency peaks at high percentiles, achieving spectacular improvements in P99.

In container and Kubernetes environments, the network stack and connection tracking are under particular stress. Parameters such as net.ipv4.ip_forward, net.bridge.bridge-nf-call-iptables, net.netfilter.nf_conntrack_maxThe use of reserved ports for NodePort or the adjustment of ARP tables are common in nodes that handle hundreds or thousands of pods. Adjusting them correctly prevents problems. truncated connections, timeouts, or table saturation with conntrack.

Aside from these specific profiles, some organizations opt for a performance-focused archive with enhanced security, combining large network buffers and reduced swappiness with measures such as tcp_syncookies or the blocking of dangerous redirects. The goal is to achieve a balance point between aggressive performance and reasonable hardening.

Kernel profiling and monitoring tools

Adjusting the kernel without measuring is like playing an audio mixer blindfolded. Linux offers an arsenal of tools to understand what the system is actually doing, from simple counters to detailed traces of kernel functions, allowing you to correlate configuration changes with measurable effects.

Among the everyday utilities are vmstat, iostat, sar, ss, slabtop, htop o pidstatwhich provide quick information about processes, CPU, I/O, sockets, and cache usage. Combined with system logs and metrics exported to Prometheus, Grafana, or collectd, and with resources such as advanced system monitor for LinuxThey give a fairly complete picture of how the host behaves before and after each tuning.

To go into more detail, tools such as perf They allow profiling CPU usage at the function level, both user and kernel, logging events for a few seconds or minutes and then providing an interactive report. With perf stat, perf record y perf report you can see where CPU time is actually being consumed and what portion corresponds to the core, identifying, for example, an interruption storm or a poorly adjusted scheduler.

If you need even more granularity, ftrace The kernel's tracing subsystem offers the ability to activate specific tracers (for example, the function tracer) and examine in real time or post-mortem what calls are occurring. Enabling the appropriate tracer and analyzing its contents /sys/kernel/debug/tracing/trace reveals to you hotspots within the kernel itselfThis is very useful when high-level metrics are not enough.

In parallel, it is advisable to implement automated monitoring scripts that periodically dump key data (network statistics, memory counters, number of open file descriptors, etc.) into rotating logs. Once certain thresholds are reached (for example, exceeding a specified number of open file descriptors), these scripts can generate alerts via email or other channels, helping to detect dangerous trends before a problem escalates.

Finally, in prolonged stress testing phases (24 hours or more), tools such as stress-ng combined with the continuous collection of metrics (via vmstat, iostat, sar) allow you to assess the stability of the system with the new kernel configuration, verifying that there are no OOM events, crashes, unexpected restarts or progressive performance degradations under sustained load.

Optimizing the Linux kernel with sysctl and other mechanisms is not a one-off task, but rather an iterative process involving measuring results, adjusting parameters, and meticulously documenting everything. With a clear methodology, specific profiles for each type of workload, a robust suite of profiling tools, and strict change control, it's possible to achieve servers that fully utilize their hardware, with increased throughput, reduced latency, and significantly greater stability than the generic factory configuration.