san07.com

The Linux Out-Of-Memory (OOM) killer terminates processes when the kernel cannot satisfy memory allocations under the applicable memory constraints; the decisive log signatures include Out of memory: Killed process, oom-kill, Memory cgroup out of memory, and Killed process <PID> (<name>), while the underlying cause may be exhausted physical RAM, insufficient swap, a memory-hungry process, kernel memory pressure, or a cgroup/systemd/container memory limit. The OOM killer is a kernel-level recovery mechanism rather than an application error, and the correct fix is to identify which memory domain was exhausted before changing vm.* parameters or simply increasing RAM.

What the Linux OOM Killer Actually Does

Linux normally attempts to reclaim memory before invoking the OOM killer. Reclaim can include filesystem cache, anonymous memory management, swap activity, and other reclaimable pages. When the kernel cannot satisfy an allocation within the relevant memory policy or cgroup, an OOM condition can occur.
A typical kernel message looks like:

Out of memory: Killed process 18472 (java) total-vm:...

Other useful signatures include:

oom-kill:constraint=CONSTRAINT_NONE

Memory cgroup out of memory: Killed process 1234 (node)

Killed process 1234 (postgres)

The distinction between a system-wide OOM and a cgroup OOM is critical. A server can have apparently available RAM while a container, systemd service, virtual machine, or Kubernetes workload has already reached its assigned memory limit.

Stage 1: Log Inspection and Diagnostic Verification

Step 1: Find the Exact OOM Event

Start with the kernel journal:

sudo journalctl -k --since "24 hours ago" --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

On systems using traditional kernel logs:

sudo grep -Ei 'out of memory|oom-kill|killed process|memory cgroup' /var/log/syslog /var/log/messages 2>/dev/null

Check the current boot:

sudo journalctl -k -b --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

For the complete event surrounding an OOM kill:

sudo journalctl -k -b --no-pager | less

Do not focus only on the final Killed process line. The preceding kernel messages often reveal the memory state, cgroup, allocation type, and available swap at the time of failure.

Step 2: Check Current RAM and Swap

Run:

free -h

A more detailed view is:

cat /proc/meminfo

Pay particular attention to:

grep -E 'MemTotal|MemFree|MemAvailable|SwapTotal|SwapFree|SwapCached|SReclaimable|Shmem' /proc/meminfo

MemAvailable is generally more useful for estimating how much memory can be allocated without swapping than MemFree, because Linux deliberately uses otherwise unused RAM for caches.
Check swap devices:

swapon --show

Then:

cat /proc/swaps

If:

SwapTotal:       0 kB

appears in /proc/meminfo, the server has no configured swap space. That does not automatically mean swap must be added, but it removes an important buffer during periods of memory pressure.

Step 3: Identify the Process Consuming Memory

Use ps to rank processes by resident memory:

ps -eo pid,ppid,user,%mem,rss,vsz,comm --sort=-rss | head -n 25

For a continuously updating view:

top

Inside top, press:

Shift+M

to sort by memory consumption.
If htop is installed:

htop

The RSS value is especially useful because it represents resident memory currently held in physical RAM. VSZ is virtual address space and should not be interpreted as equivalent to actual RAM consumption.
For a specific process:

PID=1234
sudo cat /proc/$PID/status | grep -E 'VmPeak|VmSize|VmRSS|VmSwap|RssAnon|RssFile|RssShmem'

Step 4: Determine Whether Memory Usage Is Growing

A process that uses 10 GB consistently is different from one that starts at 500 MB and continuously grows until the kernel kills it.
Record process memory periodically:

while true; do
    date
    ps -p 1234 -o pid,etime,%mem,rss,vsz,cmd
    sleep 10
done

For multiple processes:

watch -n 5 'ps -eo pid,%mem,rss,vsz,comm --sort=-rss | head -n 15'

A continuously increasing RSS value can indicate a memory leak, an unbounded cache, excessive concurrency, unexpectedly large workloads, or an application configuration problem.

Step 5: Check Pressure Stall Information

Linux provides memory pressure information through PSI:

cat /proc/pressure/memory

You may see:

some avg10=0.00 avg60=0.00 avg300=0.00 total=...
full avg10=0.00 avg60=0.00 avg300=0.00 total=...

High memory-pressure values indicate that processes are spending significant time stalled because of memory contention. PSI is particularly useful when the server is suffering severe memory pressure before an OOM event occurs.

Stage 2: Core Configuration Fix and Service Recovery

Step 6: Determine Which Process the Kernel Actually Killed

Search the kernel journal for the PID:

sudo journalctl -k --since "24 hours ago" --no-pager | grep -Ei 'Killed process|oom-kill'

You may see:

Killed process 18472 (java) total-vm:...

Find information about that process if it still exists:

ps -p 18472 -o pid,ppid,user,%mem,rss,vsz,etime,cmd

If it has already exited:

sudo journalctl -k -b --no-pager | grep '18472'

The process killed by OOM is not necessarily the original cause of the memory problem. Linux chooses a process based on its OOM scoring and memory characteristics, so the terminated process may simply have been the process selected to release enough memory.

Step 7: Inspect the OOM Score

For a running process:

PID=1234
cat /proc/$PID/oom_score
cat /proc/$PID/oom_score_adj

The kernel’s OOM selection mechanism considers the process’s OOM-related score. A process with a modified oom_score_adj can therefore be treated differently from another process with the same approximate memory consumption.
Inspect important services:

for pid in $(pgrep -x sshd); do
    echo "PID=$pid"
    cat /proc/$pid/oom_score_adj
done

Do not automatically set critical applications to oom_score_adj=-1000. Making an essential service effectively immune to OOM termination can force the kernel to kill other processes or leave the machine in a prolonged memory-starved state.

Step 8: Check systemd Memory Limits

A service can encounter an OOM condition because systemd has placed it inside a memory-controlled cgroup.
Inspect the service:

sudo systemctl show nginx -p MemoryMax -p MemoryHigh -p MemoryCurrent -p MemorySwapMax

For another service:

sudo systemctl show myapp.service -p MemoryMax -p MemoryHigh -p MemoryCurrent -p MemorySwapMax

Check the complete cgroup information:

sudo systemctl status myapp.service

Also inspect:

systemctl show myapp.service | grep -Ei '^Memory(Max|High|Current|SwapMax)='

A service with:

MemoryMax=2G

can be killed by the memory controller even if the host still has unused RAM.

Step 9: Check the Cgroup Memory Controller

Determine the process’s cgroup:

PID=1234
cat /proc/$PID/cgroup

On cgroup v2 systems:

mount | grep cgroup

Inspect the memory controller:

cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.high

A value of:

max

for memory.max means there is no explicit maximum at that cgroup level.
For a systemd service, use:

systemctl status myapp.service
systemctl show myapp.service -p ControlGroup

Then inspect the corresponding cgroup path under /sys/fs/cgroup/.
This distinction matters because increasing server RAM will not necessarily solve a cgroup memory limit that is deliberately set below the host’s physical capacity.

Step 10: Reduce Application Memory Consumption

If a specific service repeatedly consumes excessive memory, correct its application-level configuration before changing kernel behavior.
For a systemd-managed application:

sudo systemctl edit myapp.service

A service-level memory limit can be explicitly defined:

[Service]
MemoryHigh=6G
MemoryMax=8G

Then:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

MemoryHigh and MemoryMax have different purposes. A high threshold can apply memory-pressure control before the hard maximum is reached, while MemoryMax provides a hard cgroup memory boundary.
Do not blindly add limits to production applications. A limit that is lower than legitimate working-set requirements can convert ordinary workload growth into repeated service termination.

Step 11: Configure Swap When Appropriate

Check existing swap:

swapon --show

If no swap exists and the workload would benefit from additional virtual memory, create a swap file.
First verify available filesystem capacity:

df -h /

Create a file:

sudo fallocate -l 4G /swapfile

Secure it:

sudo chmod 600 /swapfile

Initialize it:

sudo mkswap /swapfile

Enable it:

sudo swapon /swapfile

Verify:

swapon --show
free -h

Make it persistent:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Then validate:

sudo mount -a

Do not assume a large swap file fixes the underlying problem. Swap can provide additional virtual-memory capacity, but sustained swapping can severely degrade application performance.

Stage 3: Root Cause Prevention and Kernel/System Limit Tuning

Step 12: Inspect vm.swappiness

Check the current setting:

sysctl vm.swappiness

The setting influences the kernel’s tendency to reclaim anonymous memory through swap relative to other reclaimable memory.
For a temporary test:

sudo sysctl vm.swappiness=10

To make a deliberate configuration persistent:

sudo tee /etc/sysctl.d/90-memory.conf >/dev/null <<'EOF'
vm.swappiness = 10
EOF

Apply:

sudo sysctl --system

The appropriate value depends on the workload, kernel behavior, available RAM, storage performance, and whether swap is intentionally used. There is no universally correct production value.

Step 13: Inspect Commit-Limit Configuration

Check:

sysctl vm.overcommit_memory
sysctl vm.overcommit_ratio
grep -E 'CommitLimit|Committed_AS' /proc/meminfo

Linux memory overcommit controls influence whether memory allocations are permitted beyond immediately available physical memory and swap.
Do not change vm.overcommit_memory simply because an OOM event occurred. An inappropriate setting can alter application allocation behavior and create different failure modes.
If the application explicitly requires a particular overcommit policy, configure it through:

sudo tee /etc/sysctl.d/91-vm-overcommit.conf >/dev/null <<'EOF'
vm.overcommit_memory = 1
EOF

Then:

sudo sysctl --system

Only apply this when the workload’s documented memory-management requirements justify it.

Step 14: Check File Descriptor and Process Limits

Memory incidents can sometimes be accompanied by excessive process or descriptor creation.
Check:

ulimit -a

For a service:

sudo systemctl show myapp.service -p LimitNOFILE -p TasksMax

Check the process count:

ps -e --no-headers | wc -l

Check the kernel PID limit:

cat /proc/sys/kernel/pid_max

For a systemd service:

sudo systemctl show myapp.service -p TasksCurrent -p TasksMax

A runaway process-spawning workload can consume memory through large numbers of processes or threads even when no individual process looks exceptionally large.

Step 15: Investigate Kernel Memory Consumption

If user-space processes do not account for the missing memory, inspect slab usage:

sudo slabtop

Or:

grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo

Inspect complete memory information:

cat /proc/meminfo

A large SUnreclaim value can indicate significant unreclaimable kernel memory. Drivers, networking workloads, filesystem activity, and kernel subsystems can contribute to kernel memory pressure.
Do not attempt to solve unexplained kernel memory consumption by randomly changing cache or slab-related sysctl values. Identify the subsystem responsible first.

Step 16: Monitor Memory Pressure Before the Next OOM Event

A simple monitoring loop is:

while true; do
    date
    free -h
    echo "-----"
    cat /proc/pressure/memory
    echo "-----"
    ps -eo pid,%mem,rss,comm --sort=-rss | head -n 10
    sleep 30
done

For a production environment, collect memory utilization, swap activity, PSI, process RSS, cgroup memory usage, OOM events, and application-specific metrics.
The important objective is to identify whether memory usage rises gradually, spikes during a particular workload, or reaches a fixed cgroup limit.

Diagnosing Common OOM Scenarios

Java Applications

Check the Java process:

pgrep -af java

Then:

PID=$(pgrep -xo java)
grep -E 'VmRSS|VmSize|VmSwap' /proc/$PID/status

For JVM configuration:

jcmd $PID VM.flags

If jcmd is available, inspect heap information:

jcmd $PID GC.heap_info

Do not set the Java heap equal to all available server RAM. The JVM also requires native memory, thread stacks, class metadata, direct buffers, and other allocations outside the Java heap.

MySQL or MariaDB

Check:

sudo systemctl show mysql -p MemoryCurrent -p MemoryMax

or:

sudo systemctl show mariadb -p MemoryCurrent -p MemoryMax

Then inspect the process:

ps -C mysqld -o pid,%mem,rss,vsz,cmd

Database buffer pools and connection counts can create substantial memory demand. Correcting database configuration is generally preferable to forcing the kernel to tolerate an undersized host.

Nginx, Apache, and PHP-FPM

For PHP-FPM:

ps -C php-fpm8.2 -o pid,%mem,rss,vsz,cmd

The exact executable name varies by distribution and PHP version:

pgrep -af 'php-fpm'

Inspect pool configuration:

sudo find /etc/php -type f -path '*/fpm/pool.d/*.conf' -print

Common worker controls include:

pm = dynamic
pm.max_children = 20
pm.max_requests = 500

The correct pm.max_children depends on the application’s actual per-worker memory usage. Increasing it without measuring worker RSS can turn a traffic spike into an OOM event.

Containers and Kubernetes

Check Docker memory limits:

docker stats

For a specific container:

docker inspect CONTAINER_ID --format '{{.HostConfig.Memory}}'

For Kubernetes:

kubectl top pods -A

Inspect configured resources:

kubectl get pod POD_NAME -n NAMESPACE -o yaml

Look for:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

A container terminated with an OOM-related status may have exceeded its configured cgroup memory limit rather than exhausting the entire host.

What Not to Change During an OOM Incident

Do Not Disable the OOM Killer

The OOM killer exists because the kernel needs a recovery mechanism when memory allocation cannot be satisfied. Disabling or attempting to defeat it without solving memory pressure can leave the server unresponsive.

Do Not Arbitrarily Increase Kernel Memory Values

Changing:

vm.swappiness
vm.overcommit_memory
vm.overcommit_ratio

without understanding the workload can hide the actual problem or create another one. First establish whether the pressure originates from user-space memory, swap exhaustion, a cgroup limit, kernel memory, or excessive process creation.

Do Not Automatically Protect Critical Processes

Changing:

echo -1000 | sudo tee /proc/PID/oom_score_adj

can make the selected process harder for the kernel to kill, but it does not create additional memory. The kernel may instead terminate another important service.
If process protection is required, document why that process must survive and verify that sufficient memory capacity exists for the remaining workload.

Final Verification After the Repair

Step 17: Confirm Memory Capacity

Run:

free -h

Then:

swapon --show

Then:

grep -E 'MemAvailable|SwapFree|CommitLimit|Committed_AS' /proc/meminfo

Step 18: Confirm the Previously Affected Service

sudo systemctl status myapp.service --no-pager

Check recent failures:

sudo journalctl -u myapp.service -b --no-pager | tail -n 100

Check its current memory usage:

sudo systemctl show myapp.service -p MemoryCurrent -p MemoryHigh -p MemoryMax

Step 19: Confirm That New OOM Events Are Not Occurring

sudo journalctl -k --since "1 hour ago" --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

If this returns nothing after the repair and the workload has passed through the period that previously triggered OOM, continue monitoring rather than assuming the problem is permanently resolved.

Final Diagnostic Sequence

For a production Linux server experiencing repeated OOM kills, run the following sequence before changing kernel parameters:

sudo journalctl -k --since "24 hours ago" --no-pager | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

free -h

swapon --show

ps -eo pid,ppid,user,%mem,rss,vsz,comm --sort=-rss | head -n 25

cat /proc/meminfo

cat /proc/pressure/memory

systemctl --failed

sudo systemctl show SERVICE_NAME -p MemoryCurrent -p MemoryHigh -p MemoryMax -p MemorySwapMax

PID=$(pgrep -xo PROCESS_NAME)
sudo cat /proc/$PID/status | grep -E 'VmRSS|VmSwap|RssAnon|RssFile|RssShmem'

sysctl vm.swappiness

sysctl vm.overcommit_memory

grep -E 'Slab|SReclaimable|SUnreclaim' /proc/meminfo

The correct resolution should follow the evidence from these checks. If physical RAM and swap are genuinely exhausted, reduce application memory consumption, increase available memory, or add appropriately sized swap. If only a service cgroup is exhausted, adjust its workload or memory limit. If a single process grows continuously, investigate its configuration or memory leak. If kernel memory dominates, investigate the relevant kernel subsystem or driver. If the system has severe memory pressure without a clear user-space culprit, use /proc/meminfo, PSI, cgroup statistics, and historical monitoring to establish where the memory is being consumed before making kernel-level changes.