Summary:
A bug in the Linux kernel versions between 6.12 and 7.0 in rare cases may cause Redpanda nodes to become unresponsive, causing cluster instability, including leaderless and/or under-replicated partitions.
The trigger is normal, expected Kubernetes behavior interacting badly with a kernel scheduler defect:
- Kubernetes scales a container group's CPU priority with the node's total core count. On a large server (32 cores), this ends up about 12x higher than the typical default — this is standard, not a misconfiguration.
- On affected kernels, that high priority value confuses the Linux task scheduler responsible for background system work. Certain low-level housekeeping jobs (memory cleanup, disk I/O completion) become starved — technically ready to run, but essentially never scheduled.
When those housekeeping jobs stall, anything waiting on them also stalls indefinitely. This creates a growing backlog of stuck processes, and eventually the server stops making progress entirely — to the point where even basic remote access can stop responding.
Severity:
High
Redpanda Products Affected:
- Redpanda Self-Managed - Enterprise
- Redpanda Self-Managed - Community
Release Affected:
Affects all public Redpanda versions as the issue is a kernel problem.
Identification:
You are at risk if you have Redpanda running in Kubernetes on any Linux kernel version between 6.12 and 7.0 (vendor kernels may pull in fixes from later kernel releases).
You can check your kernel version using a command like the one below, but this might vary depending on the system.
uname -a
If you are on the impacted Kernels versions you will want to follow the Action Required section.
Redpanda has tested and identified that the following Kernel versions have the bug patched
- 6.12.94
- 6.18.38
Impact:
If you are on an impacted kernel version and running Redpanda on Kubernetes, below are some of the ways in which your cluster can be impacted.
- Redpanda stops accepting connections on open ports like 9092.
- Redpanda cluster health becomes “unhealthy” because of leaderless or under-replicated partitions.
- Redpanda stops processing messages or processes messages at an extremely slow rate.
Action required:
Avoid upgrading to impacted kernel versions
Kernel versions starting with 6.12 are known to be impacted. Kernel versions starting with 7.1 have passed our testing and appear to have resolved the scheduler starvation issues.
If you are on an impacted kernel version
Upgrade to Kernel Version 7.1+ as soon as possible
If you are seeing impacts as described in the impact section, until you can upgrade you can enable overprovisioned to mitigate the impact, which will avoid these scheduling issues.
Update the redpanda server flags to add --overprovisioned to the additionalRedpandaCmdFlags in your values.yaml
statefulset:
additionalRedpandaCmdFlags:
- "--overprovisioned"
Redpanda Cloud
Redpanda cloud clusters will automatically have these impacted kernel versions skipped, no action is needed.
Questions? If you have any questions on this TSB, or need further guidance, please contact Redpanda Support