Issue
In releases :
- 26.1.7 and lower
- 23.3.14 and lower
if your Redpanda deployment has a large number of ACLS (8k + for example ) .. then there is a chance of
memory allocation related crashes even if it appears your Redpanda broker has plenty of free emory
To check if how many ACL's you have : rpk security acl list|wc -l
Anything 8k or over you are at risk of this bug occurring
Detail
Seastar, the I/O framework Redpanda utilises, manages memory per shard using a buddy allocator. Each shard's memory arena may have plenty of free memory in aggregate, but that memory is fragmented into small regions by normal broker operation.
When handling ACL's the memory allocation size is approximately:
peak allocation ≈ (ACL count) × 168 bytes × 2
For example, with 9,350 ACLs the peak reallocation request is approximately 1.5 MB, which requires a 2 MB contiguous span. If the 2 MB span pool on the shard is exhausted, the broker crashes.
Example crash sequence (values may vary):
06:43:14.787 Recorded crash reason to crash file on shard 3 (OOM)
06:43:14.796 [ERROR] seastar_memory — Dumping seastar memory diagnostics
06:43:14.796 Free memory: 2917M
06:43:14.796 Used memory: 1179M
06:43:14.797 [ERROR] seastar — Failed to allocate 1572864 bytes
06:43:14.797 Total memory: 4G | Hard failures: 1
06:43:14.797 Aborting on shard 3, in scheduling group kafka.
06:43:14.797 Segmentation fault: si_code: 4294967290, ip: 00007f3e837109fc
06:43:14.797 Segmentation fault on shard 3, in scheduling group kafka.
06:43:14.797 Segmentation fault: resolved ip: 0x00000000000969fc
in /opt/redpanda/lib/libc.so.6[0x00007f3e8367a000+0x2261a5]
In the memory diagnostics from the crash .. in this case where the alloc size was around 1.5MB .. the 2MB span size was exhausted , hence the memory crash
index span-size free used spans
─────────────────
0 4K 22M 190M 54k │
1 8K 22M 24M 6k │
2 16K 14M 53M 4k │
3 32K 295M 143M 14k │ large free pool
4 64K 516M 220M 12k │ large free pool
5 128K 778M 386M 9k │ large free pool
6 256K 757M 5M 3k │ large free pool
7 512K 398M 0B 795 │ large free pool
8 1M 115M 1M 116 │ some free
9 2M 0B 22M 11 │ *** EMPTY — this is the cause of crash ***
10 4M 0B 8M 2 │ EMPTY
11 8M 0B 16M 2 │ EMPTY
12 16M 0B 48M 3 │ EMPTY
13 32M 0B 0B 0 │
14 64M 0B 64M 1 │
15+ ≥128M 0B 0B 0 │
Resolution
The fix (CORE-14944) replaces `std::vector` with `chunked_vector` — Seastar's non-contiguous vector implementation — so the ACL result set is built in fixed-size chunks that always fit within the small-pool allocation range, regardless of ACL table size.
The bug is fixed in the following releases :
26.1.8 and higher via PR 30401
25.3.15 and higher via PR 30389