Summary: 

After upgrading to Redpanda v26.1 or later, clusters running on nodes with 128 or more CPU cores may experience TLS RPC connection failures on specific shards, resulting in Raft heartbeat failures and degraded produce latency.

The root cause is an OpenSSL issue surfaced by the OpenSSL version upgrade in v26.1: OpenSSL's BIO_get_new_index function caps custom BIO method indices at 127. When Redpanda runs with 128 shards (one per core), the one of the shards fails to register its TLS BIO method, producing errors such as:

WARN [shard 126:main] seastar - Exceptional future ignored: std::__1::system_error

(error OpenSSL:1074528519, Failed to obtain new BIO index: error:400C0107:lib(128)::operation fail)

This causes those shards to fail to establish outgoing RPC connections, leading to Raft heartbeat failures and elevated produce latency.

This issue was not present before v26.1 because earlier Redpanda versions used an older OpenSSL that silently allowed BIO indices beyond 127 (colliding with internal flags), whereas newer OpenSSL enforces the limit and returns an error. See: openssl/openssl#23655.

This is fixed in Redpanda v26.1.3.

Severity: 

Medium

Redpanda Products Affected:

• Redpanda Self-Managed - Enterprise
• Redpanda Self-Managed - Community

Release Affected:

• Redpanda Core v26.1.0
• Redpanda Core v26.1.1
• Redpanda Core v26.1.2

Identification:

You are potentially impacted if ALL of the following are true:

• You are running Redpanda v26.1.0, v26.1.1, or v26.1.2.
• Your Redpanda brokers are configured to have 128 or more CPU cores (i.e., Redpanda is running with --smp 128 or higher).
• TLS is enabled for any of the broker interfaces (Kafka API, inter broker communication, Admin API, schema registry) or tiered storage is enabled.

Additionally, if you are impacted by this you may see logs similar to the example in the summary.

Impact: 

Affected shards fail to establish TLS connections for inter-broker RPC or any other TLS based communication. This causes:

• Raft heartbeat failures — shards that cannot make outgoing RPC connections fail to send or receive heartbeats, causing replicas on those shards to fall out of sync with the rest of the cluster.
• Elevated produce latency — clients may experience degraded p999 latency as a result of the heartbeat failures.
• Stuck partition moves — rpk cluster partitions move operations targeting affected nodes may stall.
• Tiered storage upload/download failures — TS based topics will fail to upload or download data on the affected shards

rpk cluster health may report leaderless partitions. This is a secondary reporting artifact: replicas on affected shards lose contact with the leader and misreport themselves as leaderless, but the Raft group itself retains its leader and produce/consume continues to function. The cluster is degraded, not down.

Action required: 

On an Impacted Version with 128+ core nodes:

If impacted please upgrade to 26.1.3+, if you cannot upgrade immediately, reduce the number of CPU cores used per node to 120 or fewer (e.g., set --smp 120 or configure CPU resource limits in Helm/Operator). This avoids exhausting the 127-index limit.

For any core count downgrade you will need both of the following set to true

rpk cluster config set core_balancing_on_core_count_change true

rpk cluster config set core_balancing_continuous true

See: Redpanda CPU cores configuration

Running Redpanda Version < 26.1.1 128+ core nodes:

Do not upgrade to Redpanda versions 26.1.1-26.1.2, when ready, upgrade directly to 26.1.3.

Not using nodes with 128+ cores:

No action needed.  If expanding to nodes with 128 or more cores do not upgrade to Redpanda versions 26.1.1-26.1.2, when ready, upgrade directly to 26.1.3.

References

Questions? If you have any questions on this TSB, or need further guidance, please contact Redpanda Support