Skip to main content

TSB-2024-15 Intermittent Client Disconnects with TLS/SSL Enabled Redpanda Clusters due to OpenSSL Error Stack Overflow

  • Updated

Technical Service Bulletin (2024-15)

Subject: 

A bug with Redpanda’s implementation of OpenSSL can cause random client disconnects.

Summary: 

Redpanda was not clearing the OpenSSL error stack, and those errors were leaking into future connections, causing customer network exceptions.  This can occur when a legitimate SSL error is encountered, such as Tiered Storage S3 rate limiting, and that error then being passed to a client connection, causing a connection issue.

Severity: 

Medium

Redpanda Products Affected: 

  • Redpanda Self-Managed - Enterprise
  • Redpanda Self-Managed - Community
  • Redpanda Cloud - Dedicated
  • Redpanda Cloud - BYOC
  • Redpanda Cloud - Serverless

Release Affected:

Redpanda Versions 24.2.1 - 24.2.11

Identification:

Due to the varied nature of client error messages this error can be difficult to detect purely from client logging, but if you are seeing random, unexplainable disconnects, network errors or connectivity issues from clients connection to a Redpanda cluster you can check your cluster logs for the following error signature:

applying protocol, Broken pipe

If these errors roughly align with the client disconnect or errors you are seeing, and you are on an affected version, you could be impacted by this issue.

Impact: 

Depending on the configuration of your clients you may see no issue at all if they are retrying or reconnecting, however some clients may not reconnect automatically or require manual intervention.

This can also generate additional logging on your client depending on your configuration.

Action required: 

Dedicated, BYOC and Serverless cloud customers are upgraded to at least Redpanda version 24.2.12 and will not need to take any action.

Self-Managed customers will need to upgrade to at least Redpanda version 24.2.12 to alleviate the issue.  Additionally, depending on the client and workflow, they can potentially be configured to retry or reconnect to get around this issue.

Additionally if on a version below 24.2.1 and planning to upgrade we recommend upgrading to at least Redpanda version 24.2.12 or higher.

 

Questions? If you have any questions on this TSB, or need further guidance, please contact support@redpanda.com