Summary:
Older versions of the Sarama client before version 1.45.1, the Redpanda Connect Kafka Output (using the Sarama 1.43.3 client), and potentially custom and unverified clients can incorrectly set max-timestamp to -1 or null. Redpanda does not update the max-timestamp with a recalculated correct value, and uses the supplied value as is while applying retention in tiered storage. When max-timestamp is set to -1 or null, hence, the retention for data in tiered storage is calculated as if the data had max-timestamp of Dec 31,1969.
As a result, users of Tiered Storage with a cleanup.policy of delete (the default) or compact,delete will find their data deleted from the cloud as soon as Redpanda's housekeeping process kicks in. Additionally because of this max-timestamp in the past time queries for this topic will not function properly.
Severity:
High
Redpanda Products Affected:
- Redpanda Self-Managed - Enterprise
- Redpanda Self-Managed - Community
- Redpanda Cloud -BYOC
- Redpanda Cloud - Dedicated
- Redpanda Cloud - Serverless
- Redpanda Connect - Kafka Output
Release Affected:
Redpanda Versions <25.2.5
Redpanda Connect version < 4.64.0
Identification:
Currently Redpanda is aware that the Sarama client on versions <1.45.1 can incorrectly set max-timestamp to -1, you can verify if you are impacted by checking to see if you are using an impacted Sarama client version.
We have checked popular clients like Kafka Java client, librdkafka, KafkaJS, Sarama and FranzGo, and only found Sarama versions <1.45.1 to not be setting max-timestamp.
Self Hosted Users
If you are using Sarama or another client not listed here, or if you want to be doubly sure, you can check your redpanda logs for the following log line: Produced batch has max_timestamp left unset. If you see this line in your logs it is likely that you have a producer that has max-timestamp set to -1 or null.
Cloud Users
Your Customer Success or Support teams will reach out to you if we believe you are using an impacted producer.
Impact:
If you are impacted by this, the retention for data in tiered storage is calculated as if the data had max-timestamp of 1st Jan, 1970 depending on your retention settings this data may be unexpectedly unavailable.
As a result, users of Tiered Storage with a cleanup.policy of delete (the default) or compact,delete will find their data deleted from the cloud as soon as Redpanda's housekeeping kicks in
Action required:
Sarama Client
If you are using the Sarama client, you will want to upgrade all clients to version 1.45.1+ immediately.
Redpanda Connect
If you are using Redpanda Connect version <4.64.0, and using the Kafka output, that will be using the Sarama 1.43.3 client. You will want to upgrade to Redpanda Connect version 4.64.0 or greater immediately.
If you cannot make this change you should review the section below about upgrading your Redpanda Cluster.
Other Clients
If you see you are impacted by checking your log lines, you will want to research your provider to see if there are any patches or configurations you can change to prevent sending max-timestamp -1 or null.
Redpanda Upgrades
Redpanda will be releasing an update to Redpanda to alleviate this issue in Redpanda version 25.2.5.
Cloud Customers
- Redpanda Cloud customers will automatically be upgraded to the patched version. Any customers where we identify the problematic log line have been contacted by their CS teams alread as well.
- If you are using a known impacted client you should still upgrade your client version as well.
Self Hosted
- Impacted Self Hosted customers should immediately upgrade to Redpanda Version 25.2.5 or greater.
- If you are using a known impacted client you should upgrade your client version as well.
NOTE: Redpanda has accounted for clients sending max-timestamp to -1 or null in version 25.2.5, however if you are using Producer compression and your client is still sending max-timestamps of -1 or null you will see additional CPU usage from the Redpanda broker as redpanda decompressed batches with the incorrect timestamp.
Questions? If you have any questions on this TSB, or need further guidance, please contact Redpanda Support