Technical Service Bulletin (2024-11)
Subject:
Avoid installing or upgrading to Redpanda v24.2.1 and instead directly go to Redpanda v24.2.2
Summary:
Redpanda introduced a bug fix during v24.2.1 development causing the internal raft controller log to adopt the value of cluster configuration property log_cleanup_policy. If the value of log_cleanup_policy was or is set to a specific non-default value at any time, the controller log could be incorrectly modified after upgrade. This log is fundamental in the operation of Redpanda and unintentional modification of this log will result in cluster errors and instability. This state might be unrecoverable.
Severity:
High Risk
Redpanda Products Affected:
Redpanda Enterprise Self-Hosted and Redpanda Community
Releases Affected:
v24.2.1 or any v24.2.1 Release Candidate
Identification:
This bug will be encountered if:
- Upgrading a cluster to v24.2.1 that has cluster property log_cleanup_policy set to compact or compact,delete
- A brand new v24.2.1 cluster where the cluster administrator changes log_cleanup_policy to compact or compact,delete
- The default value for log_cleanup_policy is delete, unless these settings have been intentionally changed you are not impacted.
Impact:
If log_cleanup_policy is set to compact or compact,delete, this causes the controller log to compact out important controller state used to track items such as partition movements, node membership, role-based access control information, data transform deployments, audit history, leading to inconsistent behavior and availability of cluster functions.
While there is impact to cluster stability, health and potentially availability, this does not lead to data loss in topics.
Action Required:
Do not install or upgrade to Redpanda v24.2.1 - use Redpanda v24.2.2 instead.
Systems already using Redpanda v24.2.1:
-
If log_cleanup_policy is set to compact or compact,delete, do the following.
-
Immediately set log_cleanup_policy to delete.
- If unable to perform this step, reach out to Redpanda Support.
- Perform a rolling restart of the cluster.
- Reach out to Redpanda Support to help recover the cluster.
-
Immediately set log_cleanup_policy to delete.
- If log_cleanup_policy is not set to compact or compact,delete , upgrade immediately to v24.2.2.
- Additionally avoid setting log_cleanup_policy or to compact or compact,delete while on v24.2.1.
Addressed by:
Redpanda has removed the v24.2.1 release and released Redpanda v24.2.2. All users of Redpanda v24.2.x should use version v24.2.2 or greater.
v24.2.2 includes PR 22747 that addresses this issue.
Questions? If you have any questions on this TSB, or need further guidance, please contact support@redpanda.com