Skip to main content

TSB-2024-17 - Unbounded Growth in Consumer Offsets Topic after upgrading

  • Updated

Technical Service Bulletin (2024-17)

Subject:

Unbounded Growth in Consumer Offsets Topic after upgrading

Summary:

Customers that have been upgraded from 24.1.x or earlier to 24.2 or 24.3 and who use group transactions can hit a scenario that results in one or more partitions of the Consumer Offsets topic to stop compacting. If the Consumer Offsets topic grows unbounded it can result in eventual slow down or stopping of the consumer group coordinators, and subsequent loss of service.

Severity:

High

Redpanda Products affected: 

  • Redpanda Enterprise and Redpanda Community

Releases affected:

  • 24.2.x, 24.3.x

Impact:

If left untreated, this situation could lead to the unavailability of data or inability to produce.

Immediate Action required: Yes

Action required:

Redpanda will be releasing an updated version of 24.2 and 24.3 in the coming days. Until this happens, customers are advised to monitor the size of the Consumer Offsets topic. If the Consumer Offsets topic is shown to be growing, please raise a support case as soon as possible with TSB-2024-17 in the title.

One example of monitoring the Consumer Offsets topic would be to the following prometheus query.  More information on monitoring Redpanda metrics is found here.

sum by(topic) (vectorized_storage_log_partition_size{redpanda_cluster="<cluster-name>", host=".*", topic="__consumer_offsets" }) 

If you are on Redpanda version 24.1.x and using group transactions hold off on upgrading to 24.2 and 24.3 until a fix for this issue is released.

Addressed in (RP Version): The fix is anticipated to be in Redpanda 24.2.15 and 24.3.3.  You can monitor progress of the fix by watching PR 24367 and our release pages.

 

Questions?: If you have any questions on this TSB, or need further guidance, please contact customer-success@redpanda.com / support@redpanda.com