Overview
Consumer offsets used by Consumer Groups are are stored in internal topic
__consumer_offsets
further details are in the following Redpanda Documentation :
https://docs.redpanda.com/current/develop/consume-data/consumer-offsets/
There may be different requirements depending on use case as to how you want to manage older offsets that are no longer needed.
For example, if a deployment had lots of consumer groups which were used for testing, where the tests are now complete and the offsets/groups will not be need in the future.
This article details two different options for managing consumer offsets.
One is to let Redpanda automatically handle the deletion of consumer offsets for empty groups
Note : this is the default for new clusters created on Redpanda version 23.1 or higher
The other is to manage them manually.
These options are outlined below.
(Please note if you are a Redpanda Cloud customer and need to adjust any of the parameters listed in options below , please log a Support Ticket referencing this article and what you would like to change)
Detail
1. Redpanda remove consumer offsets for empty groups automatically
For Redpanda environments created on release 23.1 or higher..
the default behaviour is for Redpanda to delete consumer offsets for Consumer Groups that are in an Empty state (the group has no no members) after the retention time specified in parameter
group_offset_retention_sec
The parameter that controls how often Redpanda checks for consumer offsets which are eligible for removing is: group_offset_retention_check_ms
For Redpanda environments that have been upgraded to 23.1 from earlier versions,
(22.3.x for example) then automatic deletion of consumer offsets for empty groups
is not enabled by default.
To enable automatic consumer offset deletion, will need to set this parameter to true
legacy_group_offset_retention_enabled
e.g
rpk cluster config set legacy_group_offset_retention_enabled true
2. Manually remove consumer offsets for empty groups as required
If you do not want consumer offsets to be removed automatically by Redpanda, then set
parameter - group_offset_retention_sec
to a value of 'null'
e.g
rpk cluster config set group_offset_retention_sec null
It is possible to manually delete offsets from an empty Consumer group via command :
If the empty consumer group is no longer required at all then you can delete via command: