Shadowing: How to deal with scenario of faulted topics due to them being dropped/recreated on source cluster – Redpanda

Issue

When using the Redpanda Shadowing feature, if a replicated topic is later deleted and recreated on the source cluster it will go into a FAULTED state on the Shadow

Shadowing uses internal topic-id's to track replication of topic data + metadata.

If a topic changes its topic-id on the source cluster , as it would when a topic is dropped/recreated then Shadowing will mark the shadow topic as faulted. This is by design.

There may be occasions where it was intentional to drop/recreate the topic on the source cluster.
In the resolution section are details of how to fix topic(s) in this scenario.

Resolution

Note: This option currently only works in helm/operator due to issue described in https://support.redpanda.com/hc/en-us/articles/38823476119063-rpk-shadow-update-appends-existing-values-for-auto-create-shadow-topic-filters

For each affected topic add a LITERAL exclude filter doc using either of mechanisms described here https://docs.redpanda.com/current/manage/kubernetes/shadowing/k-shadow-linking/#update-a-shadow-link
Confirm updates applied via : kubectl describe shadowlink --namespace namespace shadow-name
Delete the topics on shadow cluster via rpk topic delete
Update the shadow link configuration again to remove the previously added EXCLUDE filters using either of mechanisms described here https://docs.redpanda.com/current/manage/kubernetes/shadowing/k-shadow-linking/#update-a-shadow-link
Confirm updates applied via via : kubectl describe shadowlink --namespace namespace shadow-name
Wait a short while and check 'rpk shadow status <link>" and the topic should no longer be in faulted state and active again .

Example

rpk shadow status to show state before topics deleted on source cluster

 rpk shadow status disaster-recovery-link -t
TOPICS
======
Name: theSmiths-0, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          33       33       33       0
Name: theSmiths-1, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          12       12       12       0
Name: theSmiths-2, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          10       10       10       0
Name: theSmiths-3, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          20       20       20       0
Name: humDrumTown-0, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          32       32       32       0
Name: humDrumTown-1, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          15       15       15       0
Name: humDrumTown-2, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          17       17       17       0
Name: humDrumTown-3, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          26       26       26       0

Synced Shadow Topic Properties:
 - compression.type
 - cleanup.policy
 - max.message.bytes
 - message.timestamp.type
 - min.compaction.lag.ms
 - retention.ms
 - replication.factor
 - delete.retention.ms
 - retention.bytes
 - max.compaction.lag.ms

Delete and recreate topics on the source cluster


rpk topic delete theSmiths-0 humDrumTown-3 ; rpk topic create theSmiths-0 humDrumTown-3
TOPIC          STATUS
humDrumTown-3  OK
theSmiths-0    OK
TOPIC          STATUS
theSmiths-0    OK
humDrumTown-3  OK

Deleted/recreated topics are now marked as FAULTED on the shadow (expected)

rpk shadow status disaster-recovery-link -t
TOPICS
======

Name: theSmiths-0, State: FAULTED
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG

Name: theSmiths-1, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          12       12       12       0

Name: theSmiths-2, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          10       10       10       0

Name: theSmiths-3, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          20       20       20       0


Name: humDrumTown-0, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          32       32       32       0

Name: humDrumTown-1, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          15       15       15       0

Name: humDrumTown-2, State: ACTIVE
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
      0          17       17       17       0

Name: humDrumTown-3, State: FAULTED
      PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG


Synced Shadow Topic Properties:
  - cleanup.policy
  - max.message.bytes
  - message.timestamp.type
  - retention.bytes
  - retention.ms
  - delete.retention.ms
  - min.compaction.lag.ms
  - compression.type
  - max.compaction.lag.ms
  - replication.factor

The Redpanda logs on the shadow also reports the cause :

WARN  2026-03-05 14:17:15,384 [shard 0:cl  ] shadow_link - Source Topic Sync - source_topic_syncer.cc:498 - Topic theSmiths-0 has changed its topic ID from tF9ih4CNSoeK0A0P3Wxdvw== -> 7prUqGm5SI2tRXc9347SJg==.  Marking as failed
WARN  2026-03-05 14:17:15,384 [shard 0:cl  ] shadow_link - Source Topic Sync - source_topic_syncer.cc:498 - Topic humDrumTown-3 has changed its topic ID from oeRxvx0ZQ6Cemj5+hreNvQ== -> jGyErbVVR+e5vCD6VZ3Liw==.  Marking as failed

In this example editing the shadowlink direct :-

kubectl edit shadowlink --namespace redpanda disaster-recovery-link

  topicMetadataSyncOptions:
   autoCreateShadowTopicFilters:
   - filterType: include
     name: '*'
     patternType: literal
   - filterType: exclude
     name: 'theSmiths-0'
     patternType: literal
   - filterType: exclude
     name: 'humDrumTown-3'
     patternType: literal
   interval: 30s
   startOffset: earliest

Check the excludes have been appplied....

kubectl describe shadowlink --namespace redpanda disaster-recovery-link

 Topic Metadata Sync Options:
   Auto Create Shadow Topic Filters:
     Filter Type:   include
     Name:          *
     Pattern Type:  literal
     Filter Type:   exclude
     Name:          theSmiths-0
     Pattern Type:  literal
     Filter Type:   exclude
     Name:          humDrumTown-3
     Pattern Type:  literal
   Interval:        30s
   Start Offset:    earliest

As the 'excludes' have been applied we can now delete the 2 topics on the shadow.

rpk topic delete  theSmiths-0 humDrumTown-3
TOPIC          STATUS
humDrumTown-3  OK
theSmiths-0    OK

Remove the previously created excludes
In this case, using :
kubectl edit shadowlink --namespace redpanda disaster-recovery-link

and remove previously added excludes..so in this example it now looks like this :

  topicMetadataSyncOptions:
   autoCreateShadowTopicFilters:
   - filterType: include
     name: '*'
     patternType: literal
   interval: 30s
   startOffset: earliest

Now check status again ...

rpk shadow status disaster-recovery-link -t

we can see our recreated topics are now in an ACTIVE state and copying again

TOPICS
======
Name: humDrumTown-4, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          0        0        0        0
Name: theSmiths-1, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          12       12       12       0
Name: theSmiths-2, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          10       10       10       0
Name: theSmiths-3, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          20       20       20       0
Name: humDrumTown-0, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          32       32       32       0
Name: humDrumTown-1, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          15       15       15       0
Name: humDrumTown-2, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          17       17       17       0
Name: theSmiths-0, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          6        6        6        0
Name: humDrumTown-3, State: ACTIVE
     PARTITION  SRC_LSO  SRC_HWM  DST_HWM  LAG
     0          5        5        5        0