Skip to main content

Cluster is unhealthy due to no leader controller / no_elected_controller what shall I do ?

  • Updated

Issue / Scenario


 rpk cluster health  output shows the cluster marked as not healthy , 
with no_elected_controller reported  as a reason + a negative value for Controller ID:

For example :

CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          false
Unhealthy reasons:                [leaderless_partitions no_elected_controller nodes_down]
Controller ID:                    -1
All nodes:                        [0 1 2 3]
Nodes down:                       [1 2]
Nodes in recovery mode:           []
Leaderless partitions (1):        [redpanda/controller/0]
Under-replicated partitions (0):  []
Cluster UUID:                     90423fd3-b10b-4d4c-83ae-90a9ef58db2a



Additionally errors such as this from returned from API calls / in the Redpanda logs ..

- Currently there is no leader controller elected in the cluster
- Service Unavailable, Leader not available

redpanda admin brokers list will typically report multiple nodes with IS-ALIVE=false

ID    HOST                                               PORT   RACK  CORES  MEMBERSHIP  IS-ALIVE  VERSION    UUID
0     redpanda-0.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      true      25.3.1   20c08752-c6d2-4582-a79e-032f85443906
1     redpanda-1.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      false     -        45bcf4de-215c-42ed-a77d-ff00b080ad52
2     redpanda-2.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      false     -        4a910f3b-14a4-422f-939c-30fdda37cc35
3     redpanda-2.redpanda.humdrumtown.svc.cluster.local.  33145  -     1      active      true      25.3.1   12834a5b-0f96-478a-b955-4391c74b8aa0


Cause 

no_elected_controller  happens when the cluster cannot get quorum and is unable to elect a leader controller 


One cause of this could be if multiple brokers are down  (for whatever reason ) .. for example 3 brokers down in a 5 broker cluster.

Another scenario is after disk crashes/replacements on multiple nodes , where additional nodes (new node-ids) are started up and are unable to join the cluster leaving a node count where the number of failed/down nodes compared to running  nodes is less than the required raft quorum .


 

Action to take 

If you encounter a no_elected_controller scenario please log a ticket with Redpanda Support  (severity 1 if a production environment  ) so we can provide steps to fix your cluster ..
 

Provided the following in information on the ticket to ensure the quickest resolution for this scenario...

  • Detailed information on what lead to this problem state.  For example was there any disk/node crashes or node upgrades for example. 
  • Logs from each broker . Ideally from Redpanda startup.

  • Output of Redpanda commands
    Please note that it's important to specify the correct flags to access your cluster in the rpk command - these will be different for each environment, but would generally include:

    -X user=my_username

    -X pass=mY_p4sSw0rD

    -X sasl.mechanism=SCRAM-SHA-256  (or SCRAM-SHA-512 )

    -X tls.enabled=true 
     -X admin.tls.enabled if configured on the cluster
    -X admin.tls.insecure_skip_verify=true (to bypass certificate checks) 

     Or you can set up an rpk profile to pass all of these for you automatically.
     

rpk cluster health;
rpk redpanda admin brokers list --include-decommissioned ;
rpk topic describe -r '*';

# on a redpanda node that has IS-ALIVE=true in 'rpk redpanda admin brokers list'
rpk debug bundle 
# end points (change to https:// if admin api not tls enabled ) 
curl -k https://localhost:9644/v1/broker_uuids #(to capture all UUIDs)
curl -k https://localhost:9644/v1/brokers #(to capture core count)

 

  • for Kubernetes/K8s 

    kubectl -n <namespace> get pv,pvc,po;
    kubectl -n <namespace> describe pods;
    kubectl get events --namespace <namespace> --sort-by='.metadata.creationTimestamp'