Run SUSE Observability on Longhorn

This page describes the Longhorn decisions that are specific to SUSE Observability. Use it together with the Longhorn documentation for installation, capacity planning, monitoring, backup targets, and version-specific procedures.

Before installation

Use SSD-backed disks that meet the SUSE Observability system requirements. Provide at least three eligible Longhorn storage nodes and spread them across the failure domains that the installation must tolerate. Configure Longhorn replica node and zone anti-affinity so that this separation is enforced, rather than only preferred. Keep enough free capacity to rebuild all replicas from one failed or drained node.

Do not use Longhorn replicas as a replacement for backups. Configure SUSE Observability backups in external object storage and test the restore procedure.

StorageClass strategy

Use two Longhorn replicas by default for durable SUSE Observability volumes in both Non-HA and HA installations. This follows the Longhorn I/O performance best practices, which recommend two replicas for data availability with lower disk-space usage and less impact on system performance, especially for data-intensive applications. After one replica-host failure, a two-replica volume has only one healthy copy until Longhorn completes the rebuild; do not continue maintenance while any volume is degraded.

An HA installation can reduce replication traffic, and improve write performance, by using one Longhorn replica for selected services that already maintain independent application-level replicas. This optimization makes the loss of one storage node a service-specific recovery operation: the application must rebuild the lost member after its PVC is replaced. Use it only when that procedure has been tested.

Keep the HA profile’s default hard pod anti-affinity enabled when using one-replica Longhorn volumes. It spreads pods of each stateful service across different nodes. Pod anti-affinity and Longhorn replica anti-affinity protect different layers and don’t replace each other. Best-effort data locality does not guarantee that each volume is on its pod’s node: verify that different application members' one-replica volumes also occupy independent storage nodes. Ensure that the cluster has enough eligible nodes and spare capacity to satisfy the scheduling rules during maintenance; otherwise, replacement pods remain pending. See Configure Kubernetes affinities.
Strategy Example StorageClass Longhorn replicas Data locality Use

Default durable

longhorn-durable-2

2

Disabled

Use for all durable Non-HA and HA data unless applying the optional HA overrides below. Keep single-replica ClickHouse, HDFS NameNodes, VictoriaMetrics, Workload Observer, AI Assistant SQLite data, anomaly-detection manager data, settings, backups, StackPacks, and other state without a tested peer-recovery procedure on this class.

Application-replicated HA

longhorn-ha-database

1

Best effort

HA installations only: Elasticsearch members, HDFS DataNodes, ZooKeeper members, and replicated ClickHouse with verified application replication and volume placement. ClickHouse and Kafka have additional prerequisites below. Losing the volume loses that member and requires application-level recovery.

Transient

longhorn-transient

1

Best effort

Temporary buffers and working data, such as vmagent and temporary component PVCs. Data in these volumes can be lost during a node failure.

The mixed-class examples below target the upcoming release, SUSE Observability 0.0.1. They require its Elasticsearch StorageClass override, Kafka and ZooKeeper per-component precedence over global.storageClass, and Kafka transaction-state replication migration. On earlier releases, use the global durable class and do not apply the mixed-class HA examples below.

The following example creates the durable, HA database, and transient classes. Adapt the names and other Longhorn parameters to your environment.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-durable-2
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "2"
  dataLocality: disabled
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-ha-database
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
  numberOfReplicas: "1"
  dataLocality: best-effort
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-transient
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  numberOfReplicas: "1"
  dataLocality: best-effort

Set the two-replica durable class as the default for the chart:

global:
  storageClass: longhorn-durable-2

Changing the chart’s StorageClass value doesn’t migrate existing PVCs.

For the optional HA optimization, after meeting the release and recovery requirements above, add these overrides:

elasticsearch:
  volumeClaimTemplate:
    storageClassName: longhorn-ha-database

hbase:
  hdfs:
    datanode:
      persistence:
        storageClass: longhorn-ha-database

zookeeper:
  persistence:
    storageClass: longhorn-ha-database

stackstate:
  components:
    checks:
      tmpToPVC:
        storageClass: longhorn-transient
    healthSync:
      tmpToPVC:
        storageClass: longhorn-transient
    state:
      tmpToPVC:
        storageClass: longhorn-transient
    sync:
      tmpToPVC:
        storageClass: longhorn-transient
    vmagent:
      persistence:
        storageClass: longhorn-transient

Replicated ClickHouse

The 4000-ha sizing profile runs ClickHouse with multiple replicas. For this profile, use the HA database class when each shard has at least two healthy replicas of its application data:

clickhouse:
  persistence:
    storageClass: longhorn-ha-database

Keep ClickHouse on longhorn-durable-2 when it has only one replica per shard, including the default 150-ha, 250-ha, and 500-ha profiles.

Kafka transaction-state replication in HA

This section applies to HA installations considering the optional one-replica Longhorn class for Kafka. Before SUSE Observability 0.0.1, the 150-ha, 250-ha, and 500-ha sizing profiles configured transaction_state with replication factor one. This was the only topic with that replication gap; application topics and consumer_offsets already had HA replication. SUSE Observability 0.0.1 brings __transaction_state to replication factor two in those HA profiles and migrates the existing topic during upgrades.

In Non-HA installations, Kafka replication factor one is expected. Keep Kafka on longhorn-durable-2; the HA check and one-replica StorageClass override below do not apply.

For HA, keep Kafka on the durable class until the release requirement above is met and every partition of __transaction_state has at least two replicas with all assigned replicas in sync.

An upgrade attempts to increase the existing transaction-state topic’s replication, but can defer that migration with a warning when brokers are unavailable. A successful chart upgrade alone is not proof that the migration completed. Kafka’s HA defaults use a replication factor of two, and the transaction-state topic permits a minimum in-sync replica count of one. Losing one replica therefore leaves less redundancy; restore full replication before any further maintenance.

Check the transaction-state topic

Find a Kafka broker pod in the HA installation’s SUSE Observability namespace:

kubectl -n <suse-observability-namespace> get pods -l app.kubernetes.io/name=kafka

Run the following read-only command in one of those pods to describe __transaction_state:

kubectl -n <suse-observability-namespace> exec <kafka-pod-name> -c kafka -- kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic __transaction_state

This example uses the chart’s default plaintext client listener. If you customized the listener, use its address and port; for an authenticated listener, also supply --command-config <client-properties-path> with a client configuration available inside the pod.

Inspect every partition line in the output, not only the topic’s summary. For example, these two partition lines meet the requirement:

Topic: __transaction_state  Partition: 0  Leader: 0  Replicas: 0,1  Isr: 1,0
Topic: __transaction_state  Partition: 1  Leader: 1  Replicas: 1,2  Isr: 1,2

  • Replicas must list at least two distinct broker IDs.

  • Isr must contain every broker ID listed in Replicas; the order can differ.

  • Leader must identify an available broker, not -1.

Replicas: 0 with Isr: 0 is fully in sync but has only one copy, so it fails this check. Replicas: 0,1 with Isr: 0 has two assigned copies but one has not caught up, so it also fails. If __transaction_state has not been created yet, repeat the check after a transactional workload has created it; absence does not verify its replication.

Only after these HA checks, add the Kafka topic-data override:

kafka:
  persistence:
    storageClass: longhorn-ha-database

Optional component transaction-log PVCs

Component transaction-log PVCs store debugging data. If you enable this feature, use longhorn-transient:

stackstate:
  features:
    storeTransactionLogsToPVC:
      enabled: true
      storageClass: longhorn-transient

These PVCs do not inherit global.storageClass; the setting above applies only to component transaction logs. Most other component PVCs, including AI Assistant SQLite storage, inherit global.storageClass. See Storage classes and customization for the complete set of storage settings. After installation, verify that every PVC uses the intended class and that every Longhorn volume is healthy.

Understand the failure modes

Failure Likely effect Protection

One node or disk holding a replica fails

A two-replica volume becomes degraded but remains available, with no remaining storage redundancy until it rebuilds. A one-replica volume becomes unavailable and its data is lost if the disk cannot be recovered.

Spread replicas across independent nodes and failure domains. Restore the replica count before starting maintenance elsewhere.

Failures exceed the available replica count

Volumes become unavailable and may require restore. SUSE Observability components that depend on them become unavailable.

Maintain failure-domain separation and keep tested backups outside the cluster.

A one-replica application member is lost

The remaining application replicas may continue serving, but capacity or quorum is reduced. The lost PVC and member must be replaced and rebuilt.

Use this layout only with the HA profile, monitor application health, and test the component recovery procedure.

A disk fills or there is no destination for rebuilding

Longhorn cannot restore the requested replica count. A second failure can then cause unavailability or data loss.

Reserve rebuild capacity, alert on disk pressure and degraded volumes, and stop node maintenance until all volumes are healthy.

A PVC is deleted

With the Delete reclaim policy, the Longhorn volume can also be deleted.

Restrict destructive access and rely on external backups for recovery.

The cluster or an entire failure domain is lost

Local replicas and snapshots can become unreachable together.

Place replicas according to the required failure boundary and store SUSE Observability backups outside that boundary.

Maintain or rotate nodes

This guidance applies regardless of Kubernetes distribution or cluster management platform.

HA maintenance limitation

The current database liveness and readiness probes do not verify that application-level replication has recovered after a node disruption. Pod or node Ready status, a successful drain, and satisfied PodDisruptionBudgets are not sufficient checks. Healthy Longhorn volumes do not establish that application-level replication is complete.

Operate on one node hosting stateful SUSE Observability workloads or Longhorn replicas at a time. Before each node, confirm that:

  • SUSE Observability is healthy.

  • Application replication is complete: HDFS has no missing or under-replicated blocks, Elasticsearch has allocated its replicas, Kafka partitions have all assigned replicas in sync, and ClickHouse replicated tables have caught up with their required replicas available.

  • All Longhorn volumes are healthy and have their expected replica count.

  • Other nodes have enough storage and compute capacity for the workloads and replica rebuilds.

  • A recent external backup is available.

Automated Kubernetes upgrades, node rotations, and node restarts must perform these checks before disrupting the next node. SUSE Observability does not automatically add database-replication checks to your cluster’s maintenance workflow. Configure the workflow to wait for recovery and stop further node disruptions if a check fails or times out. If the workflow cannot perform these checks, pause it between nodes and verify recovery before continuing; a fixed delay is not a substitute.

The official Longhorn node maintenance and drain policy guide is the source of truth for maintenance procedures and drain policy behavior. The following commands are examples for planning a procedure; verify them against the documentation for your installed Longhorn version and test the complete procedure before using it in production.

Longhorn drain policies act when a node is cordoned, even if it is not subsequently drained. Change the policy before cordoning the first node and restore the previous value after maintenance.

Short restart or in-place upgrade

For a short restart, operating-system update, or in-place Kubernetes upgrade, Block For Eviction If Contains Last Replica automatically relocates the last healthy replica of a volume. This allows one-replica volumes to drain while avoiding the evacuation of every replica on the node.

For example:

LONGHORN_DRAIN_POLICY="$(kubectl -n longhorn-system get settings.longhorn.io node-drain-policy -o jsonpath='{.value}')"

kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"block-for-eviction-if-contains-last-replica"}'

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=0

Then:

  1. If the drain is blocked, inspect Longhorn events and the replicas on the node. Do not force the drain past Longhorn’s data-protection checks.

  2. Complete the maintenance and return the node.

  3. Uncordon the node so workloads can be scheduled, then wait until Longhorn has restored the expected replica count and application replication is complete before proceeding to the next node.

Allow If Last Replica Is Stopped avoids rebuilding a one-replica volume when the same node and disk are expected to return, but the volume is lost if they do not return. Use that policy only when this risk is explicitly accepted.

Permanent node removal or replacement

  1. Add replacement capacity before removing capacity.

  2. Temporarily set the Longhorn node drain policy to Block For Eviction.

  3. Cordon and drain the old node. Longhorn automatically requests eviction and blocks the drain until all replicas have moved.

  4. Confirm that no volumes remain attached, then remove the node from Longhorn and Kubernetes.

  5. Wait until Longhorn has restored the expected replica count and application replication is complete before removing another node.

For example:

LONGHORN_DRAIN_POLICY="$(kubectl -n longhorn-system get settings.longhorn.io node-drain-policy -o jsonpath='{.value}')"

kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"block-for-eviction"}'

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --timeout=0

Follow the Longhorn replica eviction procedure. Eviction rebuilds data over the storage network and can take a long time; schedule it as a storage-intensive maintenance operation.

After the maintenance window, restore the previous policy:

kubectl -n longhorn-system patch settings.longhorn.io node-drain-policy --type=merge -p '{"value":"'"${LONGHORN_DRAIN_POLICY}"'"}'

Do not use Always Allow in production.

Operational checks

Monitor degraded or faulted Longhorn volumes, unschedulable replicas, disk pressure, node readiness, and replica rebuild failures. Treat a degraded volume as reduced fault tolerance, not as a normal steady state.

Before and after maintenance, list the relevant resources:

kubectl -n longhorn-system get nodes.longhorn.io
kubectl -n longhorn-system get volumes.longhorn.io
kubectl -n <suse-observability-namespace> get pvc

Do not proceed to the next maintenance step until Longhorn has restored the expected replica count and application replication is complete. Pod readiness alone does not establish that the databases have finished rebuilding their data.