Kafka, Kubernetes and Not enough space
So I came across a situation and it took me a while to figure it out. So I'm putting this together as it might help others as well.
Let's say you have Bitnami's packaged Kafka cluster running on Kubernetes on StatefulSets, (so not managed by an operator like Strimzi). And the pods start restarting...
Looking at the logs you might see stuff like this:
-9ee616bdcf73, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) with 1 segments in 2ms (32564/32564 loaded in /bitnami/kafka/data) (kafka.log.LogManager)
[2022-07-06 20:46:05,895] INFO Loaded 32564 logs in 161618ms. (kafka.log.LogManager)
[2022-07-06 20:46:05,896] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-07-06 20:46:05,896] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-07-06 20:46:05,908] INFO Starting the log cleaner (kafka.log.LogCleaner)
[2022-07-06 20:46:05,999] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
[2022-07-06 20:46:06,296] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Starting (kafka.server.BrokerToControllerRequestThread)
[2022-07-06 20:46:06,419] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
[2022-07-06 20:46:06,428] INFO Awaiting socket connections on 0.0.0.0:9093. (kafka.network.Acceptor)
[2022-07-06 20:46:06,463] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(INTERNAL) (kafka.network.SocketServer)
[2022-07-06 20:46:06,464] INFO Updated connection-accept-rate max connection creation rate to 2147483647 (kafka.network.ConnectionQuotas)
[2022-07-06 20:46:06,464] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2022-07-06 20:46:06,473] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Created data-plane acceptor and processors for endpoint : ListenerName(CLIENT) (kafka.network.SocketServer)
[2022-07-06 20:46:06,481] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Starting (kafka.server.BrokerToControllerRequestThread)
[2022-07-06 20:46:06,508] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-07-06 20:46:06,509] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-07-06 20:46:06,511] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-07-06 20:46:06,512] INFO [ExpirationReaper-0-ElectLeader]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-07-06 20:46:06,529] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2022-07-06 20:46:06,611] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2022-07-06 20:46:06,642] INFO Stat of the created znode at /brokers/ids/0 is: 511242113,511242113,1657140366624,1657140366624,1,0,0,72062637399277581,364,0,511242113
(kafka.zk.KafkaZkClient)
[2022-07-06 20:46:06,643] INFO Registered broker 0 at path /brokers/ids/0 with addresses: INTERNAL://kafka-0.kafka-headless.default.svc.cluster.local:9093,CLIENT://kafka-0.kafka-headless.default.svc.cluster.local:9092, czxid (broker epoch): 511242113 (kafka.zk.KafkaZkClient)
[2022-07-06 20:46:06,758] INFO [ControllerEventThread controllerId=0] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2022-07-06 20:46:06,765] INFO …