Disk Limit Configuration
The disk management methods include automatic log cleaning and non-operation. Cleaning logs means that when the disk reaches the set threshold, it starts cleaning log files to prevent the UKafka service from becoming unavailable due to the disk being filled up.
Log Cleaning
When the disk usage rate of a node in the cluster reaches the threshold you have set, UKafka will start automatic log cleaning.
Cleaning policy: According to the actual proportion of each topic on the current node to the total capacity of the disk, the use of each topic is calculated proportionally to be cleaned to below the disk threshold. This value is used to set the topic retention size. When the disk usage rate drops below the threshold, the retention size setting of the topic will be restored.
Each topic partition on the disk will correspond to multiple segment data files, and the segment file is the smallest operable unit, so the smallest granularity of cleaning is also 1G.
When the cluster disk usage reaches the threshold, if the actual usage of the topic partition is less than 2% of the node disk, to prevent the topic from becoming unavailable due to too small a limit, the cleaning service will skip these topics.
When the topic usage grows too fast, disk cleaning may be triggered frequently. The disk cleaning strategy can prevent the Kafka service from becoming unavailable due to the disk being filled up, but the cleaning strategy can only passively decide based on the actual topic ratio. It is not the plan that best meets the actual business needs, so when this situation occurs, it is recommended that you adjust the topic retention strategy according to your own business.
If all the topic partitions in the cluster have not reached the cleaning limit size, but because there are too many topic partitions, the disk usage rate reaches the threshold, the log cleaning service will not work at this time. Users must adjust the topic or expand the cluster in this situation.
Enable Log Cleaning
This function can be enabled during cluster creation and cluster operation.
- When creating a cluster
The default disk management method is “log cleaning”. It is not recommended to choose “non-operation”, which will cause the kafka service to be unavailable after the disk is filled.
The default disk usage threshold is 80%, and the optional range is 70% to 90%.
- During cluster operation
Click on [Disk Limit], and then you can modify the disk management method and disk usage threshold. The adjustment takes effect immediately.
No Operation
If you choose not to manage the disk by UKafka, we highly recommend that you enable disk usage rate alarms in monitoring. When the disk usage rate reaches 80%, you can clean the old logs of each topic in the console.
The cleaning method is to modify the Topic Parameter Configuration and appropriately reduce the “log retention time”.