Pipeline Management

Logstash implements data collection processing through pipelines. It includes mandatory input and output plugins, optional filter plugins, and supports parallel running of multiple pipelines. This article discusses how to manage pipelines via configuration files, including creating, modifying, and deleting pipelines.

Creating Pipelines

1. Log in to the ULogstash console and enter the Logstash instance list page.

2. On the instance list page, click on the instance name to enter the instance details page, then go to the pipeline management tab and click create pipeline.

3. Fill in the pipeline name (support for letters, numbers, _)

4. Config configuration

   Users modify according to the task content.

   Parameter Description

   Parameter	Description
   input	Configuration of the input data source. For types of input data sources supported by Logstash, refer to Input plugins
   filter	Configuration for data filtering or preprocessing. For types of filter plugins supported by Logstash, refer to Filter plugins
   output	Configuration of the output data source. For types of output data sources supported by Logstash, refer to Output plugins

   For detailed pipeline task configuration, refer to Configuration file structure.

5. Pipeline parameter configuration

   Parameter	Description	Default Value
   Pipeline name	pipeline.id, the unique identifier of the pipeline	-
   Pipeline worker threads	pipeline.workers, the number of worker threads in the pipeline, which are also the worker threads running the filter and output of the pipeline in parallel	Single-node CPU core count of the instance
   Pipeline batch size	pipeline.batch.size, the maximum number of events processed in each batch	125
   Pipeline batch delay	pipeline.batch.delay, the maximum waiting time for each batch when the pipeline batch size is not met, in milliseconds	50ms
   Queue type	queue.type, the queue model used for event buffering, the optional value is memory (in-memory queue). Changes are not currently supported	memory
   Maximum byte of queue	queue.max_bytes, when choosing the persisted queue type, the maximum byte count that can be stored in the queue. Make sure this value is less than the disk capacity of the single-node instance. Changes are not currently supported	1024MB
   Number of queue checkpoint writes	queue.checkpoint.acks, when choosing the persisted queue type, the maximum number of written events in forceful checkpoint execution. If set to 0, means unlimited. Changes are not currently supported	1024

Modifying Pipelines

Changes to pipelines need to be restarted to take effect

1. In the pipeline list, click the modification button on the right side of the pipeline you want to modify to enter the pipeline modification page.
2. Modify the task configuration and parameter configuration of the pipeline on the pipeline modification page.
3. Click save to complete pipeline modification, the modified content will take effect when the pipeline is restarted next time.

Deleting Pipelines

Note:

• Pipelines cannot be recovered after deletion
• Running pipelines need to be stopped first before deletion can be carried out

1. In the pipeline list, find the pipeline to be deleted, and click delete in the operation list on the right.