Defining or reading pipelines

In order to obtain a pipeline, you can either define one manually, or read it from a source in various formats. This guide explores all options built in Mustash.

Gather an ingest pipeline from an ElasticSearch instance

Todo

This feature is not supported by Mustash as of today.

Read a JSON document or blob representing an ingest pipeline

In order to read a pipeline from a raw ElasticSearch pipeline or list of ingest processors, you can use parse_ingest_pipeline(). For example, here’s a snippet that reads the pipeline from the raw pipeline definition:

from __future__ import annotations

from mustash.es import parse_ingest_pipeline


raw = {"name": "my-pipeline", "processors": [{"json": {"field": "raw"}}]}
pipeline = parse_ingest_pipeline(raw)

Read a pipeline from a Logstash configuration

Todo

This section is provisional, and the feature is not implemented yet.

In order to read a pipeline from the filter plugin settings in a Logstash configuration, you can use parse_from_config(). For example, here’s a snippet that reads the pipeline from a raw Logstash configuration:

from __future__ import annotations

from mustash.logstash import parse_from_config


raw_config = """
filter {
    json {
        source => "message"
    }
    mutate {
        add_field => {
            "@hello" => "wow"
        }
    }
}
"""

pipeline = parse_from_config(raw_config)

Define a pipeline manually

In order to define a pipeline manually, you must use Pipeline and subclasses of Processor or FieldProcessor. For example, here’s a snippet that defines a pipeline that parses a JSON field:

from __future__ import annotations

from mustash.core import Pipeline
from mustash.processors import JsonProcessor


pipeline = Pipeline(processors=[JsonProcessor(field="raw")])