Defining or reading pipelines¶
In order to obtain a pipeline, you can either define one manually, or read it from a source in various formats. This guide explores all options built in Mustash.
Gather an ingest pipeline from an ElasticSearch instance¶
Todo
This feature is not supported by Mustash as of today.
Read a JSON document or blob representing an ingest pipeline¶
In order to read a pipeline from a raw ElasticSearch pipeline or list of
ingest processors, you can use parse_ingest_pipeline()
.
For example, here’s a snippet that reads the pipeline from the raw pipeline
definition:
from __future__ import annotations
from mustash.es import parse_ingest_pipeline
raw = {"name": "my-pipeline", "processors": [{"json": {"field": "raw"}}]}
pipeline = parse_ingest_pipeline(raw)
Read a pipeline from a Logstash configuration¶
Todo
This section is provisional, and the feature is not implemented yet.
In order to read a pipeline from the filter plugin settings in a Logstash
configuration, you can use parse_from_config()
.
For example, here’s a snippet that reads the pipeline from a raw Logstash
configuration:
from __future__ import annotations
from mustash.logstash import parse_from_config
raw_config = """
filter {
json {
source => "message"
}
mutate {
add_field => {
"@hello" => "wow"
}
}
}
"""
pipeline = parse_from_config(raw_config)
Define a pipeline manually¶
In order to define a pipeline manually, you must use Pipeline
and subclasses of Processor
or FieldProcessor
.
For example, here’s a snippet that defines a pipeline that parses a JSON field:
from __future__ import annotations
from mustash.core import Pipeline
from mustash.processors import JsonProcessor
pipeline = Pipeline(processors=[JsonProcessor(field="raw")])