mustash.es – ElasticSearch ingest pipeline conversion utilities

pydantic model mustash.es.ESProcessor

Bases: ABC, BaseModel

ElasticSearch processor.

This class is used for parsing and rendering ElasticSearch ingest pipelines, in order to ensure that we check all options, forbid additional options, and so on.

Config:

• extra: str = forbid

Fields:

• description (str | None)

• if_ (str | None)

• ignore_failure (bool)

• on_failure (list[mustash.es._ESProcessorWrapper] | None)

• tag (str | None)

field description: str | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESAppendProcessor

Bases: ESProcessor

ElasticSearch append processor.

See Append processor for more information.

Config:

• extra: str = forbid

Fields:

• allow_duplicates (bool)

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• tag ()

• value (Element | list[Element])

field allow_duplicates: bool = True

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field value: Element | list[Element] [Required]

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESBytesProcessor

Bases: ESProcessor

ElasticSearch bytes processor.

See Bytes processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESCommunityIDProcessor

Bases: ESProcessor

ElasticSearch Community ID processor.

See Community ID processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• destination_ip (str)

• destination_port (str)

• iana_number (str)

• icmp_code (str)

• icmp_type (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• seed (Annotated[int, Ge(0), Lt(65536)])

• source_ip (str)

• source_port (str)

• tag ()

• target_field (str)

• transport (str)

field description: str | None = None

field destination_ip: str = 'destination.ip'

field destination_port: str = 'destination.port'

field iana_number: str = 'network.iana_number'

field icmp_code: str = 'icmp.code'

field icmp_type: str = 'icmp.type'

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = True

field on_failure: list[_ESProcessorWrapper] | None = None

field seed: Annotated[int, Ge(0), Lt(65536)] = 0

Constraints:

• ge = 0

• lt = 65536

field source_ip: str = 'source.ip'

field source_port: str = 'source.port'

field tag: str | None = None

field target_field: str = 'network.community_id'

field transport: str = 'network.transport'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESConvertProcessor

Bases: ESProcessor

ElasticSearch convert processor.

See Convert processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

• type (Literal['integer', 'long', 'float', 'double', 'string', 'boolean', 'ip', 'auto'])

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

field type: Literal['integer', 'long', 'float', 'double', 'string', 'boolean', 'ip', 'auto'] [Required]

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESCSVProcessor

Bases: ESProcessor

ElasticSearch CSV processor.

See CSV processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• empty_value (str)

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• quote (Annotated[str, StringConstraints(min_length=1, max_length=1)])

• separator (Annotated[str, StringConstraints(min_length=1, max_length=1)])

• tag ()

• target_fields (list[str])

• trim (bool)

field description: str | None = None

field empty_value: str = ''

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field quote: Annotated[str, StringConstraints(min_length=1, max_length=1)] = '"'

Constraints:

• min_length = 1

• max_length = 1

field separator: Annotated[str, StringConstraints(min_length=1, max_length=1)] = ','

Constraints:

• min_length = 1

• max_length = 1

field tag: str | None = None

field target_fields: list[str] [Required]

field trim: bool = False

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESDateProcessor

Bases: ESProcessor

ElasticSearch date processor.

See Date processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• formats (list[str])

• if_ ()

• ignore_failure ()

• locale (str)

• on_failure ()

• output_format (str)

• tag ()

• target_field (str)

• timezone (str)

field description: str | None = None

field field: str [Required]

field formats: list[str] [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field locale: str = 'ENGLISH'

field on_failure: list[_ESProcessorWrapper] | None = None

field output_format: str = "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"

field tag: str | None = None

field target_field: str = '@timestamp'

field timezone: str = 'UTC'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESDateIndexNameProcessor

Bases: ESProcessor

ElasticSearch date index name processor.

See Date index name processor for more information.

Config:

• extra: str = forbid

Fields:

• date_formats (str | list[str])

• date_rounding (Literal['y', 'M', 'w', 'd', 'h', 'm', 's'])

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• index_name_format (str)

• index_name_prefix (str | None)

• locale (str)

• on_failure ()

• tag ()

• timezone (str)

field date_formats: str | list[str] = "yyyy-MM-dd'T'HH:mm:ss.SSSXX"

field date_rounding: Literal['y', 'M', 'w', 'd', 'h', 'm', 's'] [Required]

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field index_name_format: str = 'yyyy-MM-dd'

field index_name_prefix: str | None = None

field locale: str = 'ENGLISH'

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field timezone: str = 'UTC'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESDissectProcessor

Bases: ESProcessor

ElasticSearch dissect processor.

See Dissect processor for more information.

Config:

• extra: str = forbid

Fields:

• append_separator (str)

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• pattern (str)

• tag ()

field append_separator: str = ''

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field pattern: str [Required]

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESDotExpander

Bases: ESProcessor

ElasticSearch dot expander processor.

See Dot expander processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• override (bool)

• path (str | None)

• tag ()

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field override: bool = False

field path: str | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESDropProcessor

Bases: ESProcessor

ElasticSearch drop processor.

See Drop processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• if_ ()

• ignore_failure ()

• on_failure ()

• tag ()

field description: str | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESFailProcessor

Bases: ESProcessor

ElasticSearch fail processor.

See Fail processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• if_ ()

• ignore_failure ()

• message (str)

• on_failure ()

• tag ()

field description: str | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field message: str [Required]

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESFingerprintProcessor

Bases: ESProcessor

ElasticSearch fingerprint processor.

See Fingerprint processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• fields (list[str])

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• method (Literal['MD5', 'SHA-1', 'SHA-256', 'SHA-512', 'MurmurHash3'])

• on_failure ()

• salt (str | None)

• tag ()

• target_field (str)

field description: str | None = None

field fields: list[str] [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field method: Literal['MD5', 'SHA-1', 'SHA-256', 'SHA-512', 'MurmurHash3'] = 'SHA-1'

field on_failure: list[_ESProcessorWrapper] | None = None

field salt: str | None = None

field tag: str | None = None

field target_field: str = 'fingerprint'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESGeoIPProcessor

Bases: ESProcessor

ElasticSearch GeoIP processor.

See GeoIP processor for more information.

Config:

• extra: str = forbid

Fields:

• database_file (str)

• description ()

• download_database_on_pipeline_creation (bool)

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• properties (list[str])

• tag ()

• target_field (str)

field database_file: str = 'GeoLite2-City.mmdb'

field description: str | None = None

field download_database_on_pipeline_creation: bool = True

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field properties: list[str] = ['continent_name', 'country_iso_code', 'country_name', 'region_iso_code', 'region_name', 'city_name', 'location']

field tag: str | None = None

field target_field: str = '@timestamp'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESGrokProcessor

Bases: ESProcessor

ElasticSearch grok processor.

See Grok processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• ecs_compatibility (Literal['disabled', 'v1'])

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• pattern_definitions (dict[str, str] | None)

• patterns (list[str])

• tag ()

• trace_match (bool)

field description: str | None = None

field ecs_compatibility: Literal['disabled', 'v1'] = 'disabled'

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field pattern_definitions: dict[str, str] | None = None

field patterns: list[str] [Required]

field tag: str | None = None

field trace_match: bool = False

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESGsubProcessor

Bases: ESProcessor

ElasticSearch gsub processor.

See Gsub processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• pattern (str)

• replacement (str)

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field pattern: str [Required]

field replacement: str [Required]

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESHTMLStripProcessor

Bases: ESProcessor

ElasticSearch HTML strip processor.

See HTML strip processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESJoinProcessor

Bases: ESProcessor

ElasticSearch join processor.

See Join processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• separator (str)

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field separator: str [Required]

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESJSONProcessor

Bases: ESProcessor

ElasticSearch JSON processor.

See JSON processor for more information.

Config:

• extra: str = forbid

Fields:

• add_to_root (bool)

• add_to_root_conflict_strategy (Literal['replace', 'merge'])

• allow_duplicate_keys (bool)

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• strict_json_parsing (bool)

• tag ()

• target_field (str | None)

field add_to_root: bool = False

field add_to_root_conflict_strategy: Literal['replace', 'merge'] = 'replace'

field allow_duplicate_keys: bool = False

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field strict_json_parsing: bool = False

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESKVProcessor

Bases: ESProcessor

ElasticSearch KV processor.

See KV processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• exclude_keys (list[str] | None)

• field (str)

• field_split (re.Pattern)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• include_keys (list[str] | None)

• on_failure ()

• prefix (str)

• strip_brackets (bool)

• tag ()

• target_field (str | None)

• trim_key (str)

• trim_value (str)

• value_split (re.Pattern)

field description: str | None = None

field exclude_keys: list[str] | None = None

field field: str [Required]

field field_split: re.Pattern [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field include_keys: list[str] | None = None

field on_failure: list[_ESProcessorWrapper] | None = None

field prefix: str = ''

field strip_brackets: bool = False

field tag: str | None = None

field target_field: str | None = None

field trim_key: str = ''

field trim_value: str = ''

field value_split: re.Pattern [Required]

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESLowercaseProcessor

Bases: ESProcessor

ElasticSearch lowercase processor.

See Lowercase processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESNetworkDirectionProcessor

Bases: ESProcessor

ElasticSearch network direction processor.

See Network direction processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• destination_ip (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• internal_networks (list[str] | None)

• internal_networks_field (str | None)

• on_failure ()

• source_ip (str)

• tag ()

• target_field (str)

Validators:

• _validate » all fields

field description: str | None = None

field destination_ip: str = 'destination.ip'

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = True

field internal_networks: list[str] | None = None

field internal_networks_field: str | None = None

field on_failure: list[_ESProcessorWrapper] | None = None

field source_ip: str = 'source.ip'

field tag: str | None = None

field target_field: str = 'network.direction'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESRedactProcessor

Bases: ESProcessor

ElasticSearch redact processor.

See Redact processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• pattern_definitions (dict[str, str] | None)

• patterns (list[str])

• prefix (str)

• suffix (str)

• tag ()

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field pattern_definitions: dict[str, str] | None = None

field patterns: list[str] [Required]

field prefix: str = '<'

field suffix: str = '>'

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESRegisteredDomainProcessor

Bases: ESProcessor

ElasticSearch registered domain processor.

See Registered domain processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = True

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str = ''

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESRemoveProcessor

Bases: ESProcessor

ElasticSearch remove processor.

See Remove processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str | list[str] | None)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• keep (str | list[str] | None)

• on_failure ()

• tag ()

Validators:

• _validate » all fields

field description: str | None = None

field field: str | list[str] | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field keep: str | list[str] | None = None

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESRenameProcessor

Bases: ESProcessor

ElasticSearch rename processor.

See Rename processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• override (bool)

• tag ()

• target_field (str)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field override: bool = False

field tag: str | None = None

field target_field: str [Required]

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESRerouteProcessor

Bases: ESProcessor

ElasticSearch reroute processor.

See Reroute processor for more information.

Config:

• extra: str = forbid

Fields:

• dataset (str)

• description ()

• destination (str | None)

• if_ ()

• ignore_failure ()

• namespace (str)

• on_failure ()

• tag ()

field dataset: str = '{{data_stream.dataset}}'

field description: str | None = None

field destination: str | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field namespace: str = '{{data_stream.namespace}}'

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESScriptProcessor

Bases: ESProcessor

ElasticSearch script processor.

See Script processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• id (str | None)

• if_ ()

• ignore_failure ()

• lang (Literal['painless'])

• on_failure ()

• params (dict[str, Any] | None)

• source (str | dict | None)

• tag ()

field description: str | None = None

field id: str | None = None

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field lang: Literal['painless'] = 'painless'

field on_failure: list[_ESProcessorWrapper] | None = None

field params: dict[str, Any] | None = None

field source: str | dict | None = None

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESSetProcessor

Bases: ESProcessor

ElasticSearch set processor.

See Set processor for more information.

Config:

• extra: str = forbid

Fields:

• copy_from (str | None)

• description ()

• field (str)

• if_ ()

• ignore_empty_value (bool)

• ignore_failure ()

• media_type (str)

• on_failure ()

• override (bool)

• tag ()

• value (str | None)

Validators:

• _validate » all fields

field copy_from: str | None = None

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_empty_value: bool = False

field ignore_failure: bool = False

field media_type: str = 'application/json'

field on_failure: list[_ESProcessorWrapper] | None = None

field override: bool = True

field tag: str | None = None

field value: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESSetSecurityUserProcessor

Bases: ESProcessor

ElasticSearch set security user processor.

See Set security user processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• properties (list[str])

• tag ()

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field properties: list[str] = ['username', 'roles', 'email', 'full_name', 'metadata', 'api_key', 'realm', 'authentication_type']

field tag: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Raises:

NotImplementedError – No processor is currently available for this configuration of the processor.

Return type:

Processor

pydantic model mustash.es.ESSortProcessor

Bases: ESProcessor

ElasticSearch sort processor.

See Sort processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• on_failure ()

• order (Literal['asc', 'desc'])

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field order: Literal['asc', 'desc'] [Required]

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESSplitProcessor

Bases: ESProcessor

ElasticSearch split processor.

See Split processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• preserve_trailing (bool)

• separator (re.Pattern)

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field preserve_trailing: bool = False

field separator: re.Pattern [Required]

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESTrimProcessor

Bases: ESProcessor

ElasticSearch trim processor.

See Trim processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESUppercaseProcessor

Bases: ESProcessor

ElasticSearch uppercase processor.

See Uppercase processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESURIPartsProcessor

Bases: ESProcessor

ElasticSearch URI parts processor.

See URI parts processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• keep_original (bool)

• on_failure ()

• remove_if_successful (bool)

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field keep_original: bool = True

field on_failure: list[_ESProcessorWrapper] | None = None

field remove_if_successful: bool = False

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESURLDecodeProcessor

Bases: ESProcessor

ElasticSearch URL decode processor.

See URL decode processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• tag ()

• target_field (str | None)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field tag: str | None = None

field target_field: str | None = None

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

pydantic model mustash.es.ESUserAgentProcessor

Bases: ESProcessor

ElasticSearch user agent processor.

See User agent processor for more information.

Config:

• extra: str = forbid

Fields:

• description ()

• field (str)

• if_ ()

• ignore_failure ()

• ignore_missing (bool)

• on_failure ()

• properties (list[str])

• regex_file (str | None)

• tag ()

• target_field (str)

field description: str | None = None

field field: str [Required]

field if_: Annotated[str | None, Field(alias='if')] = None (alias 'if')

field ignore_failure: bool = False

field ignore_missing: bool = False

field on_failure: list[_ESProcessorWrapper] | None = None

field properties: list[str] = ['name', 'major', 'minor', 'patch', 'build', 'os', 'os_name', 'os_major', 'os_minor', 'device']

field regex_file: str | None = None

field tag: str | None = None

field target_field: str = 'user_agent'

build(cls: type[Processor], /, **kwargs) → Processor

Obtain a Mustash processor out of the current processor.

This also manages common parameters for all processors.

Parameters:

cls (type[Processor])

Return type:

Processor

convert() → Processor

Convert the ElasticSearch processor into a Mustash processor.

Returns:

Converted processor.

Return type:

Processor

class mustash.es.ESIngestPipelineParser(*, name: str | None = None, processors: dict[str, type[ESProcessor]] | None = None)

Bases: object

ElasticSearch ingest pipeline converter for mustash.

Parameters:

• name (str | None) – Optional name with which the parser wants to be represented.

• processors (dict[str, type[ESProcessor]] | None) – Processors supported by the pipeline.

copy(*, with_processors: dict[str, ESProcessor] | None = None, without_processors: Iterable[str] | None = None) → ESIngestPipelineParser

Copy the parser.

Parameters:

• with_processors (dict[str, ESProcessor] | None) – Processors to add in the new parser. If the key exists in the current parser, the processor will be replaced automatically in the new parser.

• without_processors (Iterable[str] | None) – Processors to remove from the current parser.

Returns:

New parser with the modified processors.

Return type:

ESIngestPipelineParser

validate(raw: Any, /) → list[dict]

Validate the provided pipeline.

Parameters:

raw (Any) – Pipeline or processor list dictionary, or JSON-encoded version of the same.

Returns:

Validated object, as Python.

Return type:

list[dict]

convert(raw: Any, /) → Pipeline

Convert a raw list of processors into a list of processors.

Parameters:

raw (Any) – Pipeline or processor list dictionary, or JSON-encoded version of the same.

Returns:

Decoded processor.

Return type:

Pipeline

mustash.es.DEFAULT_INGEST_PIPELINE_PARSER = DEFAULT_INGEST_PIPELINE_PARSER

Default ElasticSearch ingest pipeline parser instance.

This instance defines all of the default processors available in all contexts, including on ElasticSearch and in Logstash’s elastic_integration filter.

mustash.es.parse_ingest_pipeline(raw: Any, /, *, parser: ESIngestPipelineParser = DEFAULT_INGEST_PIPELINE_PARSER) → Pipeline

Parse an ElasticSearch ingest pipeline.

Parameters:

• raw (Any) – Raw ingest pipeline, either as a dictionary or a raw JSON-encoded string.

• parser (ESIngestPipelineParser) – Parser to use to read the pipeline.

Returns:

Parsed pipeline.

Return type:

Pipeline

mustash.es.validate_ingest_pipeline(raw: Any, /, *, parser: ESIngestPipelineParser = DEFAULT_INGEST_PIPELINE_PARSER) → list[dict]

Validate an ElasticSearch ingest pipeline.

Parameters:

• raw (Any) – Raw ingest pipeline to validate, either as a dictionary or a raw JSON-encoded string.

• parser (ESIngestPipelineParser)

Returns:

Validated ElasticSearch processors.

Return type:

list[dict]

mustash.es.render_as_ingest_pipeline(pipeline: Pipeline, /) → list

Render a list of processors as an ElasticSearch ingest pipeline.

Parameters:

pipeline (Pipeline) – Pipeline to render as an ElasticSearch ingest pipeline.

Returns:

Rendered pipeline.

Raises:

ValueError – The pipeline is not renderable as an ElasticSearch ingest pipeline.

Return type:

list