Logfmt Parser Transform

The Vector logfmt_parser transform parses logs

Configuration

• Common
• Advanced

• vector.toml
• vector.yaml
• vector.json

[transforms.my_transform_id]
  # General
  type = "logfmt_parser" # required
  inputs = ["my-source-or-transform-id"] # required
  drop_field = true # optional, default
  field = "message" # optional, default

  # Types
  types.status = "int" # example
  types.duration = "float" # example
  types.success = "bool" # example
  types.timestamp_iso8601 = "timestamp|%F" # example
  types.timestamp_custom = "timestamp|%a %b %e %T %Y" # example
  types.parent.child = "int" # example

• commonoptionalbool

  drop_field

  If the specified field should be dropped (removed) after parsing.

  • Default: true
  • View examples
• commonoptionalstring

  field

  The log field to parse. See Format Specification for more info.

  • Default: "message"
  • View examples
• commonoptionaltable

  types

  Key/value pairs representing mapped log field names and types. This is used to coerce log fields into their proper types.

Output

Telemetry

This component provides the following metrics that can be retrieved through the internal_metrics source. See the metrics section in the monitoring page for more info.

• counter

  processing_errors_total

  The total number of processing errors encountered by this component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • error_type - The type of the error

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  processed_events_total

  The total number of events processed by this component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • file - The file that produced the error

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

• counter

  processed_bytes_total

  The total number of bytes processed by the component. This metric includes the following tags:

  • component_kind - The Vector component kind.

  • component_name - The Vector component ID.

  • component_type - The Vector component type.

  • instance - The Vector instance identified by host and port.

  • job - The name of the job producing Vector metrics.

Examples

• Heroku Router Log
• Loosely Structured

Given the following Vector event:

{
  "log": {
    "message": "at=info method=GET path=/ host=myapp.herokuapp.com request_id=8601b555-6a83-4c12-8269-97c8e32cdb22 fwd=\"204.204.204.204\" dyno=web.1 connect=1ms service=18ms status=200 bytes=13 tls_version=tls1.1 protocol=http"
  }
}

And the following configuration:

[transforms.logfmt_parser]
type = "logfmt_parser"
field = "message"
drop_field = true
types.bytes = "int"
types.status = "int"

The following Vector log event will be output:

{
  "at": "info",
  "method": "GET",
  "path": "/",
  "host": "myapp.herokuapp.com",
  "request_id": "8601b555-6a83-4c12-8269-97c8e32cdb22",
  "fwd": "204.204.204.204",
  "dyno": "web.1",
  "connect": "1ms",
  "service": "18ms",
  "status": 200,
  "bytes": 13,
  "tls_version": "tls1.1",
  "protocol": "http"
}

How It Works

Format Specification

Logfmt is, unfortunately, a very loosely defined format. There is no official specification for the format and Vector makes a best effort to parse key/value pairs delimited with a =. It works by splitting the field's value on non-quoted white-space and then splitting each token by a non-quoted = character. This makes the parsing process somewhat flexible in that the string does not need to be strictly formatted.

For example, the following log line:

log event

{
  "message": "Hello world duration=2s user-agent="Firefox/47.3 Mozilla/5.0""
}

Will be successfully parsed into:

log event

{
  "message": "Hello world duration=2s user-agent="Firefox/47.3 Mozilla/5.0"",
  "duration": "2s",
  "user-agent": "Firefox/47.3 Mozilla/5.0"
}

Key/Value Parsing

This transform can be used for key/value parsing. Logfmt refers to a loosely defined spec that parses a key/value pair delimited by a = character. This section, and it's keywords, is primarily added to assist users in finding this transform for these terms.

Quoting Values

Values can be quoted to capture spaces, and quotes can be escaped with \. For example

key1="value with spaces" key2="value with spaces and \""

Would result in the following log event:

log event

{
  "key1": "value with spaces",
  "key2": "value with spaces and \""
}