Skip to content
Wombat

Output

An output is a sink where we wish to send our consumed data after applying an optional array of processors. Only one output is configured at the root of a Wombat config. However, the output can be a broker which combines multiple outputs under a chosen brokering pattern, or a switch which is used to multiplex against different outputs.

An output config section looks like this:

output:
  label: my_s3_output


  aws_s3:
    bucket: TODO
    path: '${! metadata("kafka_topic") }/${! json("message.id") }.json'


  # Optional list of processing steps
  processors:
    - mapping: '{"message":this,"meta":{"link_count":this.links.length()}}'

Back Pressure

Wombat outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Wombat will gracefully stop consuming until the issue is resolved.

Retrying

When a Wombat output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka).

It’s possible to instead have Wombat indefinitely retry an output until success with a retry output. Some other outputs, such as the broker output, might also retry indefinitely depending on their configuration.

Output Processors

Output processors are functions applied to messages just before they are sent to the output. They are entirely optional but can be used to transform the message content, or to add metadata to the message. For example, imagine you want to add a count of the number of links in a message to the message itself before sending the message to an S3 bucket:

output:
  aws_s3:
    bucket: TODO
  processors:
    - mapping: |-
        root.message = this
        root.meta.link_count = this.links.length()

Metrics

The following metrics are exposed for an output:

  • output_sent: A count of the number of messages sent by the output.
  • output_batch_sent: A count of the number of message batches sent by the output.
  • output_error: A count of the number of send attempts that have failed. On failed batched sends this count is incremented once only.
  • output_latency_ns: Latency of writes in nanoseconds. This metric may not be populated by outputs that are pull-based such as the http_server.
  • batch_created: A count of each time an output-level batch has been created using a batching policy. Includes a label mechanism describing the particular mechanism that triggered it, one of; count, size, period, check.
  • output_connection_up: For continuous stream based outputs represents a count of the number of the times the output has successfully established a connection to the target sink. For poll based outputs that do not retain an active connection this value will increment once.
  • output_connection_failed: For continuous stream based outputs represents a count of the number of times the output has failed to establish a connection to the target sink.
  • output_connection_lost: For continuous stream based outputs represents a count of the number of times the output has lost a previously established connection to the target sink.