Resources

Resources are components within Wombat that are declared with a unique label and can be referenced any number of times within a configuration. Only one instance of each named resource is created, but it is safe to use it in multiple places as they can be shared without consequence.

The following components can be defined as resources:

• inputs
• processors
• outputs
• caches
• rate limits

Some components such as caches and rate limits can only be created as a resource. However, for components where it’s optional there are a few reasons why it might be advantageous to do so.

How to use resources?

Resources are defined on the root level of the pipeline configuration file. Each resource is defined as a list under a dedicated key that corresponds to the type of resource.

Input

## Using the resource
input:
  resource: foo_input

## Defining the resource
input_resources:
  - label: foo_input
    file:
      paths: [ ./in.txt ]

Processor

## Using the resource
pipeline:
  processors:
    - resource: foo_processor

## Defining the resource
processor_resources:
  - label: foo_processor
    mapping: 'root = content.lowercase()'

Output

## Using the resource
output:
  resource: foo_output

## Defining the resource
output_resources:
  - label: foo_output
    file:
        path: ./out.txt

Cache

## Using the resource
pipeline:
  processors:
    - cache:
        resource: foo_cache
        operator: set
        key: ${! json("id") }
        value: ${! content() }

## Defining the resource
cache_resources:
  - label: foo_cache
    memory:
      default_ttl: 300s

Rate Limit

## Using the resource
pipeline:
  processors:
    - http_client:
        url: http://example.com
        verb: GET
        rate_limit: foo_rate_limit

## Defining the resource
rate_limit_resources:
  - label: foo_rate_limit
    local:
      count: 10
      interval: 1s

Why use resources?

Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component as a resource.

In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to “cleanse” the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource get_foo:

pipeline:
  processors:
    - resource: get_foo
    - catch:
        - mapping: |
            root = this
            root.content = this.content.strip_html()
        - resource: get_foo

processor_resources:
  - label: get_foo
    http:
      url: http://example.com/foo
      verb: POST
      headers:
        SomeThing: "set-to-this"
        SomeThingElse: "set-to-something-else"