Caches

Note

A cache is defined as a resource which can be used by other components. Find out more about resources in this document.

A cache is a key/value store which can be used by certain components for applications such as deduplication or data joins. Caches are configured as a named resource:

cache_resources:
  - label: foobar
    memcached:
      addresses:
        - localhost:11211
      default_ttl: 60s

Tip

It’s possible to layer caches with read-through and write-through behaviour using the multilevel cache.

And then any components that use caches have a field resource that specifies the cache resource:

pipeline:
  processors:
    - cache:
        resource: foobar
        operator: add
        key: '${! json("message.id") }'
        value: "storeme"
    - mapping: root = if errored() { deleted() }

For the simple case where you wish to store messages in a cache as an output destination for your pipeline check out the cache output. To see examples of more advanced uses of caches such as hydration and deduplication check out the cache processor.

Metrics

The following metrics are exposed for a cache:

All cache metrics have a label operation denoting the operation that triggered the metric series, one of; add, get, set or delete.

• cache_success: A count of the number of successful cache operations.
• cache_error: A count of the number of cache operations that resulted in an error.
• cache_latency_ns: Latency of operations in nanoseconds.
• cache_not_found: A count of the number of get operations that yielded no value due to the item not being found. This count is separate from cache_error.
• cache_duplicate: A count of the number of add operations that were aborted due to the key already existing. This count is separate from cache_error.