Skip to content
Wombat

sftp

Consumes files from an SFTP server.

Introduced in version 3.39.0.

# Common config fields, showing default values
input:
  label: ""
  sftp:
    address: "" # No default (required)
    credentials:
      username: ""
      password: ""
      host_public_key_file: "" # No default (optional)
      host_public_key: "" # No default (optional)
      private_key_file: "" # No default (optional)
      private_key: "" # No default (optional)
      private_key_pass: ""
    paths: [] # No default (required)
    auto_replay_nacks: true
    scanner:
      to_the_end: {}
    watcher:
      enabled: false
      minimum_age: 1s
      poll_interval: 1s
      cache: ""

Metadata

This input adds the following metadata fields to each message:

  • sftp_path
  • sftp_mod_time

You can access these metadata fields using function interpolation.

Fields

address

The address of the server to connect to.

Type: string

connection_timeout

The connection timeout to use when connecting to the target server.

Type: string

Default: "30s"

credentials

The credentials to use to log into the target server.

Type: object

credentials.username

The username to authenticate with the SFTP server.

Type: string

Default: ""

credentials.password

The password for the specified username to connect to the SFTP server.

Type: string

Default: ""

credentials.host_public_key_file

The path to the SFTP server’s public key file, used for host key verification.

Type: string

credentials.host_public_key

The raw contents of the SFTP server’s public key, used for host key verification.

Type: string

credentials.private_key_file

The path to the private key file, used for authenticating the username.

Type: string

credentials.private_key

The raw contents of the private key, used for authenticating the username.

Type: string

credentials.private_key_pass

Optional passphrase for decrypting the private key, if it’s encrypted.

Type: string

Default: ""

max_sftp_sessions

The maximum number of SFTP sessions.

Type: int

Default: 10

paths

A list of paths to consume sequentially. Glob patterns are supported.

Type: array

auto_replay_nacks

Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to false these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation.

Type: bool

Default: true

scanner

The scanner by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the csv scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once.

Type: scanner

Default: {"to_the_end":{}} Requires version 4.25.0 or newer

delete_on_finish

Whether to delete files from the server once they are processed.

Type: bool

Default: false

watcher

An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files.

Type: object

Requires version 3.42.0 or newer

watcher.enabled

Whether file watching is enabled.

Type: bool

Default: false

watcher.minimum_age

The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to.

Type: string

Default: "1s"

# Examples


minimum_age: 10s


minimum_age: 1m


minimum_age: 10m

watcher.poll_interval

The interval between each attempt to scan the target paths for new files.

Type: string

Default: "1s"

# Examples


poll_interval: 100ms


poll_interval: 1s

watcher.cache

A cache resource for storing the paths of files already consumed.

Type: string

Default: ""