SFTP File Uploader

Secure File Transfer Protocol (SFTP) can be used for exporting files the following file types:

• json
• csv
• compressed csv

Prerequisites

Exports to an SFTP location require that you create an SFTP key in the Key Editor with a Username, Host, Port and Authentication type (Password or Private Key) is necessary to utilize this upload configuration,

Parameters

compression string

optional

Indicates the compression method. Accepted values include gzip. If utilized, the configured file extension in the destination parameter must end with .gz

destination string

required

The path to the location where the file will be uploaded. If the path does not exist, it will be created.

 

The following date patterns may be used to fill in the current date automatically.

• 2022-06-01 (YYYY-MM-DD)

• 2022-6-1 (single digit month and date)

• 2022-Jun-1 (three letter month abbreviation)

• 2022-June-1 (full month name)

 

The string UUID will be replaced with an eight digit alphanumeric string to ensure the uniqueness of the filename. It may be used in conjunction with date formatting such that /switchboard-example/fileYYYYMMDD_UUID.csv.gz will be formatted as ../file20220727_30FG06et.csv.gz

 

We strongly recommend that if you are going to use these two patterns together you use the date first so that you can sort the files by date.

format string

required

The format data should be uploaded as. Allowed values incude csv, json, parquet, and avro

headers boolean

optional

Indicates whether or not a header row should be written to the destination file (for csv files)

partition_count integer

optional

Some systems can’t handle really large files and as such uploads must be “partitioned” (made smaller). Typically you want only one partition, which is the default, but it may be convient if downstream processes consuming the data require smaller files.

primary_source_name string

required

The name of the initial download from which data is being uploaded. Required for uploads where date a pattern is utilized in the naming convention.

test_destination string

optional

The location to write data for the purpose of test runs — unless this is specified, test data will not be uploaded. Note that date based variables cannot be used in a test destination name.

Switchboard Script Syntax

upload example_sftp_report to {
    type: "sftp_ng";
    key: "my_sftp_key";
    
    destination: "/My_SFTP_Target/example_sftp_report_name_YYYY-MM-DD.csv";
    format: "csv";
    compression: "none"; //option compression method, if utilized, the configured file extension in destination must end with with the specified compression file extension 
    partition_count: 1;
    headers: true;

	primary_source_name: "example_sftp_report_raw"; //only required for uploads where date is utilized in the naming convention, corresponds with the initial download name

    test_destination: "/My_SFTP_Target/example_sftp_report_name_test.csv"; //Test destinations are supported by using this field
};