SFTP Upload Target#
The SFTP upload target is used for uploading data which resides either on the local machine or in a Hadoop compatible file system (HDFS, S3, …) to an external SFTP server.
Plugin#
The sftpUpload
target is provided by the SFTP Plugin, which needs to be enabled in your
namespace.yml
file. See namespace documentation for more information for configuring plugins.
Example#
connections:
my-sftp-connection:
kind: ssh
host: "sftp.server.dimajix.net"
port: "22"
username: "testuser"
password: "12345678"
keyFile: "/home/user/private_key"
jobs:
main:
upload_data:
kind: sftpUpload
description: "Upload some data to a SFTP server"
connection: my-sftp-connection
source: "$hdfs_export_basedir/export_table/${processing_date}"
target: "${sftp_target}/inbox/_${processing_date}.txt"
merge: true
overwrite: true
You can also directly embed the connection into the target as follows:
jobs:
main:
upload_data:
kind: sftpUpload
description: "Upload some data to a SFTP server"
connection:
kind: ssh
host: "sftp.server.dimajix.net"
port: "22"
username: "testuser"
password: "12345678"
keyFile: "/home/user/private_key"
source: "$hdfs_export_basedir/export_table/${processing_date}"
target: "${sftp_target}/inbox/_${processing_date}.txt"
merge: true
overwrite: true
Fields#
kind
(mandatory) (type: string):sftpUpload
description
(optional) (type: string): A textual description of the build target.source
(mandatory) (type: string): Specifies the source location in the Hadoop compatible file system. This may be either a single file or a whole directory.target
(mandatory) (type: string): Specifies the target location on the SFTP server. This may be either a single file or a whole directory.connection
(mandatory) (type: string): Specifies the name of the connection to use when connecting to the SFTP server.merge
(optional) (type: boolean) (default: false): Flowman can merge multiple files in thesource
directory into a single output file. This is useful when working with Hadoop or Spark outputs which often consist of multiple files. Per default this is set tofalse
, meaning that multiple files will be transferred individually.delimiter
(optional) (type: string) (default: empty): Specifies an optional delimiter, which should be used between files when they are concatenated whenmerge
is set totrue
. Has no effect otherwise.overwrite
(optional) (type: boolean) (default: true): Set totrue
in order to overwrite existing files on the SFTP server. Otherwise, an existing file will result in an error.
Description#
Supported Execution Phases#
BUILD
- This will upload the specified file via SFTP
Read more about execution phases.