ftp

Allows you to browse the file system on the FTP server and transmit the input records to the file.

Syntax

ftp PROFILE SUBCOMMAND [OPTIONS] PATH
Required Parameter
PROFILE

FTP connect profile. You can configure the profile in the web console.

SUBCOMMAND

Command to be executed in the FTP session: ls, cat, put.

ls
Lists files and directories in the PATH on the FTP server.
cat
Reads files in the PATH and outputs their contents as records in the fields line by line. The available export file formats are CSV, JSON, TSV and plain text files.
put
Converts the name and values specified by the fields option into the file in a format specified by the format option, then transmits that file to the PATH on the FTP server.
PATH

Path to a directory or file. If you use a wildcard (*) in the file name, you can retrieve all files containing a specific string pattern in the file name (e.g. /var/log/httpd/access.*).

Optional Parameter

The options for each SUBCOMMAND are as follows:

Optionscatputls
append-O-
encodingOOO
fields-O-
formatOO-
limitO--
offsetO--
overwrite-O-
append=BOOL
Option to enable appending data to the end of the file specified in the PATH (default: f).
  • t: Appends the field records to the end of the file specified by PATH. If the file does not exist, the file is created. You cannot set this to t when overwrite=t.
  • f: NOT append the field records to the end of the file specified by PATH. The query fails if the file exists.
Caution
When using the 'append=t' option, always keep the list order of the 'fields' option the same so that data can be consistent.
encoding=CHARSET
Character set (default: utf-8). Use the preferred MIME name or aliases registered in the following document: http://www.iana.org/assignments/character-sets/character-sets.xhtml
fields=FIELD_LIST
Fields to be transmitted to the FTP server (default: line). Use comma (,) without any whitespace, as a separator. If there is no line field or the specified field is empty, it is replaced with a hyphen symbol (-) in the output to indicate the field is empty.
format={csv|json|tsv}
File format (csv, json, tsv, default: plain text).
  • csv or tsv:
    • When SUBCOMMAND is cat, the first line is considered a regular record. Field name (column header) is assigned in the form columnN (N is a number starting from 0).
    • , When SUBCOMMAND is put, field names (column header) are assigned with the field names specified by the fields option.
  • json:
    • When SUBCOMMAND is cat, it parses the file into the records of key-value pairs line by line. Field names are specified as keys and field values as values.
    • When SUBCOMMAND is put, it transmits the records consisting of the key-value pairs of the fields specified by the fields option. If the fields option is not specified, records consisting of all field values are transmitted.
  • Not specified (plain text):
    • When SUBCOMMAND is cat, it loades the values to the line field line by line.
    • When SUBCOMMAND is put, it transmits the file in a text format. Values are separated by tab characters in plain text, and empty values (nulls) are replaced with hyphens (-).
limit=INT
Number of records to be output when importing files from the FTP server (default: unlimited).
offset=INT
Number of records to skip when importing files from the FTP server (default: 0).
overwrite=BOOL
Option to enable overwriting the file specified as PATH, if it exists (default:f).
  • t: Overwrites the file specified as PATH, if it exists. You cannot set this to t when the append is t.
  • f: NOT overwrite the file specified as PATH, if it exists. The query fails if the file exists.

Usage

You first need to configure an FTP connect profile (PROFILE) and an Apache weblog parser (httpd) to run this example. You can specify the Apache weblog parser using the following options:

Parser NameParser TypeLog Format
httpdApache Web Log%h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i"
  1. Parse the wp-nginx.log file and transmit it to the FTP server as a CSV file.

    To better understand the format option, try putting (transmit) the file by not specifying the format option or specifying it to json or tsv.

    wget url="https://raw.githubusercontent.com/logpresso/dataset/main/wp-nginx.log"
    | eval line = subarray(split(line, "\n"), 0)
    | explode line
    | parse httpd
    | ftp FTP_PROFILE put format=csv overwrite=t
      fields=remote_host,login,user,date,request,status,sent,referer,user_agent
      /opt/logpresso/wp.csv
    
  2. List directories or files from the FTP server.

    ftp PROFILE ls /opt/logpresso
    ftp PROFILE ls /opt/logpresso/wp.*
    

    Each query result field has the following meanings:

    • type (string): dir when it is a directory, file when it is a file
    • name (string): Directory or file name
    • file_size (integer): File size, 0 when it is a directory
    • owner (string): Owner
    • group (string): Owned group
    • modified_at (date): Last modified time
  3. Read the first 5 records of the wp.csv file.

    ftp PROFILE cat limit=5 /opt/logpresso/wp.csv
    
  4. Read the wp.json file into JSON format

    ftp PROFILE cat format=json /opt/logpresso/wp.json