ftp
Allows you to browse the file system on the FTP server and transmit the input records to the file.
Syntax
Required Parameter
PROFILE
-
FTP connect profile. You can configure the profile in the web console.
SUBCOMMAND
-
Command to be executed in the FTP session:
ls
,cat
,put
.ls
- Lists files and directories in the
PATH
on the FTP server. cat
- Reads files in the
PATH
and outputs their contents as records in the fields line by line. The available export file formats are CSV, JSON, TSV and plain text files. put
- Converts the name and values specified by the
fields
option into the file in a format specified by theformat
option, then transmits that file to thePATH
on the FTP server.
PATH
-
Path to a directory or file. If you use a wildcard (
*
) in the file name, you can retrieve all files containing a specific string pattern in the file name (e.g./var/log/httpd/access.*
).
Optional Parameter
The options for each SUBCOMMAND are as follows:
Options | cat | put | ls |
---|---|---|---|
append | - | O | - |
encoding | O | O | O |
fields | - | O | - |
format | O | O | - |
limit | O | - | - |
offset | O | - | - |
overwrite | - | O | - |
append=BOOL
- Option to enable appending data to the end of the file specified in the
PATH
(default:f
).t
: Appends the field records to the end of the file specified byPATH
. If the file does not exist, the file is created. You cannot set this tot
whenoverwrite=t
.f
: NOT append the field records to the end of the file specified byPATH
. The query fails if the file exists.
CautionWhen using the 'append=t' option, always keep the list order of the 'fields' option the same so that data can be consistent.
encoding=CHARSET
- Character set (default: utf-8). Use the preferred MIME name or aliases registered in the following document: http://www.iana.org/assignments/character-sets/character-sets.xhtml
fields=FIELD_LIST
- Fields to be transmitted to the FTP server (default:
line
). Use comma (,
) without any whitespace, as a separator. If there is no line field or the specified field is empty, it is replaced with a hyphen symbol (-
) in the output to indicate the field is empty. format={csv|json|tsv}
- File format (
csv
,json
,tsv
, default: plain text).csv
ortsv
:- When SUBCOMMAND is
cat
, the first line is considered a regular record. Field name (column header) is assigned in the formcolumnN
(N
is a number starting from0
). - , When SUBCOMMAND is
put
, field names (column header) are assigned with the field names specified by thefields
option.
- When SUBCOMMAND is
json
:- When SUBCOMMAND is
cat
, it parses the file into the records of key-value pairs line by line. Field names are specified as keys and field values as values. - When SUBCOMMAND is
put
, it transmits the records consisting of the key-value pairs of the fields specified by thefields
option. If thefields
option is not specified, records consisting of all field values are transmitted.
- When SUBCOMMAND is
- Not specified (plain text):
- When SUBCOMMAND is
cat
, it loades the values to theline
field line by line. - When SUBCOMMAND is
put
, it transmits the file in a text format. Values are separated by tab characters in plain text, and empty values (nulls) are replaced with hyphens (-
).
- When SUBCOMMAND is
limit=INT
- Number of records to be output when importing files from the FTP server (default: unlimited).
offset=INT
- Number of records to skip when importing files from the FTP server (default:
0
). overwrite=BOOL
- Option to enable overwriting the file specified as
PATH
, if it exists (default:f
).t
: Overwrites the file specified asPATH
, if it exists. You cannot set this tot
when theappend
ist
.f
: NOT overwrite the file specified asPATH
, if it exists. The query fails if the file exists.
Usage
You first need to configure an FTP connect profile (PROFILE) and an Apache weblog parser (httpd) to run this example. You can specify the Apache weblog parser using the following options:
Parser Name | Parser Type | Log Format |
---|---|---|
httpd | Apache Web Log | %h %l %u %t "%r" %>s %O "%{Referer}i" "%{User-Agent}i" |
-
Parse the
wp-nginx.log
file and transmit it to the FTP server as a CSV file.To better understand the
format
option, try putting (transmit) the file by not specifying theformat
option or specifying it tojson
ortsv
.wget url="https://raw.githubusercontent.com/logpresso/dataset/main/wp-nginx.log" | eval line = subarray(split(line, "\n"), 0) | explode line | parse httpd | ftp FTP_PROFILE put format=csv overwrite=t fields=remote_host,login,user,date,request,status,sent,referer,user_agent /opt/logpresso/wp.csv
-
List directories or files from the FTP server.
ftp PROFILE ls /opt/logpresso ftp PROFILE ls /opt/logpresso/wp.*
Each query result field has the following meanings:
- type (string):
dir
when it is a directory,file
when it is a file - name (string): Directory or file name
- file_size (integer): File size,
0
when it is a directory - owner (string): Owner
- group (string): Owned group
- modified_at (date): Last modified time
- type (string):
-
Read the first 5 records of the wp.csv file.
ftp PROFILE cat limit=5 /opt/logpresso/wp.csv
-
Read the wp.json file into JSON format
ftp PROFILE cat format=json /opt/logpresso/wp.json