wget

Receives web resources through HTTP communication, or assigns the received data to the line field and the HTTP code of the server to the _wget_code field.

Syntax

wget [OPTIONS]
Optional Parameters
auth="ID:PASSWD"

Basic HTTP authentication information for HTTP access. For details, refer to the following link: https://datatracker.ietf.org/doc/html/rfc7617

body=FIELD

Fields to be used as the HTML body. Use with method=post or method=put.

encoding=CHARSET

Character set (default: utf-8). Use the preferred MIME name or aliases registered in the following document: http://www.iana.org/assignments/character-sets/character-sets.xhtml

format=FORMAT

Message format from form, json, and xml (default: form). This can be used for REST API call.

header=FIELD_MAP_TYPE

Map type field consisting of a key-value as an HTTP header. It transmits only values that key and value are both string types. You can also specify a key-value map using the dict() function. See usage #2.

method=HTTP_METHOD

HTTP method from get, post, put, delete (default: get).

The post method has the following characteristics.

  • It transmits the key-value of the input record in the form of x-www-form-urlencoded using URL encoding.
  • It cannot be used with the url option, so use the header option to provide the HTTP header directly.
selector="CSS_SELECTOR"

Elements to be selected from the HTML DOM tree. The same syntax is used when defining a selector in CSS.

timeout=INT

HTTP connection timeout time in seconds (default: 30).

url="SITE_URL"

Web service URL to connect to. The command transmits the HTTP request to the specified web address and receives a HTTP response.

Usage

  1. Load the RSS feed title.

    wget url="https://logpresso.com/feed/" selector="item title" 
    | explode elements 
    | eval title = valueof(elements, "own_text") 
    | fields title
    
  2. Access the Logpresso homepage by directly providing the HTTP header.

    json "{}"
    | eval HEADER=
        dict("Accept", "*/*",
             "Accept-Language", "ko-kr",
             "Connection", "keep-alive",
             "Accept-Encoding", "gzip, deflate",
             "User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:64.0) Gecko/20100101 Firefox/64.0"
        ) 
    | wget header=HEADER url="https://ko.logpresso.com"
    
  3. Call FireMon REST API.

    wget auth="firemon:firemon"
         url="https://HOSTNAME/firemon/api/1.0/rules.json?q=rule%7btrue%7d"
    | parsejon 
    | explode devices | parsemap field=devices 
    | explode configSets | parsemap field=configSets 
    | explode policies | parsemap field=policies 
    | explode rules | parsemap field=rules
    | fields uid, log, name, action, comment, disabled