wget

Receives web resources through HTTP communication, or assigns the received data to the line field and the HTTP code of the server to the _wget_code field.

Syntax

wget [auth="ID:PASSWD"] [body=FIELD] [encoding=CHARSET] [format={form|json|xml}] [header=FIELD_MAP_TYPE] [method={delete|get|post|put}] [selector="CSS_SELECTOR"] [timeout=NUM] [url="SITE_URL"]
Optional Parameter
auth="ID:PASSWD"

Basic HTTP authentication information for HTTP access. For details, refer to the following link: https://datatracker.ietf.org/doc/html/rfc7617

body=FIELD

Fields to be used as the HTML body. Use with method=post or method=put.

encoding=CHARSET

Character set (default: utf-8). Use the preferred MIME name or aliases registered in the following document: https://www.iana.org/assignments/character-sets/character-sets.xhtml

format=FORMAT

Message format from form, json, and xml (default: form). This can be used for REST API call.

header=FIELD_MAP_TYPE

Map type field consisting of a key-value as an HTTP header. It transmits only values that key and value are both string types. You can also specify a key-value map using the dict() function. See usage #2.

method=HTTP_METHOD

HTTP method from get, post, put, delete (default: get).

The post method has the following characteristics.

  • It transmits the key-value of the input record in the form of x-www-form-urlencoded using URL encoding.
  • It cannot be used with the url option, so use the header option to provide the HTTP header directly.
selector="CSS_SELECTOR"

Elements to be selected from the HTML DOM tree. The same syntax is used when defining a selector in CSS.

timeout=INT

HTTP connection timeout time in seconds (default: 30).

url="SITE_URL"

Web service URL to connect to. The command transmits the HTTP request to the specified web address and receives a HTTP response.

Usage

  1. Load the RSS feed title.

    wget url="https://logpresso.com/feed/" selector="item title" 
    | explode elements 
    | eval title = valueof(elements, "own_text") 
    | fields title
    
  2. Check IP reputations from AbuseIPDB.

     json "{}"
    | eval ip = "118.25.6.39"
    | eval headers = dict("Key", "YOUR_API_KEY", "Accept", "application/json")
    | eval url = concat("https://api.abuseipdb.com/api/v2/check?ipAddress=", ip, "&maxAgeInDays=90")
    | wget method=get header=headers
    | parsejson
    | parsemap field=data
    
  3. Report IP addresses to AbuseIPDB.

     json "{}"
    | eval ip = "47.236.18.74", categories=14, comment = "Port scanning (count: 2790)"
    | eval headers = dict("Key", "YOUR_API_KEY", "Accept", "application/json")
    | eval url = concat("https://api.abuseipdb.com/api/v2/report")
    | wget method=post header=headers