Query Types

Query types of Logpresso are roughly classified into four types depending on how they are executed.

Ad-hoc Query

An ad-hoc query refers to arbitrary queries that you can create and execute at any given time. You can programmatically execute queries at any given time through the log query menu of Logpresso's web console, the Logpresso terminal accessed via SSH, or the Logpresso client SDK.

If you switch a query that runs for a long time to the background, it will continue to run even when the current session is logged out or disconnected. You can then switch that query back to the foreground to see the result of the query.

Real-time Query

A real-time query refers to a query that processes data received in real time while running for a specified time range from the point of execution. It can use the logs in real time as soon as logs are collected by the log collector, the results of the stream query are output, or the data is provided in a table. This is useful for diagnosis because it immediately samples data without storing all the data on the disk.

Real-time query commands include logger, stream, and table.

Stream Query

A stream query refers to a query that runs indefinitely in the background for real-time data sources until the system shuts down. A stream query runs queries while continuously guaranteeing the input sequence.

Stream queries support the following three stream types as input:

Log Collector

All logs collected by the logger are provided into the stream query. You can configure loggers in Loggers > Loggers.

Table

Every time a collector or write command records a new row to the table, the stream query receives that row as input. It can be thought of as an evolved use case of the triggers used in a relational database management system (RDBMS). You can configure tables in Settings > Tables.

Stream Query

You can use the output of other stream queries as input. Take, for example, a scenario in which a stream query that parses for unstructured logs is placed at the frontend, and multiple analysis stream queries that use that stream query as input are placed. A stream query is divided into streaming mode and refresh mode.

Streaming mode

Processes and outputs each record immediately as it arrives. All commands in the query must be streamable. Streamable commands are those that can handle one record at a time independently — such as filtering (search), field calculation (eval), parsing (rex, parse variants), field selection (fields), and lookup (lookup).

Refresh mode

Sends an input completion signal and re-executes the query at a set interval. Aggregation commands that require all input before they can operate — such as stats, sort, dedup, and top — can be used.

In Sonar, the two modes are exposed as detection scenario types. Real-time detection scenarios run in streaming mode, while batch detection scenarios run in refresh mode with a 60-second interval. If the query you want to use in a scenario includes aggregation commands, you must implement it as a batch detection scenario.

If you use stream queries to calculate statistics for specific time units and to store the result in an intermediate statistics table, and you design this table to query for the final statistical results, you can get statistical results in real time for large data streams while using very little disk space. You can use Groovy scripting for highly complex real-time analysis and processing.

You can use the system-streams command to list registered stream queries and check their running status.

Scheduled Query

A scheduled query runs automatically according to a schedule defined by a cron expression. You can selectively save query results and send results that match alert conditions by email.

You can register and manage scheduled queries in Analysis > Scheduled Queries. For detailed instructions, see the Scheduled Queries documentation.