parallel

Processes input data by running a subquery in parallel across a thread pool. Input records are distributed across multiple threads, the subquery runs concurrently on each thread, and the results from all threads are merged and passed to the next command.

Command properties

Property	Description
Command type	Transforming
Required permission	None
License usage	N/A
Parallel execution	Supported
Distributed execution	Runs on Data Node (mapper)

Syntax

parallel [core=INT] [ SUBQUERY ]

Options

core=INT: Number of threads to use for parallel processing. (Default: number of CPU cores) Cannot exceed the system's CPU core count.

Target

SUBQUERY: Subquery to process input records. Enclose the subquery in square brackets ([ ]). Only stream-processing commands are allowed in the subquery. Commands that require collecting all input before processing — such as sort or aggregation — cannot be used.

Error codes

Parse errors

Error code	Message	Description
22400	parallel core option should be smaller than [max]	The `core` option value exceeds the system's CPU core count.
22401	cannot nest parallel command	A `parallel` command is used inside the subquery of another `parallel` command.
22402	parallel sub query can use only streamable command: [cmd]	The subquery contains a command that does not support stream processing.
90204	sqbracket-mismatch	The square brackets of the subquery are not properly paired.
90206	invalid-subquery	No subquery was specified.

Runtime errors

None

Description

The parallel command divides input records into batches and distributes them across threads in a thread pool. Each thread runs an independent subquery pipeline to process its assigned records, and the results are merged back into a single stream and passed to the next command.

Only stream-processing commands such as eval, rex, parse, and search can be used in the subquery. Commands that require collecting all input — such as sort and stats — cannot be used.

By default, the number of threads equals the system's CPU core count. You can reduce the thread count using the core option, but it cannot exceed the core count. If all threads in the pool are busy, the calling thread processes records directly.

The order of output records is not guaranteed. Records are distributed in batches of at least 50.

Examples

Run a subquery in parallel
```
table duration=1h web_logs
| parallel [ eval line_len = len(line) ]
```
Processes records from the web_logs table in parallel and computes the length of the line field.
Run in parallel with a specified thread count
```
table duration=1h web_logs
| parallel core=4 [ rex field=line "(?<method>GET|POST|PUT|DELETE) (?<path>\S+)" ]
```
Uses 4 threads to extract the method and path fields from the line field using a regular expression.
Run multiple transforming commands in parallel as a subquery
```
table duration=1h web_logs
| parallel [ eval msg = lower(line) | search msg == "*error*" ]
```
Converts the line field to lowercase and filters records containing the string error, running the pipeline in parallel.