Index Operations

The index auto-merge feature is available starting with Logpresso Sonar 4.0.2609.0. The 5 configuration keys described in this document and the logpresso.mergeIndexAll, logpresso.mergeIndexTable, logpresso.cancelMerge, logpresso.markFrozenRange, logpresso.autoMergeStats, logpresso.indexMergeStats, and logpresso.compactIndex shell commands are all available starting with 4.0.2609.0, and do not apply to earlier versions.

Fulltext indexes are stored as .pos/.seg raw file pairs for each daily partition. This structure works well while data keeps flowing in, but once a partition reaches a stable state where it no longer changes, leaving it spread across multiple files still means repeatedly scanning that many files at search time, which reduces search efficiency.

Logpresso Sonar provides an index auto-merge feature that automatically consolidates Hot-tier index partitions that have reached a stable state into a single merged index (a .merged directory). This feature is enabled by default and runs during low-load hours (early morning by default) without any action required. The merged index is automatically included in the search scope of the fulltext command with no additional options, and it follows the same lifecycle as the original data, including redundancy replication, tier rollover, and purge.

Overview

Index auto-merge works in the following order.

Marking: The marking worker inspects Hot-tier partitions on the hour and marks eligible ones for merging. Only partitions that pass all stability guards (storage tier, era, presence of the raw files, whether the table has settled, whether index build has completed, and so on) are marked, and older partitions are inspected first.
Auto-merge: The auto-merge worker consolidates partitions that are marked for merging and not yet merged, in a single-threaded merge pool, according to the configured cron schedule (by default, 4 times daily at 02:30, 03:30, 04:30, and 05:30). On a successful merge, the original .pos/.seg files are deleted.
Standby replication: In a redundant configuration, the completed .merged directory is transferred to the standby node.
Consistency checker and residue sweep: A consistency checker runs daily at 01:00. It compares the merged indexes on both nodes, recovers any missing merged index from the peer node's copy, and cleans up residue left by crashes or race conditions, such as orphaned .merged directories or incomplete directories.
Rollover hold and companion move: A partition scheduled for merging holds its tier rollover for a grace period (7 days by default). Once the grace period elapses, rollover proceeds without merging. When a rollover runs, the merged index moves to the next tier together with the original data.
Purge companion deletion: When you purge (delete) data, the merged index is deleted along with it.

Configuration

Index auto-merge is controlled by the following configuration keys. Use the logpresso.indexGlobals, logpresso.setIndexGlobal, and logpresso.resetIndexGlobal commands in the Logpresso shell to view, change, and reset all keys.

Key	Type	Default	Meaning
`index_auto_merge_enabled`	bool	`true`	Whether the auto-merge feature is enabled
`index_auto_merge_cron`	string	`30 2,3,4,5 * * *`	Execution schedule for the auto-merge cycle (cron expression). By default, it runs daily at 02:30, 03:30, 04:30, and 05:30
`index_freeze_marker_max_merge_mb_per_run`	int	`0` (unlimited)	Upper limit (in MB) on the amount of new merge work that can be added in a single marking cycle. This is calculated from the combined `.pos` + `.seg` file size of each partition. In high-ingestion environments, set this limit so that marking does not get too far ahead of ingestion volume.
`index_rollover_hold_grace_days`	int	`7`	Number of grace days a partition scheduled for merging holds its rollover. Setting this to `0` makes it wait indefinitely until the merge completes. A negative value is forced to the default (7).
`mark_horizon_days`	int	See below	Number of recent days to inspect for marking (the marking window). Hot-tier partitions older than this period are inspected first.

Note

If mark_horizon_days is not set (its initial state), the effective value is not 30 days. Instead, the following order applies: 1) if the default lifecycle profile has a Hot-tier retention period configured, that value is used, and 2) if not configured, 365 days is used. The value of mark_horizon_days is not itself a filter that excludes partitions from merging — it is a window that determines the order in which older partitions are inspected first, and Hot-tier partitions outside the window are still eventually included in the inspection scope. The logpresso.indexGlobals command displays the default value declared in the source for any key that has no stored value, so the query result shows 30 even when mark_horizon_days has not been set. Do not mistake the displayed 30 for the actual effective value.

The point at which a change takes effect differs by configuration key.

index_freeze_marker_max_merge_mb_per_run and mark_horizon_days are re-read on every marking cycle (on the hour), so a change takes effect immediately.
index_rollover_hold_grace_days is re-read whenever an individual partition's rollover-hold status is evaluated, rather than on the marking cycle, so this value also takes effect immediately when changed.
Changing index_auto_merge_enabled from true to false takes effect immediately on the next merge cycle and skips merging. Manual commands such as logpresso.mergeIndexAll and logpresso.mergeIndexTable also check the configuration value at the time they are called, so once you set this to false, these manual commands are also rejected immediately, perform no work, and print only the auto merge worker is disabled message. (This is also why the compactIndex recommendation in the "Batch Index Merging" section below holds.) However, if the value was false when the process booted and you later change it to true, the auto-merge schedule (cron) registration happens only once at boot time, so you must restart the service for the automatic cycle to resume.
Changing index_auto_merge_cron does not affect an already-registered schedule. You must restart the service to apply the new schedule. If you specify an invalid cron expression, the service logs an error at boot time and falls back to the default.

# View all current configuration values
logpresso.indexGlobals

# Set the per-marking-cycle merge work limit to 500MB
logpresso.setIndexGlobal index_freeze_marker_max_merge_mb_per_run 500

# Reset a configuration value (revert to default)
logpresso.resetIndexGlobal index_freeze_marker_max_merge_mb_per_run

Manual Operations

Use the following commands when you need to run or cancel a merge immediately instead of waiting for the auto-merge cycle. Run all of them in the Logpresso shell.

logpresso.mergeIndexAll [start-date] [end-date] [standby=true|false]: Triggers a merge cycle immediately for all tables (runs asynchronously). It follows the same path as the automatic cron cycle. The start and end dates use the yyyy-MM-dd format and can be omitted.
logpresso.mergeIndexTable <table-name> [index-name] [start-date] [end-date] [standby=true|false]: Triggers a merge immediately for the specified table (and index). If you omit the index name, all indexes of that table are targeted.
logpresso.cancelMerge [table-name] [start-date] [end-date]: Cancels auto-merge jobs that are in progress or pending. If you omit the table name, it cancels merge jobs for all tables, and a confirmation prompt (y/n) appears before it runs. Builds run with logpresso.compactIndex cannot be canceled with this command.
logpresso.markFrozenRange <start-date> <end-date> [table-name]: Bypasses the per-marking-cycle work limit (budget) and immediately marks partitions in the specified date range as eligible for merging. If you omit the table name, it applies to all tables.

# Trigger a merge cycle immediately for all tables
logpresso.mergeIndexAll

# Trigger a merge only for the 2026-07 partitions of the SONAR_EVENTS table
logpresso.mergeIndexTable SONAR_EVENTS default_idx 2026-07-01 2026-07-31

# Cancel all merge jobs in progress or pending (includes a confirmation prompt)
logpresso.cancelMerge

# Mark partitions from 2026-07-01 through 2026-07-31 immediately, with no limit
logpresso.markFrozenRange 2026-07-01 2026-07-31

Caution

The standby=true option is an emergency option intended for situations where you must operate the standby node alone for an extended period. Using it during normal redundant operation can produce a merged index built on data that has not yet been fully replicated, and that merged index can remain permanently.

Checking Status

logpresso.autoMergeStats: Prints execution statistics for the auto-merge cycle. You can check the time of the last cycle run, the merge backlog size, the number of partitions currently being merged, and the number of successful and failed merges.
logpresso.indexMergeStats [table-name]: Aggregates, at the partition level, whether partitions are marked for merging (frozen) and their merge status. If you specify a table name, it prints the detailed per-partition status for that table (whether marked for merging, whether merged, and whether the merge is complete).

logpresso.autoMergeStats
logpresso.indexMergeStats SONAR_EVENTS

You can also check the status of merged indexes using Logpresso queries.

Searching the type field of the system-index-disk-usages command for the value merged_fulltext shows the partitions that have a merged index and their disk usage.
Specifying the verbose=t option on the system-indices command shows disk usage in the storage_consumption field, including the size of the merged index.
Checking the hit rate of the index block cache with the system-cache-stats command lets you see whether the reduced segment count from merging is reflected in cache efficiency.

Batch Index Merging

The logpresso.compactIndex command is also available starting with 4.0.2609.0, like the other merge-related shell commands.

logpresso.compactIndex <table-name> <index-name> <start-date (yyyy-MM-dd)> [end-date (yyyy-MM-dd)]: Converts partitions in the specified date range to the merged index format. Partitions that are already merged, or that have no raw files, are skipped with a reason. A partition that is actively being indexed in real time cannot be merged; the command prints an error with a stack trace and moves on to the next partition, and you must retry after that partition becomes idle.

logpresso.compactIndex SONAR_EVENTS default_idx 2026-07-01 2026-07-31

Caution

Unlike mergeIndexAll/mergeIndexTable, compactIndex builds separately and does not go through the same queue (merge pool) as auto-merge. If you run compactIndex in an environment where auto-merge is enabled (the default), auto-merge may contend with it to build the same partition at the same time. A merged index created by compactIndex is not replicated to the standby node, so using it in a redundant configuration can leave the two nodes' index formats continually out of sync. The official manual merge paths are mergeIndexTable and mergeIndexAll; using compactIndex is recommended only in environments where auto-merge is disabled, or for testing purposes.

Index Maintenance

Apart from automatic index merging, two shell commands operate on online index memory. Both have been available since before 4.0.2609.0.

logpresso.flushIndexers <table name pattern>: Writes the online index memory buffer of the specified tables to disk.
logpresso.triggerOnlineIndexGC <table name pattern>: Reclaims memory that the online index of the specified tables no longer uses.

Both commands target tables on the local node only, and the table name pattern accepts wildcards (*). Case sensitivity for literal names without a wildcard differs by series. In the 4.0 series starting with 4.0.2609.0, names are matched case-sensitively. The 5.0 series (through 5.0.2605.2) does not distinguish case.

logpresso.flushIndexers SONAR_EVENTS
logpresso.triggerOnlineIndexGC SONAR_*

Limitations and Cautions

Marking and rollover-hold markers are stored in the index files' extended attributes (xattr). If the index storage path is on a file system that does not support xattr, auto-merge does not work (an error is logged to inform you). Manual merge commands such as logpresso.mergeIndexTable are still available in this case.
Index merging is not recommended in environments that use a WORM (Write Once Read Many) file system as the storage location for the direct copy tier. WORM cannot delete the raw files, so the benefit of merging (reclaiming the raw files) does not apply, and residue left behind during a merge also cannot be deleted.
The standby=true option should not be used during normal redundant operation. See the caution in the Manual Operations section above for details on the risk.