Lifecycle

For an overview of the data lifecycle, refer to here. The lifecycle can be viewed and configured under Settings > Cluster > Lifecycle.

Important Considerations

Before applying the lifecycle to the Logpresso Cluster, please be aware of the following:

Use the Same Storage URI for All Data Nodes

Hot storage utilizes the local directory of each node, so no URI is required. For Warm and Cold storage, which are identified by mount points specified in URI format, all nodes must use the same mount point (URI).

It is Recommended to Use Storage of the Same Capacity for All Data Nodes

Ensure that all nodes storing data utilize local storage of the same capacity. If the storage capacities of the data nodes do not match, unexpected rollovers may occur at unspecified nodes in response to storage exhaustion, leading to persistent discrepancies in storage usage across nodes.

Do Not Specify the Same Shared Path for Warm Storage URIs of Nodes A/B

Nodes A and B, which form a node pair, rollover data from Hot to Warm independently. This means that each node manages its data individually. If both nodes use the same shared path, data integrity issues or loss may occur. Therefore, nodes A and B must be configured to use different shared paths.

Below is an example of setting up mount points and shared paths in the operating system based on NFS (configuration of the NFS server, mounting methods, etc., are omitted).

The Cold tier does not support redundancy, so it is acceptable for nodes A and B to use URIs pointing to the same shared path. In other words, in the Cold tier, having data consolidated in one location does not affect system operation.

Do Not Change Storage URI After Defining the Lifecycle

Do not arbitrarily change the storage path or URI after defining the lifecycle. Changing the path during cluster node operation will result in new data being stored at the modified path, while data that existed at the previous path will not be retrievable using query commands.

Manage Maximum Storage Capacity

Each node has a local storage exhaustion response feature based on local settings. The Local Storage Management feature periodically checks the capacity of the volume containing the data storage directory. If the available space falls below a specified threshold, the system will automatically delete older data or, if necessary, stop data collection to prevent storage overflow.

To prevent the storage usage from reaching the minimum free space of the node, set the lifecycle's Maximum Storage Capacity to a value lower than the threshold allowed by local storage management.

Implementation Strategy

When first introducing the storage lifecycle, continuous rollovers may place a load on the system, so it is advisable to implement it in the following manner:

1. Schedule the rollover during late-night hours to avoid impacting regular security operations, and increase the bandwidth limit during the rollover.
   • The rollover schedule and bandwidth limit can be set in the storage screen.
   • The initial bandwidth limit is 40MB/s.
   • After data has been appropriately rolled over, reduce the bandwidth limit.
2. Set a longer data retention period for the storage tiers and gradually decrease it.
   • Changing the retention period to the target schedule from the outset when a large amount of data is stored can significantly extend the time required for the initial rollover.
   • For example, if there is data stored in the Hot tier for 30 days and the target retention periods are 10 days for Hot and 180 days for Warm, it is advisable to sequentially reduce the Hot tier retention period from 25 days on the first day, to 20 days on the second day, to 15 days on the third day, and to 10 days on the fourth day.

Manage Lifecycle

View Lifecycle

The following illustration shows an example of a cluster with only the Hot tier configured. The defined tiers in the lifecycle display status information, while tiers that are not defined show the message "Hot/Warm/Cold tier is not configured." along with the Set Up button.

• Hot/Warm/Cold Tier Storage URI: URI of the local storage path or storage mount directory
• Retention (days): Duration for which data is retained in storage
• Capacity: Volume of data stored in the storage tier
• Usage per Node: Capacity used by each node pair in the storage tier
  • Node: Name of the node pair
  • Capacity: Total volume of data stored in each node pair
  • Table: Volume of data table
  • Index: Volume of data index
  • Free: Unused capacity

Set Lifecycle

To define the lifecycle for each storage tier:

1. Click Set Up in the property area for the tier you wish to set (Hot, Warm, Cold).

2. In the Hot/Warm/Cold window, enter the lifecycle properties and click OK.

   

   • URI: Mount path or URI of the storage
     • Hot: Local storage that is defined by the DATADIR variable when installing Logpresso Sonar; this is not displayed during configuration.
     • Warm: Enter as a file path (e.g., /opt/logpresso/storage/warm) or in URI format (e.g., file:///opt/logpresso/storage/warm).
     • Cold: Enter as a file path (e.g., /opt/logpresso/storage/cold) or in URI format (e.g., file:///opt/logpresso/storage/cold, s3:///). To use object storage, first configure the Object Storage Profile.
       • To use Cold storage for archiving purposes, select Permanent.
       • Object storage can only be used in the Cold tier.
       • Kakao Cloud Object Storage also uses s3:/// as its URI.
   • Retention Period: Duration for which data will be retained in the storage tier (range: 1–9999).
     • Data that exceeds the retention period will be rolled over to the lower tier.
     • The retention period of the upper tier must be specified before setting the retention period of the lower tier.
   • Storage: Checking this property will prevent the creation of temporary files during rollover, directly copying files to the target storage. Use this when rolling over to WORM or cloud storage.SSS