Understanding Live Rule Limitations in Sift

To ensure you get the most accurate results from Sift's live rules, it's important to understand their current limitations. This guide outlines key considerations when using live rules for event processing.

Limitations of Live Rules

1. Data Ordering and Timing

   • Out-of-Order Data: Live rules may not handle data that arrives out of chronological order effectively, which can impact the accuracy of rule evaluations.
   • Late Data: Data that arrives late might be excluded from live rule evaluations, leading to incomplete results.

2. Rule Changes During Ingestion

   • If you modify a rule while it's processing live data, there may be a mismatch between the version of the rule that started the annotation and the version that ended it. This can cause inconsistencies in the evaluation results around the time of the change.

3. Throughput Limitations

   • Maximum Data Rate: Live rules support up to 1,000 data points per second (1 kHz) across all channels included in the rule. Exceeding this limit can result in data being dropped without notification.

4. Channel Frequency Disparities

   • Mixed Frequency Channels: If you combine channels that update at different rates (e.g., one updates every minute and another at 5 times per second), the slower channel can delay or stop the evaluation of the faster one.

Recommendations

Perform Batch Rule Evaluations

Batch rule evaluations operate solely on the data available at the time of evaluation, using the event timestamps you provide. This ensures that all relevant data present when you trigger the evaluation is included in the results.

Working With Live Rules

• Monitor Data Rates: Keep the total data rate across all channels at or below 1 kHz to prevent data loss.
• Watch Low Frequency Channels: Live rules with channels with low frequencies may mask out higher rate channels. Be intentional with how you combine low and high frequency channels.
• Use Batch Rules When Appropriate: For scenarios involving high data rates or channels with varying frequencies, consider using batch rules. Batch rules process all available data based on event timestamps when you trigger the evaluation, ensuring comprehensive results.

By being aware of these limitations, you can configure your rules to work more effectively within Sift. If you have questions or need assistance, please contact our support team—we're here to help you optimize your use of live and batch rules.