Error Handling

For non-streaming requests, an error returned from an API call indicates that the specific call has failed, allowing the client to handle it appropriately.

In contrast, for streaming calls, like IngestData, the situation is different. Because the client is continuously streaming data, it is not guaranteed that an error is related to the last request sent.

Technically, if an error is returned from a streaming endpoint, it could mean that none of the data points sent in that request were saved. Thus, if the client receives an error from IngestData, the client should retry all the data attempted in that request; Sift will overwrite existing data.

Recommendation: Instead of streaming data continuously, the client should break data into discrete batches. This way, if a failure occurs, only the latest batch needs to be retried.

Note: Implementing this strategy can improve the reliability of data ingestion and reduce the impact of errors on overall data flow.