Data processing
Overview
Data processing errors are logged when ingestion fails due to invalid values or parameters. These errors can be reviewed by navigating to the Manage section and selecting Data processing in the Sift interface. This reference page describes how these errors are captured, searched, and exported, the types of errors recorded, and how long the error data is retained.
Common errors
| Error type | Description |
|---|---|
| Invalid numerical values | NaN or Infinity values submitted in the data payload |
| Missing or incorrect config ID | ingestionConfigId values that are absent or improperly specified |
| Invalid flow name | Flow names that are not recognized or formatted incorrectly |
Retention policy
DLQ parquet files are stored in S3 for 46 days and are automatically deleted using a lifecycle policy.
The corresponding database records are retained for 31 days and removed using pg_cron.