Data Processing Errors

When ingesting data, errors may occur. These errors can be invalid values or invalid ingestion parameters.

The errors will be searchable under Data Processing Errors in these cases. You can search for errors by date range and Asset.

The errors in the search results are summarized by the error message. The individual messages can be downloaded in a CSV format that includes the error and the original ingestion request message.

Error Examples

• NaNs or Infinity values in the data
• Missing or incorrect ingestionConfigIds
• Invalid flow names

Retention Policy

DLQ parquet files are kept in S3 for 46 days (1.5 months) and deleted using a lifecycle policy. The corresponding db rows are kept for only 31 days, and are deleted with pg_cron.