Skip to main content
After completing this workflow, you can confirm whether the data you expect is reaching Sift and, if it is not, identify the affected Asset and whether its records failed in the ingestion pipeline.

​
Before you begin

​
How ingestion monitoring works

Sift provides two tools for monitoring ingestion, and each answers a different question: Start with the ingestion health dashboard, then use sift_app to find the cause. For the full workflow, see Verify and troubleshoot ingestion.

​
Verify and troubleshoot ingestion

Use the ingestion health dashboard first to find which Asset is missing data before you look for a cause. Then troubleshoot that Asset in sift_app, where metrics are organized by Asset name. Use the Asset name from the dashboard to find the matching metrics. If you already know which Asset to check, go straight to sift_app.
1

Check overall volume

  1. Click your profile icon.
  2. Select Manage.
  3. In the Manage navigation menu, select Ingest Health.
  4. Select a time range: Day, Week, or Month.
  5. In the Ingest volume chart, look for unexpected drops or spikes.
2

Identify the affected Asset

  1. In Ingest Health, look for Assets that show 0.00 B in the Bytes column. These Assets are not sending data in the selected time range.
  2. In the Asset table, select an Asset that shows 0.00 B or that you suspect caused a drop in the Ingest volume chart.
    • The dashboard filters to that Asset, and the header shows Viewing followed by the Asset name.
  3. Review the Ingest volume chart to see when the Asset’s data dropped.
  4. Note the Asset name. You use it to find the Asset’s metrics in sift_app.
  5. Select Clear to return to all Assets.
3

Check for failed records

  1. Select Assets in the navigation.
  2. In the Asset name search box, enter sift_app.
  3. In the Assets table, select sift_app.
  4. Click Explore.
  5. In the Search Channels box, enter dlq followed by the Asset name you noted in the dashboard. For example, for an Asset named test-stand-01, enter dlq test-stand-01.
    • The results show only that Asset’s dlq_ingestion Channels, which track records sent to the Dead Letter Queue (DLQ) because they failed to ingest.
  6. Plot the count Channel under each reason category:
    • AlreadyExistsErrors,
    • NotFoundErrors,
    • InvalidArgumentErrors, or
    • Other.
    Each count shows how many records failed for that reason.
4

Decide whether ingestion needs intervention

Based on what you found in the dashboard and sift_app, take one of these paths:
  • Analyze a Run: Continue with this path when your Assets show data and no records failed.
  • Recover data that failed to ingest: Continue with this path when records failed for the Asset.
  • Contact support: Continue with this path when the Asset shows 0.00 B and no records failed.
    • If you use sift-stream, check its logs first. If the logs don’t show the cause, or you don’t use sift-stream, select Docs + support in the navigation and submit a ticket.

​
Reference