Skip to main content
Ingestion-config-based streaming uses IngestionConfigService.CreateIngestionConfig to register the schema of your telemetry, then streams compact structured messages to Sift over gRPC. This approach reduces packet size and improves throughput compared to schemaless methods.

CreateIngestionConfigRequest

FlowConfig

A flow is a named group of channels whose values are sent together in one request.

ChannelConfig

Channel ordering

The order of Channels in a FlowConfig must be preserved exactly when sending values in an IngestWithConfigDataStreamRequest. Sift attributes each value to a Channel by its position in the list. If data is available for some Channels in a flow but not others, send google.protobuf.Empty in the position of the missing Channel to maintain correct ordering.
Channel structure mismatch: If the ingestion client logs encountered a message that doesn't match any cached flows, the Channel data sent for that flow doesn’t match the flow’s Channel structure as registered in CreateIngestionConfig. Verify that the Channel names, order, and data types you send match the flow’s configured Channels.

Example

Given a flow reading with two Channels, a double Channel followed by a string Channel:
The corresponding send call must list values in the same order:

IngestWithConfigDataStreamRequest

Errors that occur when end_stream_on_validation_error is false appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing).
Duplicate timestamps: If a Channel receives two values at the exact same timestamp, Sift keeps only the most recently written value and discards the earlier one. This deduplication happens at write time, not read time.

Retrieving an ingestion config by client key

Updating an ingestion config

To add new flows after creation, send a CreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows and channels is backwards compatible. Modifying or removing existing flows or channels is not.