IngestionConfigService.CreateIngestionConfig to register the schema of your telemetry, then streams compact structured messages to Sift over gRPC. This approach reduces packet size and improves throughput compared to schemaless methods.
CreateIngestionConfigRequest
FlowConfig
A flow is a named group of channels whose values are sent together in one request.ChannelConfig
Channel ordering
The order of Channels in aFlowConfig must be preserved exactly when sending values in an IngestWithConfigDataStreamRequest. Sift attributes each value to a Channel by its position in the list.
If data is available for some Channels in a flow but not others, send google.protobuf.Empty in the position of the missing Channel to maintain correct ordering.
Channel structure mismatch: If the ingestion client logs
encountered a message that doesn't match any cached flows, the Channel data sent for that flow doesn’t match the flow’s Channel structure as registered in CreateIngestionConfig. Verify that the Channel names, order, and data types you send match the flow’s configured Channels.Example
Given a flowreading with two Channels, a double Channel followed by a string Channel:
IngestWithConfigDataStreamRequest
end_stream_on_validation_error is false appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing).
Duplicate timestamps: If a Channel receives two values at the exact same timestamp, Sift keeps only the most recently written value and discards the earlier one. This deduplication happens at write time, not read time.
Retrieving an ingestion config by client key
Updating an ingestion config
To add new flows after creation, send aCreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows and channels is backwards compatible. Modifying or removing existing flows or channels is not.