Skip to main content
After completing this topic, you can stream structured telemetry from any application to Sift over gRPC using an ingestion config.

​
Before you begin

​
How ingestion config streaming works

Ingestion-config-based streaming separates schema registration from data transmission. You register the structure of your telemetry once (the Asset, flows, and Channels) using IngestionConfigService.CreateIngestionConfig. After that, you stream compact messages that reference the config by ID, which reduces packet size and improves throughput. A flow is a named group of Channels whose values are sent together in one request. Every request references a specific flow by name, and the Channel values in that request must match the order of channels defined in that flow.

​
Stream telemetry to Sift

​
Step 1: Create an ingestion config

Send a CreateIngestionConfigRequest to IngestionConfigService.CreateIngestionConfig. This registers your asset, flows, and channels. 
message CreateIngestionConfigRequest {
  string asset_name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated FlowConfig flows = 2;
  string organization_id = 3 [(google.api.field_behavior) = OPTIONAL];
  string client_key = 4 [(google.api.field_behavior) = OPTIONAL];
}
Client keysSet client_key to a unique string you control. This lets you look up the config later without storing the server-assigned ID. Although optional, it is strongly recommended.
The response returns an ingestion_config_id. Save this value; you will include it in every data request.

​
Step 2: Define your flows and channels

Each FlowConfig names a group of channels that are sent together. 
message FlowConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated ChannelConfig channels = 2;
}
message ChannelConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  string component = 2;
  string unit = 3;
  string description = 4;
  sift.common.type.v1.ChannelDataType data_type = 5 [(google.api.field_behavior) = REQUIRED];
  repeated sift.common.type.v1.ChannelEnumType enum_types = 6;
  repeated sift.common.type.v1.ChannelBitFieldElement bit_field_elements = 7;
}
Channel orderingThe order of channels in a FlowConfig must be preserved exactly when you send channel values in a data request. Sift identifies which channel each value belongs to by position. Sending values out of order will cause errors or attribute data to the wrong channel.
If you have data for some channels in a flow but not others, send google.protobuf.Empty in the position of the missing channel to preserve ordering.

​
Step 3: Create a Run

A Run groups the data you stream into a named, time-bounded capture. Creating a Run is strongly recommended; without a run_id in your ingestion requests, streamed data will not be organized into a Run and will not appear in Sift’s Runs view. To create one, call RunService.CreateRun and save the returned run_id.

​
Step 4: Stream data

For each data point, construct an IngestWithConfigDataStreamRequest and send it to IngestService.IngestWithConfigDataStream. 
message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}
FieldDescription
ingestion_config_idThe ID returned when you created the ingestion config.
flowThe name of the flow this request sends data for.
timestampThe timestamp for all channel values in this request.
channel_valuesOrdered list of values matching the channel order in the flow definition.
run_idStrongly recommended. Include this in every request to associate data with a Run. Omitting it will cause streamed data to not appear in Sift’s Runs view.
end_stream_on_validation_errorSet to true only during development. Do not use in production; it severely impacts performance.
organization_idOptional unless your user belongs to multiple organizations.
Validation and performanceWhen end_stream_on_validation_error is false, any server-side errors appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing). Do not set this flag to true in production.

​
Step 5: Add flows to an existing ingestion config

To add new flows after creation, send a CreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows is backwards compatible. Do not modify or remove existing flows or channels.

​
Verify the result

After streaming, open Sift and navigate to the asset you specified in asset_name. You should see:
  • The asset listed under your organization.
  • If you created a Run, the Run appears with data for the channels you streamed.
  • Any server-side validation errors appear in the Data Processing dashboard.

​
Next steps

​
Reference