Stream telemetry from a running application

After completing this topic, you can stream structured telemetry from any application to Sift over gRPC using an ingestion config.

Before you begin

You have a Sift API key and base URL. See Authentication.
You have a gRPC client for your language. See Generate a client with buf.
Your application can call gRPC endpoints.

How ingestion config streaming works

Ingestion-config-based streaming separates schema registration from data transmission. You register the structure of your telemetry once (the Asset, flows, and Channels) using IngestionConfigService.CreateIngestionConfig. After that, you stream compact messages that reference the config by ID, which reduces packet size and improves throughput. A flow is a named group of Channels whose values are sent together in one request. Every request references a specific flow by name, and the Channel values in that request must match the order of channels defined in that flow.

Stream telemetry to Sift

Step 1: Create an ingestion config

Send a CreateIngestionConfigRequest to IngestionConfigService.CreateIngestionConfig. This registers your asset, flows, and channels.

message CreateIngestionConfigRequest {
  string asset_name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated FlowConfig flows = 2;
  string organization_id = 3 [(google.api.field_behavior) = OPTIONAL];
  string client_key = 4 [(google.api.field_behavior) = OPTIONAL];
}

Client keysSet client_key to a unique string you control. This lets you look up the config later without storing the server-assigned ID. Although optional, it is strongly recommended.

The response returns an ingestion_config_id. Save this value; you will include it in every data request.

Step 2: Define your flows and channels

Each FlowConfig names a group of channels that are sent together.

message FlowConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated ChannelConfig channels = 2;
}
message ChannelConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  string component = 2;
  string unit = 3;
  string description = 4;
  sift.common.type.v1.ChannelDataType data_type = 5 [(google.api.field_behavior) = REQUIRED];
  repeated sift.common.type.v1.ChannelEnumType enum_types = 6;
  repeated sift.common.type.v1.ChannelBitFieldElement bit_field_elements = 7;
}

Channel orderingThe order of channels in a FlowConfig must be preserved exactly when you send channel values in a data request. Sift identifies which channel each value belongs to by position. Sending values out of order will cause errors or attribute data to the wrong channel.

If you have data for some channels in a flow but not others, send google.protobuf.Empty in the position of the missing channel to preserve ordering.

Step 3: Create a Run

A Run groups the data you stream into a named, time-bounded capture. Creating a Run is strongly recommended; without a run_id in your ingestion requests, streamed data will not be organized into a Run and will not appear in Sift’s Runs view. To create one, call RunService.CreateRun and save the returned run_id.

Step 4: Stream data

For each data point, construct an IngestWithConfigDataStreamRequest and send it to IngestService.IngestWithConfigDataStream.

message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}

Field	Description
`ingestion_config_id`	The ID returned when you created the ingestion config.
`flow`	The name of the flow this request sends data for.
`timestamp`	The timestamp for all channel values in this request.
`channel_values`	Ordered list of values matching the channel order in the flow definition.
`run_id`	Strongly recommended. Include this in every request to associate data with a Run. Omitting it will cause streamed data to not appear in Sift’s Runs view.
`end_stream_on_validation_error`	Set to `true` only during development. Do not use in production; it severely impacts performance.
`organization_id`	Optional unless your user belongs to multiple organizations.

Validation and performanceWhen end_stream_on_validation_error is false, any server-side errors appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing). Do not set this flag to true in production.

Step 5: Add flows to an existing ingestion config

To add new flows after creation, send a CreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows is backwards compatible. Do not modify or remove existing flows or channels.

Verify the result

After streaming, open Sift and navigate to the asset you specified in asset_name. You should see:

The asset listed under your organization.
If you created a Run, the Run appears with data for the channels you streamed.
Any server-side validation errors appear in the Data Processing dashboard.

Get started

Ingest telemetry

Investigate

Detect and review

Transform

Export

Sift CLI

Integrate

Manage workspace

Reference

Stream telemetry from a running application

Before you begin

How ingestion config streaming works

Stream telemetry to Sift

Step 1: Create an ingestion config

Step 2: Define your flows and channels

Step 3: Create a Run

Step 4: Stream data

Step 5: Add flows to an existing ingestion config

Verify the result

Next steps

Reference

​Before you begin

​How ingestion config streaming works

​Stream telemetry to Sift

​Step 1: Create an ingestion config

​Step 2: Define your flows and channels

​Step 3: Create a Run

​Step 4: Stream data

​Step 5: Add flows to an existing ingestion config

​Verify the result

​Next steps

​Reference

Before you begin

How ingestion config streaming works

Stream telemetry to Sift

Step 1: Create an ingestion config

Step 2: Define your flows and channels

Step 3: Create a Run

Step 4: Stream data

Step 5: Add flows to an existing ingestion config

Verify the result

Next steps

Reference