Streaming Data

After creating your ingestion config from the previous section you are ready to start streaming data into Sift over gRPC.

Creating a Data Ingestion Request

Streaming is straightforward. To begin, create an instance of IngestWithConfigDataStreamRequest, which has the following shape:

message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}

ingestion_config_id: The ID of the ingestion config that was created in a previous section
flow: The name of the flow that you wish to send data for
timestamp: The timestamp to use for this flow
channel_values: An order-specific list of channel values that matches the order of channel configs for this particular flow. Read more on ordering
run_id: An optional run ID to associate the data with. Read more on creating and using runs
end_stream_on_validation_error: When true, terminate the stream if an error occurs Sift-side.
organization_id: The ID of your organization. Optional unless your user belongs to multiple organizations.

Impact of Validation on Performance

The end_stream_on_validation_error flag is useful during development to debug and verify correctness of your streaming setup. It should not be used during production as it will severely impact performance. Any errors that occur while this flag is set to false can be found in the "Data Processing Errors" section of the "Manage" page.

Sending Data

The IngestWithConfigDataStreamRequest you created can now be passed to IngestService.IngestWithConfigDataStream.

If the request is processed successfully, you should begin to see data in Sift shortly under the asset associated with the ingestion config.

Creating a Data Ingestion Request

Sending Data

On this page