Skip to main content
Ingestion-config-based streaming uses IngestionConfigService.CreateIngestionConfig to register the schema of your telemetry, then streams compact structured messages to Sift over gRPC. This approach reduces packet size and improves throughput compared to schemaless methods.

​
CreateIngestionConfigRequest

message CreateIngestionConfigRequest {
  string asset_name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated FlowConfig flows = 2;
  string organization_id = 3 [(google.api.field_behavior) = OPTIONAL];
  string client_key = 4 [(google.api.field_behavior) = OPTIONAL];
}
FieldDescription
asset_nameThe name of the asset to create. Required.
flowsOne or more flow configurations defining the schema. See FlowConfig.
organization_idYour organization ID. Only required if you belong to multiple organizations.
client_keyAn arbitrary string you choose to uniquely identify this ingestion config. Optional but strongly recommended; it simplifies lookups.

​
FlowConfig

A flow is a named group of channels whose values are sent together in one request. 
message FlowConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated ChannelConfig channels = 2;
}

​
ChannelConfig

message ChannelConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  string component = 2;
  string unit = 3;
  string description = 4;
  sift.common.type.v1.ChannelDataType data_type = 5 [(google.api.field_behavior) = REQUIRED];
  repeated sift.common.type.v1.ChannelEnumType enum_types = 6;
  repeated sift.common.type.v1.ChannelBitFieldElement bit_field_elements = 7;
}
FieldDescription
nameThe channel name. Required.
componentOptional component label.
unitOptional unit string (for example, km/hr).
descriptionOptional human-readable description.
data_typeThe data type of the channel. Required.
enum_typesEnum type definitions for enum-typed channels.
bit_field_elementsBit field element definitions for bit-field-typed channels.

​
Channel ordering

The order of channels in a FlowConfig must be preserved exactly when sending values in an IngestWithConfigDataStreamRequest. Sift attributes each value to a channel by its position in the list. If data is available for some channels in a flow but not others, send google.protobuf.Empty in the position of the missing channel to maintain correct ordering.

​
Example

Given a flow reading with two channels—a double channel followed by a string channel: 
from sift_stream_bindings import (
    ChannelConfigPy,
    ChannelDataTypePy,
    FlowConfigPy,
)

flow_config = FlowConfigPy(
    name="reading",
    channels=[
        ChannelConfigPy(
            name="mainmotor.velocity",
            unit="km/hr",
            description="vehicle speed",
            data_type=ChannelDataTypePy.Double,
            enum_types=[],
            bit_field_elements=[],
        ),
        ChannelConfigPy(
            name="log",
            description="logs",
            data_type=ChannelDataTypePy.String,
            enum_types=[],
            bit_field_elements=[],
        ),
    ],
)
The corresponding send call must list values in the same order: 
from sift_stream_bindings import ChannelValuePy, FlowPy, TimeValuePy, ValuePy
from datetime import datetime, timezone

now = datetime.now(timezone.utc)

await ingest_client.send(
    FlowPy(
        flow_name="reading",
        timestamp=TimeValuePy.from_timestamp_millis(int(now.timestamp() * 1000)),
        values=[
            # velocity channel (position 0)
            ChannelValuePy(name="mainmotor.velocity", value=ValuePy.Double(10.0)),
            # log channel (position 1)
            ChannelValuePy(name="log", value=ValuePy.String("example log")),
        ],
    )
)

​
IngestWithConfigDataStreamRequest

message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}
FieldDescription
ingestion_config_idThe ID returned when you created the ingestion config.
flowThe name of the flow this request sends data for.
timestampThe timestamp for all channel values in this request.
channel_valuesOrdered list of values. Order must match the channel order in the flow definition.
run_idOptional. Must be included if this data belongs to a Run.
end_stream_on_validation_errorWhen true, the stream terminates if a server-side error occurs. Use only during development; this flag severely impacts production performance.
organization_idOptional unless your user belongs to multiple organizations.
Errors that occur when end_stream_on_validation_error is false appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing).

​
Retrieving an ingestion config by client key

curl -G -H "Authorization: Bearer $API_TOKEN" -d "filter=client_key=='example_client_key'" $SIFT_REST_URL/api/v1/ingestion-configs

​
Updating an ingestion config

To add new flows after creation, send a CreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows and channels is backwards compatible. Modifying or removing existing flows or channels is not.