Ingestion config streaming reference

Ingestion-config-based streaming uses IngestionConfigService.CreateIngestionConfig to register the schema of your telemetry, then streams compact structured messages to Sift over gRPC. This approach reduces packet size and improves throughput compared to schemaless methods.

CreateIngestionConfigRequest

message CreateIngestionConfigRequest {
  string asset_name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated FlowConfig flows = 2;
  string organization_id = 3 [(google.api.field_behavior) = OPTIONAL];
  string client_key = 4 [(google.api.field_behavior) = OPTIONAL];
}

Field	Description
`asset_name`	The name of the asset to create. Required.
`flows`	One or more flow configurations defining the schema. See FlowConfig.
`organization_id`	Your organization ID. Only required if you belong to multiple organizations.
`client_key`	An arbitrary string you choose to uniquely identify this ingestion config. Optional but strongly recommended; it simplifies lookups.

FlowConfig

A flow is a named group of channels whose values are sent together in one request.

message FlowConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated ChannelConfig channels = 2;
}

ChannelConfig

message ChannelConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  string component = 2;
  string unit = 3;
  string description = 4;
  sift.common.type.v1.ChannelDataType data_type = 5 [(google.api.field_behavior) = REQUIRED];
  repeated sift.common.type.v1.ChannelEnumType enum_types = 6;
  repeated sift.common.type.v1.ChannelBitFieldElement bit_field_elements = 7;
}

Field	Description
`name`	The channel name. Required.
`component`	Optional component label.
`unit`	Optional unit string (for example, `km/hr`).
`description`	Optional human-readable description.
`data_type`	The data type of the channel. Required.
`enum_types`	Enum type definitions for enum-typed channels.
`bit_field_elements`	Bit field element definitions for bit-field-typed channels.

Channel ordering

The order of channels in a FlowConfig must be preserved exactly when sending values in an IngestWithConfigDataStreamRequest. Sift attributes each value to a channel by its position in the list. If data is available for some channels in a flow but not others, send google.protobuf.Empty in the position of the missing channel to maintain correct ordering.

Example

Given a flow reading with two channels—a double channel followed by a string channel:

from sift_stream_bindings import (
    ChannelConfigPy,
    ChannelDataTypePy,
    FlowConfigPy,
)

flow_config = FlowConfigPy(
    name="reading",
    channels=[
        ChannelConfigPy(
            name="mainmotor.velocity",
            unit="km/hr",
            description="vehicle speed",
            data_type=ChannelDataTypePy.Double,
            enum_types=[],
            bit_field_elements=[],
        ),
        ChannelConfigPy(
            name="log",
            description="logs",
            data_type=ChannelDataTypePy.String,
            enum_types=[],
            bit_field_elements=[],
        ),
    ],
)

The corresponding send call must list values in the same order:

from sift_stream_bindings import ChannelValuePy, FlowPy, TimeValuePy, ValuePy
from datetime import datetime, timezone

now = datetime.now(timezone.utc)

await ingest_client.send(
    FlowPy(
        flow_name="reading",
        timestamp=TimeValuePy.from_timestamp_millis(int(now.timestamp() * 1000)),
        values=[
            # velocity channel (position 0)
            ChannelValuePy(name="mainmotor.velocity", value=ValuePy.Double(10.0)),
            # log channel (position 1)
            ChannelValuePy(name="log", value=ValuePy.String("example log")),
        ],
    )
)

IngestWithConfigDataStreamRequest

message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}

Field	Description
`ingestion_config_id`	The ID returned when you created the ingestion config.
`flow`	The name of the flow this request sends data for.
`timestamp`	The timestamp for all channel values in this request.
`channel_values`	Ordered list of values. Order must match the channel order in the flow definition.
`run_id`	Optional. Must be included if this data belongs to a Run.
`end_stream_on_validation_error`	When `true`, the stream terminates if a server-side error occurs. Use only during development; this flag severely impacts production performance.
`organization_id`	Optional unless your user belongs to multiple organizations.

Errors that occur when end_stream_on_validation_error is false appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing).

Retrieving an ingestion config by client key

curl -G -H "Authorization: Bearer $API_TOKEN" -d "filter=client_key=='example_client_key'" $SIFT_REST_URL/api/v1/ingestion-configs

grpcurl -H "authorization: Bearer $API_TOKEN" -d @ $SIFT_GRPC_URL:$PORT_NUM sift.ingestion_configs.v1.IngestionConfigService/ListIngestionConfigs <<EOM
{
  "filter": "client_key == 'example_client_key'"
}
EOM

Updating an ingestion config

To add new flows after creation, send a CreateIngestionConfigFlowRequest to IngestionConfigService. Adding flows and channels is backwards compatible. Modifying or removing existing flows or channels is not.

Get started

Ingest telemetry

Investigate

Detect and review

Transform

Export

Sift CLI

Integrate

Manage workspace

Reference

Ingestion config streaming reference

CreateIngestionConfigRequest

FlowConfig

ChannelConfig

Channel ordering

Example

IngestWithConfigDataStreamRequest

Retrieving an ingestion config by client key

Updating an ingestion config

​CreateIngestionConfigRequest

​FlowConfig

​ChannelConfig

​Channel ordering

​Example

​IngestWithConfigDataStreamRequest

​Retrieving an ingestion config by client key

​Updating an ingestion config

​Related topics

CreateIngestionConfigRequest

FlowConfig

ChannelConfig

Channel ordering

Example

IngestWithConfigDataStreamRequest

Retrieving an ingestion config by client key

Updating an ingestion config

Related topics