Ingestion config streaming reference

Ingestion-config-based streaming uses IngestionConfigService.CreateIngestionConfig to register the schema of your telemetry, then streams compact structured messages to Sift over gRPC. This approach reduces packet size and improves throughput compared to schemaless methods.

CreateIngestionConfigRequest

message CreateIngestionConfigRequest {
  string asset_name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated FlowConfig flows = 2;
  string organization_id = 3 [(google.api.field_behavior) = OPTIONAL];
  string client_key = 4 [(google.api.field_behavior) = OPTIONAL];
}

Keeping client_key in sync with schema changes: If your flow schemas change often, use a hash of the schema as the client key instead of a static string. The key then changes automatically whenever the schema does, so you don’t have to update it by hand.

Deciding when to change client_key: Keep using the same client_key across Runs until an existing flow needs to change. With the same key, your program reuses the existing ingestion config instead of creating a new one.

Add a flow: Keep the same client_key.
Rename, remove, or modify an existing flow, including its Channels: Use a new client_key. You can’t change existing flows, so you need a new ingestion config. Creating an ingestion config with a client_key that already exists returns an error.

Generating a new client_key for every Run works, but isn’t recommended. Data from each Run still writes to the same Channels on the Asset, as long as the Channel names, data types, units, and enum values stay the same. The tradeoffs:

Slower startup: Each new key creates a new ingestion config, which must register all its flows before streaming starts. This adds startup time and can exceed any timeout your program sets for starting the stream.
Unused configs: The API can’t delete ingestion configs, so you accumulate one for every Run.

FlowConfig

A flow is a named group of channels whose values are sent together in one request.

message FlowConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  repeated ChannelConfig channels = 2;
}

ChannelConfig

message ChannelConfig {
  string name = 1 [(google.api.field_behavior) = REQUIRED];
  string component = 2;
  string unit = 3;
  string description = 4;
  sift.common.type.v1.ChannelDataType data_type = 5 [(google.api.field_behavior) = REQUIRED];
  repeated sift.common.type.v1.ChannelEnumType enum_types = 6;
  repeated sift.common.type.v1.ChannelBitFieldElement bit_field_elements = 7;
}

Choosing between enum-typed and string-typed Channels

If your source data has enum values, such as a Flatbuffer or Protobuf enum, you can send them as an enum-typed Channel or as a plain string-typed Channel. Register the values as an enum if the value set is stable. Use a string Channel if the value set is still changing. Registering an enum has these advantages over a string Channel:

Efficiency: Enum values ingest as a uint32 instead of the full string on every sample.
Visualization: Enum Channels get the Enum Panel in Explore, that includes color-coded state bands and enum names on the y-axis. String Channels use the Table Panel.
Validation: Sift checks each ingested value against the registered enum key set. An unexpected value surfaces as an error instead of landing silently as a string.

Rule expressions don’t distinguish between the two: enum Channels are exposed to CEL as strings, so a condition like $1 == 'FAULT' works the same way for both types.

Enum changes create a new Channel version: A Channel’s identity includes its enum elements. Adding a variant, or inserting one in the middle of the enumeration, creates a new Channel version under the same name. This can cause Rule and Report evaluation errors against the prior version and adds entries to the Channel selector. A string Channel doesn’t have this behavior. Use a string Channel instead of an enum Channel if the value set is still changing.

Channel ordering

The order of Channels in a FlowConfig must be preserved exactly when sending values in an IngestWithConfigDataStreamRequest. Sift attributes each value to a Channel by its position in the list. If data is available for some Channels in a flow but not others, send google.protobuf.Empty in the position of the missing Channel to maintain correct ordering.

Channel structure mismatch: If the ingestion client logs encountered a message that doesn't match any cached flows, the Channel data sent for that flow doesn’t match the flow’s Channel structure as registered in CreateIngestionConfig. Verify that the Channel names, order, and data types you send match the flow’s configured Channels.

Example

Given a flow reading with two Channels, a double Channel followed by a string Channel:

from sift_stream_bindings import (
    ChannelConfigPy,
    ChannelDataTypePy,
    FlowConfigPy,
)

flow_config = FlowConfigPy(
    name="reading",
    channels=[
        ChannelConfigPy(
            name="mainmotor.velocity",
            unit="km/hr",
            description="vehicle speed",
            data_type=ChannelDataTypePy.Double,
            enum_types=[],
            bit_field_elements=[],
        ),
        ChannelConfigPy(
            name="log",
            description="logs",
            data_type=ChannelDataTypePy.String,
            enum_types=[],
            bit_field_elements=[],
        ),
    ],
)

The corresponding send call must list values in the same order:

from sift_stream_bindings import ChannelValuePy, FlowPy, TimeValuePy, ValuePy
from datetime import datetime, timezone

now = datetime.now(timezone.utc)

await ingest_client.send(
    FlowPy(
        flow_name="reading",
        timestamp=TimeValuePy.from_timestamp_millis(int(now.timestamp() * 1000)),
        values=[
            # velocity channel (position 0)
            ChannelValuePy(name="mainmotor.velocity", value=ValuePy.Double(10.0)),
            # log channel (position 1)
            ChannelValuePy(name="log", value=ValuePy.String("example log")),
        ],
    )
)

IngestWithConfigDataStreamRequest

message IngestWithConfigDataStreamRequest {
  string ingestion_config_id = 1;
  string flow = 2;
  google.protobuf.Timestamp timestamp = 3;
  repeated IngestWithConfigDataChannelValue channel_values = 4;
  string run_id = 5;
  bool end_stream_on_validation_error = 6;
  string organization_id = 7;
}

Errors that occur when end_stream_on_validation_error is false appear in Sift’s Data Processing dashboard (https://app.siftstack.com/manage/data-processing).

Duplicate timestamps: If a Channel receives two values at the exact same timestamp, Sift keeps only the most recently written value and discards the earlier one. This deduplication happens at write time, not read time.

Retrieving an ingestion config by client key

curl -G -H "Authorization: Bearer $API_TOKEN" -d "filter=client_key=='example_client_key'" $SIFT_REST_URL/api/v1/ingestion-configs

grpcurl -H "authorization: Bearer $API_TOKEN" -d @ $SIFT_GRPC_URL:$PORT_NUM sift.ingestion_configs.v1.IngestionConfigService/ListIngestionConfigs <<EOM
{
  "filter": "client_key == 'example_client_key'"
}
EOM

Updating an ingestion config

To add flows after creation, send a CreateIngestionConfigFlowsRequest to IngestionConfigService. Each flow name must be unique within the ingestion config; sending a flow with a name that already exists returns an error. You can’t rename, remove, or modify existing flows, including their Channels.

Get started

Ingest telemetry

Investigate

Detect and review

Transform

Export

Sift CLI

Integrate

Manage workspace

Reference

Ingestion config streaming reference

CreateIngestionConfigRequest

FlowConfig

ChannelConfig

Choosing between enum-typed and string-typed Channels

Channel ordering

Example

IngestWithConfigDataStreamRequest

Retrieving an ingestion config by client key

Updating an ingestion config

​CreateIngestionConfigRequest

​FlowConfig

​ChannelConfig

​Choosing between enum-typed and string-typed Channels

​Channel ordering

​Example

​IngestWithConfigDataStreamRequest

​Retrieving an ingestion config by client key

​Updating an ingestion config

​Related topics

CreateIngestionConfigRequest

FlowConfig

ChannelConfig

Choosing between enum-typed and string-typed Channels

Channel ordering

Example

IngestWithConfigDataStreamRequest

Retrieving an ingestion config by client key

Updating an ingestion config

Related topics