Protobuf ingestion reference

The protocol buffers required for protobuf ingestion are not yet available in the public repository. To retrieve them, see Protocol Buffers below.

Protobuf ingestion enables sending serialized protobuf messages to Sift. Sift derives telemetry channels from the field paths defined in the message structure. The ingestion request accepts a serialized protobuf message (bytes value), and the server creates channels based on the message structure.

How channels are generated from protobuf fields

Given the following protobuf message:

enum VehicleState {
    Started = 0;
    Stopping = 1;
    Stopped = 2;
}
message Vehicle {
    float velocity = 3;
    repeated float direction = 5;
    VehicleState vehicle_state = 7;
    PropulsionSubsystem propulsion = 12;
    map<string, BatterySystem> batteries = 13;
}
message PropulsionSubsystem {
    float fuel_level
}
message BatterySystem {
    float voltage
    float temp
}

Sift generates the following channels:

Vehicle.velocity
Vehicle.direction[0]
Vehicle.direction[1] (one channel per array index)
Vehicle.vehicle_state
Vehicle.propulsion.fuel_level
Vehicle.batteries[cpu].voltage (one channel per map key)
Vehicle.batteries[cpu].temp
Vehicle.batteries[propulsion].voltage
Vehicle.batteries[propulsion].temp

Schema registration

ProtobufDescriptorService

Use ProtobufDescriptorService.AddProtobufDescriptor to register a protobuf schema before ingesting messages. Registration process

Registration is performed by message type and namespace. The namespace field enables schema separation between environments. For example, a developer can iterate on protobuf definitions in a separate namespace without affecting live telemetry.
The request takes a file_descriptor_set field generated by compiling the protobuf with protoc:

protoc --include_imports --descriptor_set_out=descriptor.output -I /path/to/protofile

Versioning Adding multiple ProtobufDescriptors with the same message type and namespace stores a new version of the descriptor. When a message is ingested, all stored descriptor sets for that message type and namespace are used to generate channel values. The unique set of channels generated is then ingested. Backward compatibility New protobuf descriptors must be backward compatible with existing descriptors that share the same message_type_full_name and namespace. A descriptor is not backward compatible when a field name changes (for example, velocity with field number 1 is renamed to speed). When a new descriptor is not backward compatible, you will receive an error such as:

incompatible protobuf descriptors found. please delete the following protobuf descriptors to
successfully add the new descriptor protobuf descriptor ID: 38cd9974-3352-4d5d-afee-05bae1419074,
message type full name: vehicle, most recent field name: speed, previous field name: velocity,
field number: 1

To resolve this, either make the new descriptor backward compatible by adding a new field and reserving the old one, or delete the old descriptor using the ID in the error message. Use CheckProtobufDescriptorCompatibility to check compatibility before registering. Deletion DeleteProtobufDescriptors removes all descriptors for a given message type and namespace. This is useful for:

Cleaning up test protobuf messages that are no longer in use.
Simplifying ingestion when a new version of the descriptor is fully backward compatible.

Data ingestion

Once the schema is registered, send serialized messages using IngestService.IngestArbitraryProtobufDataStream.

Sift protobuf options

To create more descriptive channels, add custom options from channel_parsing_options.proto to your protobuf definitions.

Option	Description
Units and Description	Add units and descriptions to primitive fields.
Bytes Decoding	Interpret a `bytes`-typed field as another type, such as UTF-8. Enum: `BYTES_DECODING_TYPE_UNSPECIFIED`, `BYTES_DECODING_TYPE_UTF8`.
Tags	Add context from field values to channel names. Useful when a field path needs additional context to be unique.
Map Key Display Overrides	Replace the display value of a map key. Useful when keys are not human-readable or have transient values.
Array Index Overrides	Replace or remove array index display values in channel names.

Tagging example

A tag source identifies a field whose value becomes a tag on related channels.

message TestProto {
    option (sift.protobuf_descriptors.v2.message_is_tag_target) = true;
    string name = 1;
    TestChild primary_child = 2 [(sift.protobuf_descriptors.v2.tag_target).allowed_tag_source=DESCENDANT_AND_SIBLING_SOURCES];
    repeated TestChild array_of_children = 3;
    int32 type_id = 4 [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=ANCESTOR_TARGETS];
}
message TestChild {
    string child_name = 1 [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=ANCESTOR_AND_SIBLING_TARGETS,
    (sift.protobuf_descriptors.v2.tag_source).tag_name="kid_name"];
    map<int32, NestedChild> map_int_to_message = 4;
}
message NestedChild {
    string sub_child_name = 1 [(sift.protobuf_descriptors.v2.tag_target).allowed_tag_source=SIBLING_SOURCES];
    int32 id = 2  [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=SIBLING_TARGETS];
}

Channels without tags (assuming basic map key and array values):

TestProto.name
TestProto.primary_child.child_name
TestProto.primary_child.map_int_to_message[1].sub_child_name
TestProto.primary_child.map_int_to_message[1].id
TestProto.array_of_children[0].child_name
TestProto.array_of_children[0].map_int_to_message[1].sub_child_name
TestProto.array_of_children[0].map_int_to_message[1].id
TestProto.type_id

Channels with tags applied:

TestProto(type_id:3).name
TestProto(type_id:3).primary_child(kid_name:childname).child_name
TestProto(type_id:3).primary_child(kid_name:childname).map_int_to_message[1].sub_child_name(id:35)
TestProto(type_id:3).primary_child(kid_name:childname).map_int_to_message[1].id
TestProto(type_id:3).array_of_children[0].child_name
TestProto(type_id:3).array_of_children[0].map_int_to_message[1].sub_child_name(id:35)
TestProto(type_id:3).array_of_children[0].map_int_to_message[1].id
TestProto(type_id:3).type_id

Multiple tag values display as field_name(field_1:value_1)(field_2:value_2).

Map key and array index override example

enum TestEnum {
  NONE = 0;
  SINGLE = 1;
  DOUBLE = 2;
}

message TestProto {
    string name = 1;

    map<int32, MapKeyTester> map_key_test = 2[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_TARGET];
    map<int32, string> map_key_removal_test = 3[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_REMOVE_KEY];
    map<int32, string> map_key_enum_test = 4[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_ENUM, (sift.protobuf_descriptors.v2.display_override_enum)="TestEnum"];

    repeated ArrayIndexTester array_index_override_test = 5[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_TARGET];
    repeated string array_index_override_remove_index = 6[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_REMOVE_INDEX];
    repeated string array_index_enum_test = 7[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_ENUM, (sift.protobuf_descriptors.v2.display_override_enum)="TestEnum"];
}

message MapKeyTester {
  string new_key = 1[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_SOURCE];
  float some_value = 2;
}

message ArrayIndexTester {
  string new_index = 1[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_SOURCE];
  float other_value = 2;
}

Channels without overrides (assuming 0 is the only map key):

TestProto.name
TestProto.map_key_test[0].new_key
TestProto.map_key_test[0].some_value
TestProto.map_key_removal_test[0]
TestProto.map_key_enum_test[0]
TestProto.array_index_override_test[0].new_index
TestProto.array_index_override_test[0].other_value
TestProto.array_index_override_remove_index[0]
TestProto.array_index_enum_test[0]

Channels with overrides applied (assuming new_key value is my-new-key and new_index value is my-new-index):

TestProto.name
TestProto.map_key_test[my-new-key].new_key
TestProto.map_key_test[my-new-key].some_value
TestProto.map_key_removal_test
TestProto.map_key_enum_test[NONE]
TestProto.array_index_override_test[my-new-index].new_index
TestProto.array_index_override_test[my-new-index].other_value
TestProto.array_index_override_remove_index
TestProto.array_index_enum_test[NONE]

If multiple override sources apply to the same target, the latest one is used and an error is logged.

Protocol buffers

Component	Description	API reference	Source
`ProtobufDescriptorService`	Registers the protobuf schema.	protobuf_descriptors	GitHub
`IngestService`	Streams serialized protobuf messages.	ingest#ingestservice	GitHub
Channel Parsing	Converts protobuf field paths to channels.	channel_parsing_options	GitHub

Get started

Ingest telemetry

Investigate

Detect and review

Transform

Export

Sift CLI

Integrate

Manage workspace

Reference

Protobuf ingestion reference

How channels are generated from protobuf fields

Schema registration

ProtobufDescriptorService

Data ingestion

Sift protobuf options

Tagging example

Map key and array index override example

Protocol buffers

​How channels are generated from protobuf fields

​Schema registration

​ProtobufDescriptorService

​Data ingestion

​Sift protobuf options

​Tagging example

​Map key and array index override example

​Protocol buffers

​Related topics

How channels are generated from protobuf fields

Schema registration

ProtobufDescriptorService

Data ingestion

Sift protobuf options

Tagging example

Map key and array index override example

Protocol buffers

Related topics