> ## Documentation Index
> Fetch the complete documentation index at: https://docs.siftstack.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Protobuf ingestion reference

> Schema registration, data ingestion, and channel parsing options for protobuf-based streaming

<Info>
  The protocol buffers required for protobuf ingestion are not yet available in the public repository. To retrieve them, see [Protocol Buffers](#protocol-buffers) below.
</Info>

Protobuf ingestion enables sending serialized protobuf messages to Sift. Sift derives telemetry channels from the field paths defined in the message structure. The ingestion request accepts a serialized protobuf message (`bytes value`), and the server creates channels based on the message structure.

## How channels are generated from protobuf fields

Given the following protobuf message:

```protobuf theme={null}
enum VehicleState {
    Started = 0;
    Stopping = 1;
    Stopped = 2;
}
message Vehicle {
    float velocity = 3;
    repeated float direction = 5;
    VehicleState vehicle_state = 7;
    PropulsionSubsystem propulsion = 12;
    map<string, BatterySystem> batteries = 13;
}
message PropulsionSubsystem {
    float fuel_level
}
message BatterySystem {
    float voltage
    float temp
}
```

Sift generates the following channels:

* `Vehicle.velocity`
* `Vehicle.direction[0]`
* `Vehicle.direction[1]` (one channel per array index)
* `Vehicle.vehicle_state`
* `Vehicle.propulsion.fuel_level`
* `Vehicle.batteries[cpu].voltage` (one channel per map key)
* `Vehicle.batteries[cpu].temp`
* `Vehicle.batteries[propulsion].voltage`
* `Vehicle.batteries[propulsion].temp`

## Schema registration

### ProtobufDescriptorService

Use `ProtobufDescriptorService.AddProtobufDescriptor` to register a protobuf schema before ingesting messages.

**Registration process**

* Registration is performed by message type and `namespace`. The `namespace` field enables schema separation between environments. For example, a developer can iterate on protobuf definitions in a separate namespace without affecting live telemetry.
* The request takes a `file_descriptor_set` field generated by compiling the protobuf with `protoc`:

```bash theme={null}
protoc --include_imports --descriptor_set_out=descriptor.output -I /path/to/protofile
```

**Versioning**

Adding multiple `ProtobufDescriptors` with the same message type and `namespace` stores a new version of the descriptor. When a message is ingested, all stored descriptor sets for that message type and namespace are used to generate channel values. The unique set of channels generated is then ingested.

**Backward compatibility**

New protobuf descriptors must be backward compatible with existing descriptors that share the same `message_type_full_name` and `namespace`. A descriptor is not backward compatible when a field name changes (for example, `velocity` with field number `1` is renamed to `speed`).

When a new descriptor is not backward compatible, you will receive an error such as:

```
incompatible protobuf descriptors found. please delete the following protobuf descriptors to
successfully add the new descriptor protobuf descriptor ID: 38cd9974-3352-4d5d-afee-05bae1419074,
message type full name: vehicle, most recent field name: speed, previous field name: velocity,
field number: 1
```

To resolve this, either make the new descriptor backward compatible by adding a new field and reserving the old one, or delete the old descriptor using the ID in the error message.

Use `CheckProtobufDescriptorCompatibility` to check compatibility before registering.

**Deletion**

`DeleteProtobufDescriptors` removes all descriptors for a given message type and namespace. This is useful for:

* Cleaning up test protobuf messages that are no longer in use.
* Simplifying ingestion when a new version of the descriptor is fully backward compatible.

## Data ingestion

Once the schema is registered, send serialized messages using `IngestService.IngestArbitraryProtobufDataStream`.

## Sift protobuf options

To create more descriptive channels, add custom options from `channel_parsing_options.proto` to your protobuf definitions.

| Option                    | Description                                                                                                                          |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Units and Description     | Add units and descriptions to primitive fields.                                                                                      |
| Bytes Decoding            | Interpret a `bytes`-typed field as another type, such as UTF-8. Enum: `BYTES_DECODING_TYPE_UNSPECIFIED`, `BYTES_DECODING_TYPE_UTF8`. |
| Tags                      | Add context from field values to channel names. Useful when a field path needs additional context to be unique.                      |
| Map Key Display Overrides | Replace the display value of a map key. Useful when keys are not human-readable or have transient values.                            |
| Array Index Overrides     | Replace or remove array index display values in channel names.                                                                       |

### Tagging example

A tag source identifies a field whose value becomes a tag on related channels.

```protobuf theme={null}
message TestProto {
    option (sift.protobuf_descriptors.v2.message_is_tag_target) = true;
    string name = 1;
    TestChild primary_child = 2 [(sift.protobuf_descriptors.v2.tag_target).allowed_tag_source=DESCENDANT_AND_SIBLING_SOURCES];
    repeated TestChild array_of_children = 3;
    int32 type_id = 4 [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=ANCESTOR_TARGETS];
}
message TestChild {
    string child_name = 1 [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=ANCESTOR_AND_SIBLING_TARGETS,
    (sift.protobuf_descriptors.v2.tag_source).tag_name="kid_name"];
    map<int32, NestedChild> map_int_to_message = 4;
}
message NestedChild {
    string sub_child_name = 1 [(sift.protobuf_descriptors.v2.tag_target).allowed_tag_source=SIBLING_SOURCES];
    int32 id = 2  [(sift.protobuf_descriptors.v2.tag_source).allowed_tag_target=SIBLING_TARGETS];
}
```

Channels without tags (assuming basic map key and array values):

* `TestProto.name`
* `TestProto.primary_child.child_name`
* `TestProto.primary_child.map_int_to_message[1].sub_child_name`
* `TestProto.primary_child.map_int_to_message[1].id`
* `TestProto.array_of_children[0].child_name`
* `TestProto.array_of_children[0].map_int_to_message[1].sub_child_name`
* `TestProto.array_of_children[0].map_int_to_message[1].id`
* `TestProto.type_id`

Channels with tags applied:

* `TestProto(type_id:3).name`
* `TestProto(type_id:3).primary_child(kid_name:childname).child_name`
* `TestProto(type_id:3).primary_child(kid_name:childname).map_int_to_message[1].sub_child_name(id:35)`
* `TestProto(type_id:3).primary_child(kid_name:childname).map_int_to_message[1].id`
* `TestProto(type_id:3).array_of_children[0].child_name`
* `TestProto(type_id:3).array_of_children[0].map_int_to_message[1].sub_child_name(id:35)`
* `TestProto(type_id:3).array_of_children[0].map_int_to_message[1].id`
* `TestProto(type_id:3).type_id`

Multiple tag values display as `field_name(field_1:value_1)(field_2:value_2)`.

### Map key and array index override example

```protobuf theme={null}
enum TestEnum {
  NONE = 0;
  SINGLE = 1;
  DOUBLE = 2;
}

message TestProto {
    string name = 1;

    map<int32, MapKeyTester> map_key_test = 2[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_TARGET];
    map<int32, string> map_key_removal_test = 3[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_REMOVE_KEY];
    map<int32, string> map_key_enum_test = 4[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_ENUM, (sift.protobuf_descriptors.v2.display_override_enum)="TestEnum"];

    repeated ArrayIndexTester array_index_override_test = 5[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_TARGET];
    repeated string array_index_override_remove_index = 6[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_REMOVE_INDEX];
    repeated string array_index_enum_test = 7[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_ENUM, (sift.protobuf_descriptors.v2.display_override_enum)="TestEnum"];
}

message MapKeyTester {
  string new_key = 1[(sift.protobuf_descriptors.v2.map_key_override_type)=MAP_KEY_OVERRIDE_SOURCE];
  float some_value = 2;
}

message ArrayIndexTester {
  string new_index = 1[(sift.protobuf_descriptors.v2.array_index_override_type)=ARRAY_INDEX_OVERRIDE_SOURCE];
  float other_value = 2;
}
```

Channels without overrides (assuming `0` is the only map key):

* `TestProto.name`
* `TestProto.map_key_test[0].new_key`
* `TestProto.map_key_test[0].some_value`
* `TestProto.map_key_removal_test[0]`
* `TestProto.map_key_enum_test[0]`
* `TestProto.array_index_override_test[0].new_index`
* `TestProto.array_index_override_test[0].other_value`
* `TestProto.array_index_override_remove_index[0]`
* `TestProto.array_index_enum_test[0]`

Channels with overrides applied (assuming `new_key` value is `my-new-key` and `new_index` value is `my-new-index`):

* `TestProto.name`
* `TestProto.map_key_test[my-new-key].new_key`
* `TestProto.map_key_test[my-new-key].some_value`
* `TestProto.map_key_removal_test`
* `TestProto.map_key_enum_test[NONE]`
* `TestProto.array_index_override_test[my-new-index].new_index`
* `TestProto.array_index_override_test[my-new-index].other_value`
* `TestProto.array_index_override_remove_index`
* `TestProto.array_index_enum_test[NONE]`

If multiple override sources apply to the same target, the latest one is used and an error is logged.

## Protocol buffers

| Component                   | Description                                | API reference                                                                        | Source                                                                                                                   |
| --------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
| `ProtobufDescriptorService` | Registers the protobuf schema.             | [protobuf\_descriptors](/api/reference/protocol-buffers/protobuf_descriptors)        | [GitHub](https://github.com/sift-stack/sift/blob/main/protos/sift/protobuf_descriptors/v2/protobuf_descriptors.proto)    |
| `IngestService`             | Streams serialized protobuf messages.      | [ingest#ingestservice](/api/reference/protocol-buffers/ingest#ingestservice)         | [GitHub](https://github.com/sift-stack/sift/blob/main/protos/sift/ingest/v1/ingest.proto)                                |
| Channel Parsing             | Converts protobuf field paths to channels. | [channel\_parsing\_options](/api/reference/protocol-buffers/channel_parsing_options) | [GitHub](https://github.com/sift-stack/sift/blob/main/protos/sift/protobuf_descriptors/v2/channel_parsing_options.proto) |

## Related topics

* [Choose a streaming method](/documentation/ingest/stream/choose-a-streaming-method)
* [Stream live telemetry](/documentation/ingest/stream/index)