Compile Protocol Buffers using Buf

Overview

With Buf, using a simple and declarative configuration, you can compile Protocol Buffers into language-specific source code, including gRPC client stubs. This approach is beneficial when working with Sift’s gRPC API, where the Protocol Buffers define the data structures and the service interfaces needed to generate a client.

Sift provides official gRPC client libraries for Python, Rust, and Go. If your preferred language is not one of these, such as TypeScript, Java, or C#, you can compile Sift’s Protocol Buffers using Buf to generate a compatible client library in that language.

Instructions

To compile Sift's Protocol Buffers using Buf, consider the following steps:

1. Create a new directory for your Python client, and navigate into it. This directory will hold your Buf configuration and the generated client code.

   mkdir sift-grpc-python-client
   cd sift-grpc-python-client

2. Install the Buf CLI globally by following the instructions in the Buf installation instructions.

3. Clone the Sift GitHub repository to access the Protocol Buffer definitions locally. These files define the gRPC services and messages that will be compiled into your Python client.

   git clone https://github.com/sift-stack/sift.git

   The .proto files are located in the protos subdirectory of the repository.

4. In your sift-python-grpc-client directory, create a file named buf.gen.yaml. This file specifies the plugins that Buf should use and where the generated code should be written.

   nano buf.gen.yaml

   version: v1
   managed:
     enabled: true
   plugins:
     - plugin: buf.build/grpc/python:v1.62.1
       out: gen
     - plugin: buf.build/protocolbuffers/python
       out: gen
     - plugin: buf.build/protocolbuffers/pyi:v26.1
       out: gen

   This configuration generates Python message classes, gRPC client and server stubs, and type hint files. All output is written to the gen/ directory by default, but you can modify the out paths to change the output location.

   Many of these plugins may require additional runtime dependencies after the code has been generated to work properly. Visit the documentation for each plugin you use to learn what packages or libraries must be installed.

5. From inside the sift-python-grpc-client directory, run the following command to export the .proto files from the Sift repository into a local directory:

   buf export sift/protos --output=<OUTPUT_DIR> --config <PATH_TO_SIFT_BUF_CONFIG>

   • <OUTPUT_DIR> is the path to the directory where the Protocol Buffer files will be emitted. For example, sift_protos.
   • <PATH_TO_SIFT_BUF_CONFIG> is the path to the buf.yaml file located in the protos subdirectory of the cloned Sift repository (sift).

   buf export sift/protos --output=sift_protos --config sift/protos/buf.yaml

   After running the command, verify that the .proto files were successfully copied into the sift_protos directory.

6. In the buf.gen.yaml file directory, run the following command to generate code from the .proto files you exported:

   buf generate <OUTPUT_DIR>

   Replace <OUTPUT_DIR> with the name of the directory containing Sift's Protocol Buffers. For example, if you used sift_protos, the command would be:

   buf generate sift_protos

   If the command runs successfully, the generated code will be written to the location specified in the out fields of your buf.gen.yaml file, such as the gen/ directory.