Protobuf and gRPC

4 min readNov 5, 2023

Protobuf (Protocol Buffers)

“Protocol Buffers are language-neutral, platform-neutral extensible mechanisms for serializing structured data.”

To put it simply, Protobufs aka Protocol Buffers are methods of storing and transporting data like XML and JSON, but in an efficient and faster way.

Why is Protobuf useful?

Provides serialisation format for packets for typed/structured data
In-built support for types and validation
Lesser boilerplate code
Faster transmission than counterparts such as XML and JSON as data is transmitted in binary format and takes less space and bandwidth

How does Protobuf work?

Protobufs are invoked on build time to generate code in the target programming languages
Each generated class contains an accessor for each field and methods to serialise and deserialise to and from raw bytes

Defining a Proto

// Specify Syntax (Default "proto2")
syntax = "proto3";

// User Model
message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  optional string address = 4;
}

Syntax: This defines the syntax which is used by proto. proto2 is the default syntax.
Message: The User message specifies fields (name/value pairs), for all the data that needs to be transmitted. Each field has a type, name and field number.

Compiling Proto

For compiling the Proto, we will need to install protoc.

Once this is done, we can use the below command to generate the corresponding models for the User message in Golang.

protoc --go_out=. *.proto

This will generate a user.pb.go which will contain Getters and Setter for the User struct.

Testing Proto in Golang

Install protobuf dependencies in Go

go get github.com/golang/protobuf

proto.Marshal(user) — Marshal methods return the marshalled / wire format encoding of the message

user := &pb.User{
    Id:    1,
    Name:  "John Doe",
    Email: "johndoe@gmail.com",
}
userMarshalled, _ := proto.Marshal(user)
log.Println("Marshalled Proto : ", userMarshalled)

proto.Unmarshal(userMarshalled, &user1) — Unmarshal method decodes the raw message bytes and stores it in the final user1 struct

user1 := pb.User{}
proto.Unmarshal(userMarshalled, &user1)
log.Println("Unmarshalled Proto : ", &user1)

gRPC

gRPC is a Remote Procedure Call (RPC) framework that can run in any environment with support in multiple languages such as Java, Python, Go. gRPC uses Protobuf as its underlying exchange format.

How is gRPC useful?

Remote Procedure call allows applications to call methods on different applications as if they were a local object.

gRPC allows to define Message and Service types using Protobuf and generates code for any supported language

Setting up a gRPC Server and Client

Installing grpc dependencies

go get "google.golang.org/grpc"

Create the user.proto file containing the Message Definition

syntax = "proto3";

option go_package = "/proto";

message User {
  int64 id = 1;
  string name = 2;
  string email = 3;
  optional string address = 4;
}

message UserId {
  int64 id = 1;
}

Now, we need to define the RPC calls in user.proto- Create and Get User.

service UserService {
  rpc Create(User) returns (User){};
  rpc Get(UserId) returns (User){};
}

Compiling the Proto will generate a user.pb.go (containing struct definitions for User and UserId) and user_grpc.pb.go (containing method definitions of the RPC methods)

protoc proto/*.proto --go_out=./ --go-grpc_out=./

Creating a gRPC server.

func main() {
   listener, err := net.Listen("tcp", ":8080")
   if err != nil {
      panic(err)
   }
  
   // Create a gRPC Server
   grpcServer := grpc.NewServer()
   pb.RegisterUserServiceServer(grpcServer, &server{})
   if err := grpcServer.Serve(listener); err != nil {
      log.Fatalf("failed to serve: %v", err)
   }
}

type server struct {
   pb.UserServiceServer
}

Setting up a gRPC Client and Testing

func main() {
   conn, _ := grpc.Dial("localhost:8080", grpc.WithTransportCredentials(insecure.NewCredentials()))
   client := pb.NewUserServiceClient(conn)
  
   created, _ := client.Create(context.Background(), &pb.User{
      Name:    "Sample User",
      Email:   "sample@gmail.com",
      Address: proto.String("21 Symphony Apartment"),
   })

   fmt.Printf("Created User with Id - {%d}\n", created.Id)
  
   user, _ := client.Get(context.Background(), &pb.UserId{Id: created.Id})
   fmt.Printf("Found User - {%s}\n", user)
}

Since there is no implementation for Create and Get methods yet, the server replies with an error.

Implement Create and Get RPC methods on the server

func (s *server) Create(ctx context.Context, user *pb.User) (*pb.User, error) {
   userTable := DB.Table("users")
   userTable.Create(user)
   return user, nil
}

func (s *server) Get(ctx context.Context, userId *pb.UserId) (*pb.User, error) {
   user := pb.User{}
   tx := DB.Table("users").Find(&user, userId.Id)
   if tx.RowsAffected == 0 {
      return nil, errors.New("user not found")
   }
   return &user, nil
}