Protocol Buffers & A Spring Boot Example

• Protocol buffers is a method of serializing data , like XML and JSON.
  
• The format is created by google as its language-neutral, platform-neutral, extensible mechanism for serializing structured data to transmit it over the wire or to store it
  
• Protocol Buffers, which is sometimes referred as Protobuf is not only a message format but also a set of language rules that define the structure of messages. 

• Google developed it with the goal to provide a better way, compared to XML, to make systems communicate.

• There are various online resources and test results that show that Protobuf gives much more speed as compared to JSON and XML.

• Protocol buffers were designed to be faster than JSON & XML by removing many responsibilities performed by these formats and focusing solely on the ability to serialize and deserialize data as fast as possible.
  
• Data transmitted during communication is in the form of binary which improves the speed of transmission compared to JSON’s string format.
  
• Protocol buffers help us to define the structure of the models once and then use generated source code to easily write and read structured data to/from a variety of data streams using a variety of languages.
  
• The structure of the models is defined in a unique .proto file and compiled with protoc command which generates source code that can be invoked by a sender or recipient of these model structures. 
  
• In JSON and XML, the data and context aren’t separate — whereas in Protobuf, it is separate.

{
  studentId: "S101",
  studentName: "Xavier"
}

• When we have to transmit the above object, which contains studentId and studentName we have to provide both of these fields every single time. As our data grows, the transmission time will be increased significantly because of the size.

• To transmit the same using Protobuf, we will have to define the configuration file which will contain the context information

{
string studentId = 1;
string studentName = 2;
}

• By using the above configuration, message will be sent as

124S101226Xavier

• In the above message 1 is Field Identifier. 2 is for data type, in this case string, 4 is for length of the data and next is the field value.
  
• Clearly JSON is highly readable but this will take very little space compared to JSON data

• A protocol buffer compiler converts the .proto file to Java class. It generates a .java file with a class for each message type, as well as a special Builder classes for creating message class instances. The protocol buffer compiler supports other languages also.

• When using Protocol Buffers for sending messages over the network, your payloads are serialized in binary, so they are much smaller than XML or JSON on the wire. This can save you bandwidth and improve network performance.

• There is a slight overhead for serializing and deserializing binary packets , so if you have thousands of tiny messages, you may not save as much in terms of milliseconds compared to REST.However, in most cases, especially long-lasting connections, gRPC will beat out REST most of the time.
  
• Since gRPC uses HTTP/2, it also has support for multiple simultaneous streams multiplexed over the same connection.

• Let us now build a REST api which will use Protobuf for data serialization. The use case is simple, it will have REST endpoint which will return student details.

• First step is to define our message type “student” in student.proto file and place in src/main/proto directory

syntax = "proto3";

package model;

option java_package = "com.raj.nola.protobuf.model";
option java_outer_classname = "StudentProto";

message Student {

  int32 student_id = 1;
  string student_name = 2;

}

• This proto file defines Student message type that has 2 fields , studentId which is integer and studentName which is string. The java file generated from proto buff compiler will be in the package – com.raj.nola.protobuf.model and have an outer class name – StudentProto and Student class.

• To use protocol buffers in Java and provide serialization and de-serialization of different formats based on Google’s protobuf Message. we need to add following Maven dependency .

       <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>${protobuf-java.version}</version>
        </dependency>
        <dependency>
            <groupId>com.googlecode.protobuf-java-format</groupId>
            <artifactId>protobuf-java-format</artifactId>
            <version>${protobuf-java-format.version}</version>
        </dependency>

• To generate java files from .proto file , we need to add the following dependency

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>add-protobuf-generate-sources</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${project.build.directory}/generated-sources</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>

• mvn install will generate source file as defined in .proto file and the generated file will have setters, getters, constructors and builders for our defined messages. It will also have some util methods for saving protobuf files and deserializing them from binary format to Java class.

• Next is a service class which will return a Student object. To create an instance of the Student object we will have to use the newBuilder method of generated class.

public Student getAllStudent(){

    Student student= Student.newBuilder()
            .setStudentId(1)
            .setStudentName("Denis")
            .build();

    return student;
}

• Controller class will expose an endpoint which will invoke the getAllStudent method from service class

@RequestMapping("/students")
Student listAllStudents() {
    return studentSvc.getAllStudent();
}

• We will also define  ProtobufHttpMessageConverter bean which will be used to convert responses returned by @RequestMapping annotated methods to protocol buffer messages.

@Bean
ProtobufHttpMessageConverter protobufHttpMessageConverter() {
    return new ProtobufHttpMessageConverter();
}

• To test this API endpoint let us create the junit test cases which will also illustrate deserialization of protocol buffer messages on the client side 

• Test case to validate the response

@Test
public void getStudentsTest() throws IOException {
    InputStream responseStream = executeHttpRequest(URL);
    String jsonOutput = convertProtobufMessageStreamToJsonString(responseStream);
    System.out.println(jsonOutput);
    assertResponse(jsonOutput);
}

• The test case will execute a GET request and convert the associated response to an InputStream instance using the URL: http://localhost:8080/students

private InputStream executeHttpRequest(String url) throws IOException {
    CloseableHttpClient httpClient = HttpClients.createDefault();
    HttpGet request = new HttpGet(url);
    HttpResponse httpResponse = httpClient.execute(request);
    return httpResponse.getEntity().getContent();
}

• Now, we will convert protocol buffer messages in the form of an InputStream object to a JSON document:

private String convertProtobufMessageStreamToJsonString(InputStream protobufStream) throws IOException {
    JsonFormat jsonFormat = new JsonFormat();
    Account account = Account.parseFrom((protobufStream));
    return jsonFormat.printToString(account);
}

• Check the response

private void assertResponse(String response) {
    assertThat(response, containsString("1"));
    assertThat(response, containsString("Denis"));
}

• Below screen shot also shows the response in JSON and the result of test cases.

• This article gives a short intro into Protocol Buffer and shows how to enable Protocol Buffers for micro services project based on Spring Boot.

Source Code for the example : https://github.com/rajeshsgr/protobuf-svc

FAQs on Protocol Buffers: https://developers.google.com/protocol-buffers/docs/faq

---

Thanks, share your comments and feedback