PROST Well Known Types JSON Serialization and Deserialization

Prost is a Protocol Buffers implementation for the Rust Language that generates simple, idiomatic Rust code from proto2 and proto3 files.

It includes prost-types which gives basic support for protobuf Well-Known-Types (WKT), but support is basic. For example, it does not include packing or unpacking of messages in the Any type, nor much support in the way of JSON serialization and deserialization of that type.

This crate can help you if you need:

• helper methods for packing and unpacking messages to/from an Any,
• helper methods for converting chrono types to Timestamp and back again,
• helper methods for converting common rust types to Value and back again,
• serde support for the types above.

To use it, include this crate along with prost:

[dependencies]
prost = "0.11"
prost-wkt = "0.4"
prost-wkt-types = "0.4"
serde = { version = "1.0", features = ["derive"] }

[build-dependencies]
prost-build = "0.11"
prost-wkt-build = "0.4"

In your build.rs, make sure to add the following options:

use std::{env, path::PathBuf};
use prost_wkt_build::*;

fn main() {
    let out = PathBuf::from(env::var("OUT_DIR").unwrap());
    let descriptor_file = out.join("descriptors.bin");
    let mut prost_build = prost_build::Config::new();
    prost_build
        .type_attribute(
            ".",
            "#[derive(serde::Serialize,serde::Deserialize)]"
        )
        .extern_path(
            ".google.protobuf.Any",
            "::prost_wkt_types::Any"
        )
        .extern_path(
            ".google.protobuf.Timestamp",
            "::prost_wkt_types::Timestamp"
        )
        .extern_path(
            ".google.protobuf.Value",
            "::prost_wkt_types::Value"
        )
        .file_descriptor_set_path(&descriptor_file)
        .compile_protos(
            &[
                "proto/messages.proto"
            ],
            &["proto/"],
        )
        .unwrap();

    let descriptor_bytes =
        std::fs::read(descriptor_file)
        .unwrap();

    let descriptor =
        FileDescriptorSet::decode(&descriptor_bytes[..])
        .unwrap();

    prost_wkt_build::add_serde(out, descriptor);
}

The above configuration will include Serialize, and Deserialize on each generated struct. This will allow you to use serde fully. Moreover, it ensures that the Any type is deserialized properly as JSON. For example, assume we have the following messages defined in our proto file:

syntax = "proto3";

import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";

package my.pkg;

message Request {
    string requestId = 1;
    google.protobuf.Any payload = 2;
}

message Foo {
    string data = 1;
    google.protobuf.Timestamp timestamp = 2;
}

After generating the rust structs for the above using prost-build with the above configuration, you will then be able to do the following:

use serde::{Deserialize, Serialize};
use chrono::prelude::*;

use prost_wkt_types::*;

include!(concat!(env!("OUT_DIR"), "/my.pkg.rs"));

fn main() -> Result<(), AnyError> {
    let foo_msg: Foo = Foo {
        data: "Hello World".to_string(),
        timestamp: Some(Utc::now().into()),
    };

    let mut request: Request = Request::default();
    let any = Any::try_pack(foo_msg)?;
    request.request_id = "test1".to_string();
    request.payload = Some(any);

    let json = serde_json::to_string_pretty(&request).expect("Failed to serialize request");

    println!("JSON:\n{}", json);

    let back: Request = serde_json::from_str(&json).expect("Failed to deserialize request");

    if let Some(payload) = back.payload {
        let unpacked: Box< dyn MessageSerde> = payload.try_unpack()?;

        let unpacked_foo: &Foo = unpacked
            .downcast_ref::<Foo>()
            .expect("Failed to downcast payload to Foo");

        println!("Unpacked: {:?}", unpacked_foo);
    }
}

The above will generate the following stdout:

JSON:
{
  "requestId": "test1",
  "payload": {
    "@type": "type.googleapis.com/my.pkg.Foo",
    "data": "Hello World",
    "timestamp": "2020-05-25T12:19:57.755998Z"
  }
}
Unpacked: Foo { data: "Hello World", timestamp: Some(Timestamp { seconds: 1590409197, nanos: 755998000 }) }

Notice that the request message is properly serialized to JSON as per the protobuf specification, and that it can be deserialized as well.

See the example sub-project for a fully functioning example.

Known Problems

oneOf types

The way prost-build generates the oneOf type is to place it in a sub module, for example:

message SomeOne {
  oneof body {
    string some_string = 1;
    bool some_bool = 2;
    float some_float = 3;
  }
}

is converted to rust as follows:

#[derive(Serialize, Deserialize)]
#[derive(Clone, PartialEq, ::prost::Message)]
#[prost(package="my.pkg")]
pub struct SomeOne {
    #[prost(oneof="some_one::Body", tags="1, 2, 3")]
    pub body: ::core::option::Option<some_one::Body>,
}
/// Nested message and enum types in `SomeOne`.
pub mod some_one {
    #[derive(Serialize, Deserialize)]
    #[derive(Clone, PartialEq, ::prost::Oneof)]
    pub enum Body {
        #[prost(string, tag="1")]
        SomeString(::prost::alloc::string::String),
        #[prost(bool, tag="2")]
        SomeBool(bool),
        #[prost(float, tag="3")]
        SomeFloat(f32),
    }
}

However, rust requires the importation of macros in each module, so each should have the following added:

use serde::{Serialize, Deserialize};

In the generated code snippet, the above statement is missing in the some_one module, and the rust compiler will complain about it. To fix it, we would have to add the appropriate use statement in the some_one module like so:

#[derive(Serialize, Deserialize)]
#[derive(Clone, PartialEq, ::prost::Message)]
#[prost(package="my.pkg")]
pub struct SomeOne {
    #[prost(oneof="some_one::Body", tags="1, 2, 3")]
    pub body: ::core::option::Option<some_one::Body>,
}
/// Nested message and enum types in `SomeOne`.
pub mod some_one {
    use serde::{Serialize, Deserialize};
    #[derive(Serialize, Deserialize)]
    #[derive(Clone, PartialEq, ::prost::Oneof)]
    pub enum Body {
        #[prost(string, tag="1")]
        SomeString(::prost::alloc::string::String),
        #[prost(bool, tag="2")]
        SomeBool(bool),
        #[prost(float, tag="3")]
        SomeFloat(f32),
    }
}

Luckily, you can achieve the above by tweaking the build.rs. The configuration below, for example, will add the required serde import to the some_one module as needed:

fn main() {
    let out = PathBuf::from(env::var("OUT_DIR").unwrap());
    let descriptor_file = out.join("descriptors.bin");
    let mut prost_build = prost_build::Config::new();
    prost_build
        .type_attribute(
            ".my.pkg.MyEnum",
            "#[derive(serde::Serialize,serde::Deserialize)]"
        )
        .type_attribute(
            ".my.pkg.MyMessage",
            "#[derive(serde::Serialize,serde::Deserialize)] #[serde(default)]"
        )
        .type_attribute(
            ".my.pkg.SomeOne.body",
            "#[derive(serde::Serialize,serde::Deserialize)]"
        )
        .extern_path(
            ".google.protobuf.Any",
            "::prost_wkt_types::Any"
        )
        .extern_path(
            ".google.protobuf.Timestamp",
            "::prost_wkt_types::Timestamp"
        )
        .extern_path(
            ".google.protobuf.Value",
            "::prost_wkt_types::Value"
        )
        .file_descriptor_set_path(&descriptor_file)
        .compile_protos(
            &[
                "proto/messages.proto"
            ],
            &["proto/"],
        )
        .unwrap();

    let descriptor_bytes =
        std::fs::read(descriptor_file).unwrap();

    let descriptor =
        FileDescriptorSet::decode(&descriptor_bytes[..]).unwrap();

    prost_wkt_build::add_serde(out, descriptor);
}

Development

Contributions are welcome!

Upgrading Prost

When upgrading Prost to the latest version, make sure to also run wkt-types/resources/update.sh script. This will grab the latest source files from prost-types and merge them into prost-wkt-types at build time. After the script has run, be sure to validate that the selected line numbers in functions process_prost_types_lib and process_prost_types_datetime in the wkt-types/build.rs are still correct!

Please see wkt-types/README.md for more info.

License

prost-wkt is distributed under the terms of the Apache License (Version 2.0).

See LICENSE for details.

Copyright 2023 Ferdinand de Antoni