A faster Protobuf serializer & deserializer.

  • By CloudWeGo
  • Last update: Dec 23, 2022
  • Comments: 9

Fastpb

A faster Protobuf serializer & deserializer.

Attention

  • only proto3 is supported now.
  • known-types(any, api, duration...) is not supported now.

Install

go install github.com/cloudwego/fastpb/[email protected]

Usage

Refer to examples, use in two steps:

  1. use fastpb to generate code. (refer here)
  2. use fastpb API to marshal/unmarshal. (refer here)

Step 1: Generate Code

Using the command line tool to generate code:

protoc --go_out=. \
  --fastpb_out=. \
  ${your_idl}.proto

or

protoc --go_opt=paths=source_relative --go_out=. \
  --fastpb_opt=paths=source_relative --fastpb_out=. \
   ${your_idl}.proto

Step 2: Codec Message

Encoding and Decoding must use fastpb's API, shown as demo:

package examples

import (
	"github.com/cloudwego/fastpb"
)

// Marshal .
func Marshal(msg fastpb.Writer) []byte {
	// TODO: buffer can be reused.
	buf := make([]byte, msg.Size())

	msg.FastWrite(buf)
	return buf
}

// Unmarshal .
func Unmarshal(buf []byte, msg fastpb.Reader) error {
	_, err := fastpb.ReadMessage(buf, int8(fastpb.SkipTypeCheck), msg)
	return err
}

Performance

goos: linux
goarch: amd64
pkg: github.com/cloudwego/fastpb/benchmark
cpu: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz

Benchmarks have compared golang/protobuf (referred to as _golang) and fastpb here.

Marshal

Benchmark_marshal_number_golang-48 375.2 ns/op ~ 96 B/op ~ 1 allocs/op
Benchmark_marshal_number_fastpb-48 145.7 ns/op -61.17% 0 B/op -100.00% 0 allocs/op
Benchmark_marshal_string_golang-48 1010 ns/op ~ 2304 B/op ~ 1 allocs/op
Benchmark_marshal_string_fastpb-48 58.57 ns/op -94.20% 0 B/op -100.00% 0 allocs/op
Benchmark_marshal_list_golang-48 8788 ns/op ~ 18432 B/op ~ 1 allocs/op
Benchmark_marshal_list_fastpb-48 3430 ns/op -60.97% 0 B/op -100.00% 0 allocs/op
Benchmark_marshal_map_golang-48 43497 ns/op ~ 21680 B/op ~ 393 allocs/op
Benchmark_marshal_map_fastpb-48 5951 ns/op -86.32% 0 B/op -100.00% 0 allocs/op

Unmarshal

Benchmark_unmarshal_number_golang-48 497.1 ns/op ~ 144 B/op 1 allocs/op
Benchmark_unmarshal_number_fastpb-48 431.6 ns/op -13.18% 144 B/op 1 allocs/op
Benchmark_unmarshal_string_golang-48 939.7 ns/op ~ 2128 B/op 3 allocs/op
Benchmark_unmarshal_string_fastpb-48 668.4 ns/op -28.87% 2128 B/op 3 allocs/op
Benchmark_unmarshal_list_golang-48 12527 ns/op ~ 20296 B/op 99 allocs/op
Benchmark_unmarshal_list_fastpb-48 12593 ns/op +0.53% 20296 B/op 99 allocs/op
Benchmark_unmarshal_map_golang-48 49868 ns/op ~ 24226 B/op 426 allocs/op
Benchmark_unmarshal_map_fastpb-48 21213 ns/op -57.46% 21467 B/op 61 allocs/op

Download

fastpb.zip

Comments(9)

  • 1

    当idl 有子目录时

    Describe the bug

    user.proto和 pet.proto 子文件夹中(如 idl/user/user.proto, idl/pet/pet.proto)时出现引用，如pet/pet.proto中引用了user/user.proto， 生成的user.pb.go的 中的 protoreflect.FileDescriptor 变量名会是File_user_user_proto，而pet.fast.pb.go中最后一行生成的却是 var _ = user.File_user_proto，从而导致报错不可用

  • 2

    fix: add import guard

    What type of PR is this?

    fix

    What this PR does / why we need it (en: English/zh: Chinese):

    en: add import guard to prevent import not being used zh: 增加 import guard 来防止 import no used 错误

    Which issue(s) this PR fixes:

    none

  • 3

    fix: ReadFieldError not used if struct has no fields

    What type of PR is this?

    fix

    What this PR does / why we need it (en: English/zh: Chinese):

    en: fix: ReadFieldError not used if struct has no fields zh: 修复定义空 struct 会导致生成多余的 ReadFieldError 的问题

    Which issue(s) this PR fixes:

    none

  • 4

    fix: do not generate code if idl contains proto2

    What type of PR is this?

    fix

    What this PR does / why we need it (en: English/zh: Chinese):

    en: fix: do not generate code if idl contains proto2 zh: 如果 idl 包含 proto2，则不生成代码。

    Which issue(s) this PR fixes:

    none

  • 5

    chore: add license header to files

    What type of PR is this?

    chore

    What this PR does / why we need it (en: English/zh: Chinese):

    en: add license header to files zh: 给文件添加许可证头

    Which issue(s) this PR fixes:

    none

  • 6

    custom go_package name, error

    Describe the bug

    When the go_package name is customized, the generated pb.fast.go is wrong

    Steps to reproduce the behavior: custom go_package:

    option go_package = "github.com/cloudwego/fastpb/examples/fastpb_gen/user/player;playercustom";

    although the official documentation does not recommend this, but it is legal.

  • 7

    fix: custom golang's pkg name

    What type of PR is this?

    fix: #14 When the file path does differ from the go_package name, the generated pb.fast.go is not available, although the official documentation does not recommend this, but it is legal. ps:

    option go_package = "github.com/cloudwego/fastpb/examples/fastpb_gen/user/player;playercustom";

    What this PR does / why we need it (en: English/zh: Chinese):

    en: Fix the generated file is not available when the package name and path of custom golang are different zh: 修复自定义golang的包名与路径不同时，生成的文件不可用

  • 8

    导入 google protobuf 报错

    Describe the bug Cannot use '&v' (type *wrappers.StringValue) as the type ReaderType does not implement 'Reader' as some methods are missing:FastRead(buf []byte, _type int8, number int32) (n int, err error)

    11

    Unresolved reference 'File_common_realm_proto'

    2022-11-03 11-46-32屏幕截图

    To Reproduce 执行了下面命令 kitex -module "gitlab.mvalley.com/fgq/kitex_demo" -service getinfo ./proto/info/kd.proto

    Screenshots

    本地文件 proto/common_realm/common.proto 2022-11-03 11-28-43屏幕截图 本地文件 proto_vendor/google/protobuf/wrappers.proto 2022-11-03 11-29-13屏幕截图

    fastpb version: Code generated by FastPB v0.0.1.

  • 9

    Benchmarks

    It's good to see this project is open source right now. Recently, I did some simple benchmarks for comparing Marshal/Unmarshal in different implementations and I also added fastpb. See https://github.com/rleungx/benchmarks/tree/main/protobuf. Please let me know if I was using the right way to do these benchmarks.

Popular Tag

Related