henry/whisper-com
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ADD: added new version of protobuf

Browse Source
This commit is contained in:
Henry Winkel
2022-12-20 10:09:28 +01:00
parent 4a79559129
commit 1e2b3dda7b
1513 changed files with 123720 additions and 83381 deletions
Download Patch File Download Diff File Expand all files Collapse all files

135
libs/protobuf/docs/cmake_protobuf_generate.md
View File

@@ -1,135 +0,0 @@
# How to use `protobuf_generate`
This document explains how to use the function `protobuf_generate` which is provided by protobuf's CMake module.
## Usage
In the same directory that called `find_package(protobuf CONFIG)` and any of its subdirectories, the CMake function `protobuf_generate` is made available by
[`protobuf-generate.cmake`](../cmake/protobuf-generate.cmake). It can be used to automatically generate source files from `.proto` schema files at build time.
### Basic example
Let us see how `protobuf_generate` can be used to generate and compile the source files of a proto schema whenever an object target called `proto-objects` is built.
Given the following directory structure:
- `proto/helloworld/helloworld.proto`
- `CMakeLists.txt`
where `helloworld.proto` is a protobuf schema file and `CMakeLists.txt` contains:
```cmake
find_package(protobuf CONFIG REQUIRED)
add_library(proto-objects OBJECT "${CMAKE_CURRENT_LIST_DIR}/proto/helloworld/helloworld.proto")
target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf)
set(PROTO_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")
target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:${PROTO_BINARY_DIR}>")
protobuf_generate(
TARGET proto-objects
IMPORT_DIRS "${CMAKE_CURRENT_LIST_DIR}/proto"
PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")
```
Building the target `proto-objects` will generate the files:
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.h`
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.cc`
and (depending on the build system) output:
```shell
[build] [1/2] Running cpp protocol buffer compiler on /proto/helloworld/helloworld.proto
[build] [2/2] Building CXX object /build/generated/helloworld/helloworld.pb.cc.o
```
### gRPC example
`protobuf_generate` can also be customized to invoke plugins like gRPC's `grpc_cpp_plugin`. Given the same directory structure as in the [basic example](#basic-example)
and let `CMakeLists.txt` contain:
```cmake
find_package(gRPC CONFIG REQUIRED)
add_library(proto-objects OBJECT "${CMAKE_CURRENT_LIST_DIR}/proto/helloworld/helloworld.proto")
target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf gRPC::grpc++)
set(PROTO_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")
set(PROTO_IMPORT_DIRS "${CMAKE_CURRENT_LIST_DIR}/proto")
target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:${PROTO_BINARY_DIR}>")
protobuf_generate(
TARGET proto-objects
IMPORT_DIRS ${PROTO_IMPORT_DIRS}
PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")
protobuf_generate(
TARGET proto-objects
LANGUAGE grpc
GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
PLUGIN "protoc-gen-grpc=\$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
IMPORT_DIRS ${PROTO_IMPORT_DIRS}
PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")
```
Then building `proto-objects` will generate and compile:
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.h`
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.cc`
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.grpc.pb.h`
- `${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.grpc.pb.cc`
And `protoc` will automatically be re-run whenever the schema files change and `proto-objects` is built.
### Note on unity builds
Since protobuf's generated source files are unsuited for [jumbo/unity builds](https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html) it is recommended
to exclude them from such builds which can be achieved by adjusting their properties:
```cmake
protobuf_generate(
OUT_VAR PROTO_GENERATED_FILES
...)
set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on)
```
## How it works
For each source file ending in `proto` of the argument provided to `TARGET` or each file provided through `PROTOS`, `protobuf_generate` will set up
a [add_custom_command](https://cmake.org/cmake/help/latest/command/add_custom_command.html) which depends on `protobuf::protoc` and the proto files.
It declares the generated source files as `OUTPUT` which means that any target that depends on them will automatically cause the custom command to execute
when it is brought up to date. The command itself is made up of the arguments for `protoc`, like the output directory, the schema files, the language to
generate for, the plugins to use, etc.
## Reference
Arguments accepted by `protobuf_generate`.
Flag arguments:
- `APPEND_PATH` — A flag that causes the base path of all proto schema files to be added to `IMPORT_DIRS`.
Single-value arguments:
- `LANGUAGE` — A single value: cpp or python. Determines what kind of source files are being generated.
- `OUT_VAR` — Name of a CMake variable that will be filled with the paths to the generated source files.
- `EXPORT_MACRO` — Name of a macro that is applied to all generated Protobuf message classes and extern variables. It can, for example, be used to declare DLL exports.
- `PROTOC_OUT_DIR` — Output directory of generated source files. Defaults to `CMAKE_CURRENT_BINARY_DIR`.
- `PLUGIN` — An optional plugin executable. This could, for example, be the path to `grpc_cpp_plugin`.
- `PLUGIN_OPTIONS` — Additional options provided to the plugin, such as `generate_mock_code=true` for the gRPC cpp plugin.
- `DEPENDENCIES` — Arguments forwarded to the `DEPENDS` of the underlying `add_custom_command` invocation.
- `TARGET` — CMake target that will have the generated files added as sources.
Multi-value arguments:
- `PROTOS` — List of proto schema files. If omitted, then every source file ending in *proto* of `TARGET` will be used.
- `IMPORT_DIRS` — A common parent directory for the schema files. For example, if the schema file is `proto/helloworld/helloworld.proto` and the import directory `proto/` then the generated files are `${PROTOC_OUT_DIR}/helloworld/helloworld.pb.h` and `${PROTOC_OUT_DIR}/helloworld/helloworld.pb.cc`.
- `GENERATE_EXTENSIONS` — If LANGUAGE is omitted then this must be set to the extensions that protoc generates.
- `PROTOC_OPTIONS` — Additional arguments that are forwarded to protoc.

338
libs/protobuf/docs/cpp_build_systems.md
View File

@@ -1,338 +0,0 @@
# How Protobuf supports multiple C++ build systems
This document explains how the Protobuf project supports multiple C++ build
systems.
## Background
Protobuf primarily uses [Bazel](https://bazel.build) to build the Protobuf C++
runtime and Protobuf compiler[^historical_sot]. However, there are several
different build systems in common use for C++, each one of which requires
essentially a complete copy of the same build definitions.
[^historical_sot]:
On a historical note, prior to its [release as Open Source
Software](https://opensource.googleblog.com/2008/07/protocol-buffers-googles-data.html),
the Protobuf project was developed using Google's internal build system, which
was the predecessor to Bazel (the vast majority of Google's contributions
continue to be developed this way). The Open Source Protobuf project, however,
historically used Autoconf to build the C++ implementation.
Over time, other build systems (including Bazel) have been added, thanks in
large part to substantial contributions from the Open Source community. Since
the Protobuf project deals with multiple languages (all of which ultimately
rely upon C++, for the Protobuf compiler), Bazel is a natural choice for a
project-wide build system -- in fact, Bazel (and its predecessor, Blaze)
was designed in large part to support exactly this type of rich,
multi-language build.
Currently, C++ Protobuf can be built with Bazel and CMake. Each of these build
systems has different semantics and structure, but share in common the list of
files needed to build the runtime and compiler.
## Design
### Extracting information from Bazel
Bazel's Starlark API provides [aspects](https://bazel.build/rules/aspects) to
traverse the build graph, inspect build rules, define additional actions, and
expose information through
[providers](https://bazel.build/rules/rules#providers). For example, the
`cc_proto_library` rule uses an aspect to traverse the dependency graph of
`proto_library` rules, and dynamically attaches actions to generate C++ code
using the Protobuf compiler and compile using the C++ compiler.
In order to support multiple build systems, the overall build structure is
defined once for each system, and expose frequently-changing metadata
from Bazel in a way that can be included from the build definition. Primarily,
this means exposing the list of source files in a way that can be included
in other build definitions.
Two aspects are used to extract this information from the Bazel build
definitions:
* `cc_file_list_aspect` extracts `srcs`, `hdrs`, and `textual_hdrs` from build
rules like `cc_library`. The sources are exposed through a provider named
`CcFileList`.
* `proto_file_list_aspect` extracts the `srcs` from a `proto_library`, and
also generates the expected filenames that would be generated by the
Protobuf compiler. This information is exposed through a provider named
`ProtoFileList`.
On their own, these aspects have limited utility. However, they can be
instantiated by custom rules, so that an ordinary `BUILD.bazel` target can
produce outputs based on the information gleaned from these aspects.
### (Aside) Distribution libraries
Bazel's native `cc_library` rule is typically used on a "fine-grained" level, so
that, for example, lightweight unit tests can be written with narrow scope.
Although Bazel does build library artifacts (such as `.so` and `.a` files on
Linux), they correspond to `cc_library` rules.
Since the entire "Protobuf library" includes many constituent `cc_library`
rules, a special rule, `cc_dist_library`, combines several fine-grained
libraries into a single, monolithic library.
For the Protobuf project, these "distribution libraries" are intended to match
the granularity of the CMake-based builds. Since the Bazel-built
distribution library covers the rules with the source files needed by other
builds, the `cc_dist_library` rule invokes the `cc_file_list_aspect` on its
input libraries. The result is that a `cc_dist_library` rule not only produces
composite library artifacts, but also collect and provide the list of sources
that were inputs.
For example:
```
$ cat cc_dist_library_example/BUILD.bazel
load("@rules_cc//cc:defs.bzl", "cc_library")
load("//pkg:cc_dist_library.bzl", "cc_dist_library")
cc_library(
name = "a",
srcs = ["a.cc"],
)
cc_library(
name = "b",
srcs = ["b.cc"],
deps = [":c"],
)
# N.B.: not part of the cc_dist_library, even though it is in the deps of 'b':
cc_library(
name = "c",
srcs = ["c.cc"],
)
cc_dist_library(
name = "lib",
deps = [
":a",
":b",
],
visibility = ["//visibility:public"],
)
# Note: the output below has been formatted for clarity:
$ bazel cquery //cc_dist_library_example:lib \
--output=starlark \
--starlark:expr='providers(target)["//pkg:cc_dist_library.bzl%CcFileList"]'
struct(
hdrs = depset([]),
internal_hdrs = depset([]),
srcs = depset([
<source file cc_dist_library_example/a.cc>,
<source file cc_dist_library_example/b.cc>,
]),
textual_hdrs = depset([]),
)
```
The upshot is that the "coarse-grained" library can be defined by the Bazel
build, and then export the list of source files that are needed to reproduce the
library in a different build system.
One major difference from most Bazel rule types is that the file list aspects do
not propagate. In other words, they only expose the immediate dependency's
sources, not transitive sources. This is for two reasons:
1. Immediate dependencies are conceptually simple, while transitivity requires
substantially more thought. For example, if transitive dependencies were
considered, then some way would be needed to exclude dependencies that
should not be part of the final library (for example, a distribution library
for `//:protobuf` could be defined not to include all of
`//:protobuf_lite`). While dependency elision is an interesting design
problem, the protobuf library is small enough that directly listing
dependencies should not be problematic.
2. Dealing only with immediate dependencies gives finer-grained control over
what goes into the composite library. For example, a Starlark `select()`
could conditionally add fine-grained libraries to some builds, but not
others.
Another subtlety for tests is due to Bazel internals. Internally, a slightly
different configuration is used when evaluating `cc_test` rules as compared to
`cc_dist_library`. If `cc_test` targets are included in a `cc_dist_library`
rule, and both are evaluated by Bazel, this can result in a build-time error:
the config used for the test contains additional options that tell Bazel how to
execute the test that the `cc_file_list_aspect` build config does not. Bazel
detects this as two conflicting actions generating the same outputs. (For
`cc_test` rules, the simplest workaround is to provide sources through a
`filegroup` or similar.)
### File list generation
Lists of input files are generated by Bazel in a format that can be imported to
other build systems. Currently only CMake-style files can be generated.
The lists of files are derived from Bazel build targets. The sources can be:
* `cc_dist_library` rules (as described above)
* `proto_library` rules
* individual files
* `filegroup` rules
* `pkg_files` or `pkg_filegroup` rules from
https://github.com/bazelbuild/rules_pkg
For example:
```
$ cat gen_file_lists_example/BUILD.bazel
load("@rules_proto//proto:defs.bzl", "proto_library")
load("//pkg:build_systems.bzl", "gen_cmake_file_lists")
filegroup(
name = "doc_files",
srcs = [
"README.md",
"englilsh_paper.md",
],
)
proto_library(
name = "message",
srcs = ["message.proto"],
)
gen_cmake_file_lists(
name = "source_lists",
out = "source_lists.cmake",
src_libs = {
":doc_files": "docs",
":message": "buff",
"//cc_dist_library_example:c": "distlib",
},
)
$ bazel build gen_file_lists_example:source_lists
$ cat bazel-bin/gen_file_lists_example/source_lists.cmake
# Auto-generated by //gen_file_lists_example:source_lists
#
# This file contains lists of sources based on Bazel rules. It should
# be included from a hand-written CMake file that defines targets.
#
# Changes to this file will be overwritten based on Bazel definitions.
if(${CMAKE_VERSION} VERSION_GREATER 3.10 OR ${CMAKE_VERSION} VERSION_EQUAL 3.10)
include_guard()
endif()
# //gen_file_lists_example:doc_files
set(docs_files
gen_file_lists_example/README.md
gen_file_lists_example/englilsh_paper.md
)
# //gen_file_lists_example:message
set(buff_proto_srcs
gen_file_lists_example/message.proto
)
# //gen_file_lists_example:message
set(buff_srcs
gen_file_lists_example/message.proto.pb.cc
)
# //gen_file_lists_example:message
set(buff_hdrs
gen_file_lists_example/message.proto.pb.h
)
# //gen_file_lists_example:message
set(buff_files
gen_file_lists_example/message-descriptor-set.proto.bin
)
# //cc_dist_library_example:c
set(distlib_srcs
cc_dist_library_example/a.cc
cc_dist_library_example/b.cc
)
# //cc_dist_library_example:c
set(distlib_hdrs
)
```
A hand-written CMake build rule could then use the generated file to define
libraries, such as:
```
include(source_lists.cmake)
add_library(distlib ${distlib_srcs} ${buff_srcs})
```
### Protobuf usage
The main C++ runtimes (lite and full) and the Protobuf compiler use their
corresponding `cc_dist_library` rules to generate file lists. For
`proto_library` targets, the file list generation can extract the source files
directly. For other targets, notably `cc_test` targets, the file list generators
use `filegroup` rules.
In general, adding new targets to a non-Bazel build system in Protobuf (or
adding a new build system altogether) requires some one-time setup:
1. The overall structure of the new build system has to be defined. It should
import lists of files and refer to them by variable, instead of listing
files directly.
2. (Only if the build system is new) A new rule type has to be added to
`//pkg:build_systems.bzl`. Most of the implementation is shared, but a
"fragment generator" is need to declare a file list variable, and the rule
type itself has to be defined and call the shared implementation.
When files are added or deleted, or when the Protobuf Bazel structure is
changed, these changes may need to be reflected in the file list logic. These
are some example scenarios:
* Files are added to (or removed from) the `srcs` of an existing `cc_library`:
no changes needed. If the `cc_library` is already part of a
`cc_dist_library`, then regenerating the source lists will reflect the
change.
* A `cc_library` is added: the new target may need to be added to the Protobuf
`cc_dist_library` targets, as appropriate.
* A `cc_library` is deleted: if a `cc_dist_library` depends upon the deleted
target, then a build-time error will result. The library needs to be removed
from the `cc_dist_library`.
* A `cc_test` is added or deleted: test sources are handled by `filegroup`
rules defined in the same package as the `cc_test` rule. The `filegroup`s
are usually given a name like `"test_srcs"`, and often use `glob()` to find
sources. This means that adding or removing a test may not require any extra
work, but this can be verified within the same package as the test rule.
* Test-only proto files are added: the `proto_library` might need to be added
to the file list map in `//pkg:BUILD.bazel`, and then the file added to
various build systems. However, most test-only protos are already exposed
through libraries like `//src/google/protobuf:test_protos`.
If there are changes, then the regenerated file lists need to be copied back
into the repo. That way, the corresponding build systems can be used with a git
checkout, without needing to run Bazel first.
### (Aside) Distribution archives
A very similar set of rules is defined in `//pkg` to build source distribution
archives for releases. In addition to the full sources, Protobuf releases also
include source archives sliced by language, so that, for example, a Ruby-based
project can get just the sources needed to build the Ruby runtime. (The
per-language slices also include sources needed to build the protobuf compiler,
so they all effectively include the C++ runtime.)
These archives are defined using rules from the
[rules_pkg](https://github.com/bazelbuild/rules_pkg) project. Although they are
similar to `cc_dist_library` and the file list generation rules, the goals are
different: the build system file lists described above only apply to C++, and
are organized according to what should or should not be included in different
parts of the build (e.g., no tests are included in the main library). On the
other hand, the distribution archives deal with languages other than C++, and
contain all the files that need to be distributed as part of a release (even for
C++, this is more than just the C++ sources).
While it might be possible to use information from the `CcFileList` and
`ProtoFileList` providers to define the distribution files, additional files
(such as the various `BUILD.bazel` files) are also needed in the distribution
archive. The lists of distribution files can usually be generated by `glob()`,
anyhow, so sharing logic with the file list aspects may not be beneficial.
Currently, all of the file lists are checked in. However, it would be possible
to build the file lists on-the-fly and include them in the distribution
archives, rather than checking them in.

8
libs/protobuf/docs/implementing_proto3_presence.md
View File

@@ -131,11 +131,11 @@ These options are demonstrated below:
```
# One option:
$ protoc test.proto --cpp_out=. --experimental_allow_proto3_optional
$ ./src/protoc test.proto --cpp_out=. --experimental_allow_proto3_optional
# Another option:
$ cp test.proto test_proto3_optional.proto
$ protoc test_proto3_optional.proto --cpp_out=.
$ ./src/protoc test_proto3_optional.proto --cpp_out=.
$
```
@@ -152,7 +152,7 @@ If you now try to invoke your own code generator with the test proto, you will
run into a different error:
```
$ protoc test_proto3_optional.proto --my_codegen_out=.
$ ./src/protoc test_proto3_optional.proto --my_codegen_out=.
test_proto3_optional.proto: is a proto3 file that contains optional fields, but
code generator --my_codegen_out hasn't been updated to support optional fields in
proto3. Please ask the owner of this code generator to support proto3 optional.
@@ -201,7 +201,7 @@ Once you have added this, you should now be able to successfully use your code
generator to generate a file containing proto3 optional fields:
```
$ protoc test_proto3_optional.proto --my_codegen_out=.
$ ./src/protoc test_proto3_optional.proto --my_codegen_out=.
```
### Updating Your Code Generator

18
libs/protobuf/docs/jvm_aot.md
View File

@@ -1,8 +1,8 @@
# Ahead Of Time (AOT) compilation for the Java Virtual Machine (JVM)
# Ahead Of Time (AOT) compilation for the Java Virtual Machine (JVM)"
Ahead Of Time (AOT) compilation build tools such as those provided by [GraalVM's `native-image`](https://www.graalvm.org/reference-manual/native-image/) can require some configuration when using protobuf.
Protobuf for the JVM uses reflection, and some of its target classes are not possible to determine in advance.
This situation is unlikely to change.
Protobuf for the JVM uses reflection and some of its target classes are not possible to determine in advance.
Historically, there were good reasons to use reflection based on APIs that were published effectively requiring them, and this situation is unlikely to change.
[The Lite version of protobuf for the JVM](https://github.com/protocolbuffers/protobuf/blob/main/java/lite.md)
avoids reflection and may be better suited for use with AOT compilation tooling. This Lite version was originally targeted for use on Android which has similar AOT compilation
@@ -10,11 +10,9 @@ goals as GraalVM's native-image tool.
## GraalVM native-image
The GraalVM `native-image` tool can be configured with options for the
[Java Native Interface](https://www.graalvm.org/22.0/reference-manual/native-image/JNI/) (JNI),
http proxying, reflection, and other resources. While these
configuration options can be manually specified in JSON files, we recommend
that you exercise the application with
This section addresses GraalVM's `native-image` configuration specifically as this AOT compilation tool due to its popularity. The `native-image` tool can be configured
with respect to: the [Java Native Interface](https://en.wikipedia.org/wiki/Java_Native_Interface) (JNI), http proxying, reflection, and other resources. While these
considerations can be manually declared as JSON files, we recommend that a JVM application is exercised along with
[the assisted configuration agent](https://www.graalvm.org/reference-manual/native-image/BuildConfiguration/#assisted-configuration-of-native-image-builds). The agent
will generate native-image.properties files that you add to the project's
META-INF/native-image/ directory. The Native Image builder uses configuration options provided in META-INF/native-image/ to construct native-image command line arguments.
will generate files that you can then subsequently point at when invoking `native-image`. We recommend that the generated files are retained with a project's source
code.

28
libs/protobuf/docs/options.md
View File

@@ -320,31 +320,3 @@ with info about your project (name and website) so we can add an entry for you.
1. Protoc-gen-authz
* Website: https://github.com/Neakxs/protoc-gen-authz
* Extension: 1145
1. Protonium
* Website: https://github.com/zyp/protonium
* Extension: 1146
1. Protoc-gen-xo
* Website: https://github.com/xo/ecosystem
* Extension: 1147
1. Ballerina gRPC
* Website: https://github.com/ballerina-platform/module-ballerina-grpc
* Extension: 1148
1. Protoc-gen-referential-integrity
* Website: https://github.com/ComponentCorp/protoc-gen-referential-integrity
* Extension: 1149
1. Oclea Service Layer RPC
* Website: https://oclea.com/
* Extension: 1150
1. mypy-protobuf
* Website: https://github.com/nipunn1313/mypy-protobuf
* Extension: 1151-1154
1. Pigweed protobuf compiler
* Website: https://pigweed.dev/pw_protobuf
* Extension: 1155

304
libs/protobuf/docs/performance.md Normal file
View File

@@ -0,0 +1,304 @@
# Protobuf Performance
The following benchmark test results were produced on a workstation utilizing an Intel® Xeon® Processor E5-2630 with 32GB of RAM.
This table contains the results of three separate languages:
* **C++** - For C++, there are three parsing methods:
* **new** - This is for using a new operator for creating a message instance.
* **new arena** - This is for using arena for creating a new message instance.
* **reuse** - This is for reusing the same message instance for parsing.
* **Java** - For Java, there are three parsing/serialization methods:
* **byte[]** - This is for parsing from a Byte Array.
* **ByteString** - This is for parsing from a
com.google.protobuf.ByteString.
* **InputStream** - This is for parsing from an InputStream.
* **Python** - For Python, there are three types of Python protobuf for testing:
* **C++-generated-code** - This is for using C++ generated code of the
proto file as a dynamic linked library.
* **C++-reflection** - This is for using C++ reflection, for which there's no
generated code, but still using C++ protobuf library as a dynamic linked
library.
* **pure-Python** - This is for the pure version of Python, which does not link with
any C++ protobuf library.
## Parsing performance
<table>
<tbody><tr>
<th rowspan="2"> </th>
<th colspan="3" rowspan="1">C++</th>
<th colspan="3" rowspan="1">C++ with tcmalloc</th>
<th colspan="3" rowspan="1">java</th>
<th colspan="3" rowspan="1">python</th>
</tr>
<tr>
<th colspan="1">new</th>
<th colspan="1">new arena</th>
<th colspan="1">reuse</th>
<th colspan="1">new</th>
<th colspan="1">new arena</th>
<th colspan="1">reuse</th>
<th colspan="1">byte[]</th>
<th colspan="1">ByteString</th>
<th colspan="1">InputStream</th>
<th colspan="1">C++-generated-code</th>
<th colspan="1">C++-reflection</th>
<th colspan="1">pure-Python</th>
</tr>
<tr>
<td>google_message1_proto2</td>
<td>368.717MB/s</td>
<td>261.847MB/s</td>
<td>799.403MB/s</td>
<td>645.183MB/s</td>
<td>441.023MB/s</td>
<td>1.122GB/s</td>
<td>425.437MB/s</td>
<td>425.937MB/s</td>
<td>251.018MB/s</td>
<td>82.8314MB/s</td>
<td>47.6763MB/s</td>
<td>3.76299MB/s</td>
</tr>
<tr>
<td>google_message1_proto3</td>
<td>294.517MB/s</td>
<td>229.116MB/s</td>
<td>469.982MB/s</td>
<td>434.510MB/s</td>
<td>394.701MB/s</td>
<td>591.931MB/s</td>
<td>357.597MB/s</td>
<td>378.568MB/s</td>
<td>221.676MB/s</td>
<td>82.0498MB/s</td>
<td>39.9467MB/s</td>
<td>3.77751MB/s</td>
</tr>
<tr>
<td>google_message2</td>
<td>277.242MB/s</td>
<td>347.611MB/s</td>
<td>793.67MB/s</td>
<td>503.721MB/s</td>
<td>596.333MB/s</td>
<td>922.533MB/s</td>
<td>416.778MB/s</td>
<td>419.543MB/s</td>
<td>367.145MB/s</td>
<td>241.46MB/s</td>
<td>71.5723MB/s</td>
<td>2.73538MB/s</td>
</tr>
<tr>
<td>google_message3_1</td>
<td>213.478MB/s</td>
<td>291.58MB/s</td>
<td>543.398MB/s</td>
<td>539.704MB/s</td>
<td>717.300MB/s</td>
<td>927.333MB/s</td>
<td>684.241MB/s</td>
<td>704.47MB/s</td>
<td>648.624MB/s</td>
<td>209.036MB/s</td>
<td>142.356MB/s</td>
<td>15.3324MB/s</td>
</tr>
<tr>
<td>google_message3_2</td>
<td>672.685MB/s</td>
<td>802.767MB/s</td>
<td>1.21505GB/s</td>
<td>985.790MB/s</td>
<td>1.136GB/s</td>
<td>1.367GB/s</td>
<td>1.54439GB/s</td>
<td>1.60603GB/s</td>
<td>1.33443GB/s</td>
<td>573.835MB/s</td>
<td>314.33MB/s</td>
<td>15.0169MB/s</td>
</tr>
<tr>
<td>google_message3_3</td>
<td>207.681MB/s</td>
<td>140.591MB/s</td>
<td>535.181MB/s</td>
<td>369.743MB/s</td>
<td>262.301MB/s</td>
<td>556.644MB/s</td>
<td>279.385MB/s</td>
<td>304.853MB/s</td>
<td>107.575MB/s</td>
<td>32.248MB/s</td>
<td>26.1431MB/s</td>
<td>2.63541MB/s</td>
</tr>
<tr>
<td>google_message3_4</td>
<td>7.96091GB/s</td>
<td>7.10024GB/s</td>
<td>9.3013GB/s</td>
<td>8.518GB/s</td>
<td>8.171GB/s</td>
<td>9.917GB/s</td>
<td>5.78006GB/s</td>
<td>5.85198GB/s</td>
<td>4.62609GB/s</td>
<td>2.49631GB/s</td>
<td>2.35442GB/s</td>
<td>802.061MB/s</td>
</tr>
<tr>
<td>google_message3_5</td>
<td>76.0072MB/s</td>
<td>51.6769MB/s</td>
<td>237.856MB/s</td>
<td>178.495MB/s</td>
<td>111.751MB/s</td>
<td>329.569MB/s</td>
<td>121.038MB/s</td>
<td>132.866MB/s</td>
<td>36.9197MB/s</td>
<td>10.3962MB/s</td>
<td>8.84659MB/s</td>
<td>1.25203MB/s</td>
</tr>
<tr>
<td>google_message4</td>
<td>331.46MB/s</td>
<td>404.862MB/s</td>
<td>427.99MB/s</td>
<td>589.887MB/s</td>
<td>720.367MB/s</td>
<td>705.373MB/s</td>
<td>606.228MB/s</td>
<td>589.13MB/s</td>
<td>530.692MB/s</td>
<td>305.543MB/s</td>
<td>174.834MB/s</td>
<td>7.86485MB/s</td>
</tr>
</tbody></table>
## Serialization performance
<table>
<tbody><tr>
<th rowspan="2"> </th>
<th colspan="1" rowspan="2">C++</th>
<th colspan="1" rowspan="2">C++ with tcmalloc</th>
<th colspan="3" rowspan="1">java</th>
<th colspan="3" rowspan="1">python</th>
</tr>
<tr>
<th colspan="1">byte[]</th>
<th colspan="1">ByteString</th>
<th colspan="1">InputStream</th>
<th colspan="1">C++-generated-code</th>
<th colspan="1">C++-reflection</th>
<th colspan="1">pure-Python</th>
</tr>
<tr>
<td>google_message1_proto2</td>
<td>1.39698GB/s</td>
<td>1.701GB/s</td>
<td>1.12915GB/s</td>
<td>1.13589GB/s</td>
<td>758.609MB/s</td>
<td>260.911MB/s</td>
<td>58.4815MB/s</td>
<td>5.77824MB/s</td>
</tr>
<tr>
<td>google_message1_proto3</td>
<td>959.305MB/s</td>
<td>939.404MB/s</td>
<td>1.15372GB/s</td>
<td>1.07824GB/s</td>
<td>802.337MB/s</td>
<td>239.4MB/s</td>
<td>33.6336MB/s</td>
<td>5.80524MB/s</td>
</tr>
<tr>
<td>google_message2</td>
<td>1.27429GB/s</td>
<td>1.402GB/s</td>
<td>1.01039GB/s</td>
<td>1022.99MB/s</td>
<td>798.736MB/s</td>
<td>996.755MB/s</td>
<td>57.9601MB/s</td>
<td>4.09246MB/s</td>
</tr>
<tr>
<td>google_message3_1</td>
<td>1.31916GB/s</td>
<td>2.049GB/s</td>
<td>991.496MB/s</td>
<td>860.332MB/s</td>
<td>662.88MB/s</td>
<td>1.48625GB/s</td>
<td>421.287MB/s</td>
<td>18.002MB/s</td>
</tr>
<tr>
<td>google_message3_2</td>
<td>2.15676GB/s</td>
<td>2.632GB/s</td>
<td>2.14736GB/s</td>
<td>2.08136GB/s</td>
<td>1.55997GB/s</td>
<td>2.39597GB/s</td>
<td>326.777MB/s</td>
<td>16.0527MB/s</td>
</tr>
<tr>
<td>google_message3_3</td>
<td>650.456MB/s</td>
<td>1.040GB/s</td>
<td>593.52MB/s</td>
<td>580.667MB/s</td>
<td>346.839MB/s</td>
<td>123.978MB/s</td>
<td>35.893MB/s</td>
<td>2.32834MB/s</td>
</tr>
<tr>
<td>google_message3_4</td>
<td>8.70154GB/s</td>
<td>9.825GB/s</td>
<td>5.88645GB/s</td>
<td>5.93946GB/s</td>
<td>2.44388GB/s</td>
<td>5.9241GB/s</td>
<td>4.05837GB/s</td>
<td>876.87MB/s</td>
</tr>
<tr>
<td>google_message3_5</td>
<td>246.33MB/s</td>
<td>443.993MB/s</td>
<td>283.278MB/s</td>
<td>259.167MB/s</td>
<td>206.37MB/s</td>
<td>37.0285MB/s</td>
<td>12.2228MB/s</td>
<td>1.1979MB/s</td>
</tr>
<tr>
<td>google_message4</td>
<td>1.56674GB/s</td>
<td>2.19601GB/s</td>
<td>776.907MB/s</td>
<td>770.707MB/s</td>
<td>702.931MB/s</td>
<td>1.49623GB/s</td>
<td>205.116MB/s</td>
<td>8.93428MB/s</td>
</tr>
</tbody></table>
\* The cpp performance can be improved by using [tcmalloc](https://gperftools.github.io/gperftools/tcmalloc.html), please follow the (instruction)[https://github.com/protocolbuffers/protobuf/blob/main/benchmarks/README.md] to link with tcmalloc to get the faster result.

1
libs/protobuf/docs/third_party.md
View File

@@ -203,4 +203,3 @@ There are miscellaneous other things you may find useful as a Protocol Buffers d
* [vim-protolint: A protobuf linter for Vim](https://github.com/yoheimuta/vim-protolint)
* [super-linter: Protocol Buffer lint as GitHub Action](https://github.com/github/super-linter)
* [protoc-gen-fieldmask - A plugin to generate static type fieldmask paths](https://github.com/idodod/protoc-gen-fieldmask)
* [protoc-gen-bq-schema - A protoc plugin to generate BigQuery schema files](https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema)