🛠️ A simple CLI utility for monitoring protobuf-encoded messages in Kafka topics

  • By Rinat Shay.
  • Last update: Oct 19, 2022
  • Comments: 1

kafka-proto-monitor

Lints Tests

🛠️ A simple CLI utility for monitoring protobuf-encoded messages in Kafka topics.

Under the hood, kafka-proto-monitor relies on protoreflect in order to use raw (not compiled) proto files in runtime for unmarshaling Kafka messages.

Installation

go install github.com/ri-nat/[email protected]

Usage

Usage:
  kafka-proto-monitor [OPTIONS]

Application Options:
  -b, --broker=               Kafka broker URL (you can use this option multiple times) (default: localhost:9092)
  -t, --topic=                Kafka topic to read from (you can use this option multiple times)
  -a, --read-earliest         Read topic starting from the beginning (false by default)
  -r, --print-headers         Print message headers (false by default)
  -p, --proto-file=           Path to proto file
  -m, --proto-message=        Proto message to use
  -e, --proto-message-header= Name of Kafka message header, that contains message's proto name

Help Options:
  -h, --help                  Show this help message

Examples

kafka-proto-monitor -p ./service.proto -m 'app.users.UserCreated' -t users -a -r
  • Use app.users.UserCreated message from service.proto
  • Connect to Kafka on localhost:9092 (default)
  • Read from topic users
  • Read messages from the beginning of the topic (-a)
  • Print message headers (-r)
kafka-proto-monitor -p ./service.proto -e proto-name -b kafka:9092 -t users
  • Use all messages from service.proto
  • Use proto-name header content as proto message name
  • Connect to Kafka on kafka:9092
  • Read from topic users
  • Read only new messages (no -a flag)
  • Do not print message headers (no -r flag)

TODO

  • Display help message on startup if no arguments provided
  • Filter messages by header value
  • Filter messages by parsed object attributes
  • Construct proto name programmatically
  • Be more verbose about message unmarshaling errors
  • Use multiple proto files at once
  • Filter printable attributes out
  • Write more tests

Licensing

This software is licensed under the MIT License. See LICENSE for the full license text.

Download

kafka-proto-monitor.zip

Comments(1)

  • 1

    build(deps): bump github.com/jhump/protoreflect from 1.13.0 to 1.14.0

    Bumps github.com/jhump/protoreflect from 1.13.0 to 1.14.0.

    Release notes

    Sourced from github.com/jhump/protoreflect's releases.

    v1.14.0

    "github.com/jhump/protoreflect/dynamic"

    Changes/fixes:

    • If converting or merging a generated message into a *dynamic.Message, if the generated message contained dynamic extensions (extension fields not known to the Protobuf runtime's global registry), they would be lost during the conversion. This has been fixed.

    "github.com/jhump/protoreflect/grpcreflect"

    Additions:

    • Adds a new grpcreflect.NewClientAuto method, which returns a client that will automatically use either the v1 version of server reflection or v1alpha, depending on what the server supports.
    • Adds a new grpcreflect.NewClientAlpha method which is meant to replace grpcreflect.NewClient. It always uses the v1alpha version of server reflection (and its predecessor, grpcreflect.NewClient, is now deprecated since it is unclearly named).
    Commits
    • 3b27be2 grpcreflect supports v1 of the reflection service (#535)
    • ac1361c update protoc to 21.7; also adds go 1.19 to CI (#534)
    • 101791c gofmt 1.19; add support for v2 extendable messages in *dynamic.Message (#533)
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

Popular Tag

Related