A Prometheus exporter for Ethereum Execution & Consensus clients

  • By Sam Calder-Mason
  • Last update: Dec 13, 2022
  • Comments: 15

🦄 Ethereum Metrics Exporter 🦄

A Prometheus metrics exporter for Ethereum execution & consensus nodes

Ethereum client implementations expose extensive Prometheus metrics however there is minimal standardization around the metrics structure. This makes observability across multiple clients a painful experience. This exporter hopes to help alleviate this problem by creating a client-agnostic set of metrics that operators can run without any additional configuration to dashboards or alerting.

To provide these client-agnostic metrics the exporter relies entirely on these well-defined APIs:

Naturally this means that the exporter is limited to metrics that are exposed by these APIs.

Built With

Usage

A tool to export the state of ethereum nodes

Usage:
  ethereum-metrics-exporter [flags]

Flags:
      --config string                   config file (default is $HOME/.ethereum-metrics-exporter.yaml)
      --consensus-url string            (optional) URL to the consensus node
      --execution-modules strings       (optional) execution modules that are enabled on the node
      --execution-url string            (optional) URL to the execution node
  -h, --help                            help for ethereum-metrics-exporter
      --metrics-port int                Port to serve Prometheus metrics on (default 9090)
      --monitored-directories strings   (optional) directories to monitor for disk usage
  -t, --toggle                          Help message for toggle

Getting Started

Grafana

Docker

Available as a docker image at samcm/ethereum-metrics-exporter

Quick start

docker run -d -it --name ethereum-metrics-exporter -p 9090:9090 -it samcm/ethereum-metrics-exporter --consensus-url=http://localhost:5052 --execution-url=http://localhost:8545

With a config file

docker run -d -it --name ethereum-metrics-exporter -v $HOST_DIR_CHANGE_ME/config.yaml:/opt/exporter/config.yaml -p 9090:9090 -it samcm/ethereum-metrics-exporter --config /opt/exporter/config.yaml

Kubernetes via Helm

Read more

helm repo add ethereum-helm-charts https://skylenet.github.io/ethereum-helm-charts

helm install ethereum-metrics-exporter ethereum-helm-charts/ethereum-metrics-exporter -f your_values.yaml

Standalone

Downloading a release

Coming soon.

Building yourself (requires Go)

  1. Clone the repo 
    go get github.com/samcm/ethereum-metrics-exporter
  2. Change directories 
    cd ./ethereum-metrics-exporter
  3. Build the binary 
     go build -o ethereum-metrics-exporter .
  4. Run the exporter 
     ./ethereum-metrics-exporter

Screenshots

Example

Contributing

Contributions are greatly appreciated! Pull requests will be reviewed and merged promptly if you're interested in improving the exporter!

  1. Fork the project
  2. Create your feature branch:
    • git checkout -b feat/new-metric-profit
  3. Commit your changes:
    • git commit -m 'feat(profit): Export new metric: profit
  4. Push to the branch: -git push origin feat/new-metric-profit
  5. Open a pull request

Contact

Sam - @samcmau

Download

ethereum-metrics-exporter.zip

Comments(15)

  • 1

    txpool_status does not exist

    It seems like this project is calling a txpool_status method on the execution RPC endpoing which does not exist on Geth 1.10.18-stable.

    Geth is complaining with these messages in the log:

    WARN [06-06|19:45:59.953] Served txpool_status                     conn=127.0.0.1:55350 reqid=0 duration="37.21µs"   err="the method txpool_status does not exist/is not available"
  • 2

    Feat(consensus): Peer client detection

    Adds a new metric, eth_con_peer_agents, which contains the amount of remote peers for each beacon client implementation.

    e.e.

    eth_con_peer_agents{agent="lighthouse",ethereum_role="consensus",module="general",node_name="consensus-client"} 73

    Note: It appears only nimbus has support for this metric. Other implementations don't return enough data in the /eth/v1/node/peers responses to derive this metric.

  • 3

    feat: Merge cleanup

    The merge is complete 🎉

    Removes all TT/TTD metrics since we don't really need them anymore. With the push for CL clients supporting merge-from-genesis, there shouldn't be any need for us to support POW anymore (e.g. testnets)

  • 4

    feat(consensus): Use beacon package

    Swaps the beacon and api packages in the consensus package out for the ones that exist in github.com/samcm/beacon. That package is more mature, has more features, and can pretty much be swapped out without issue 👍

  • 5

    fix(consensus): Fetch finality checkpoints on a loop

    This fixes a bug noticed by @remyroy where the finalized/justified epoch reported by the metrics exporter could be 1 epoch behind reality. The exporter listens for finalized_checkpoint events, and then immediately fetches the head finality_checkpoint from the beacon node. This is usually fine, but in some cases the beacon node can take a little while to return the new checkpoint.

    This fix adds a sleep of 3s when the event is received before firiing off the API request, and also just fetches the checkpoints on a loop every 60s.

  • 6

    fix(execution): Fix panic when nil block is returned

    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0xd1f789]

goroutine 90 [running]:
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).getHeadBlockStats(0xc0003acc18, {0x11907b0, 0xc00012e000})
	/src/pkg/exporter/execution/jobs/block.go:226 +0x189
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).tick(0xc0003acc18, {0x11907b0?, 0xc00012e000?})
	/src/pkg/exporter/execution/jobs/block.go:202 +0x2c
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).Start(0x0?, {0x11907b0, 0xc00012e000})
	/src/pkg/exporter/execution/jobs/block.go:196 +0x4e
created by github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution.(*metrics).StartAsync
	/src/pkg/exporter/execution/metrics.go:146 +0x30c
  • 7

    SIGSEGV at start if execution module is disabled in config

    I ran into the following issue when trying to use the exporter on a machine that only runs a consensus layer client.

    With the following config:

    consensus:
  enabled: true
  url: "http://localhost:5052"
  name: "consensus-client"
execution:
  enabled: false
diskUsage:
  enabled: false

    The exporter fails to start with the following error output:

    {"cfgFile":"/home/ethmetrics/.ethereum-metrics-exporter.yaml","level":"info","msg":"Loading config","time":"2022-08-23T00:48:48+02:00"}
{"component":"exporter","level":"info","msg":"Initializing...","time":"2022-08-23T00:48:48+02:00"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xd3bff2]

goroutine 8 [running]:
github.com/samcm/ethereum-metrics-exporter/pkg/exporter.(*exporter).Serve(0xc0000cb290, {0x10f9ac0?, 0xc00003a078}, 0x238e)
        /home/admin/ethereum-metrics-exporter/pkg/exporter/exporter.go:132 +0xb2
github.com/samcm/ethereum-metrics-exporter/cmd.glob..func1(0x16cf240, {0xf686bf?, 0x4?, 0x4?})
        /home/admin/ethereum-metrics-exporter/cmd/root.go:20 +0x5c
github.com/spf13/cobra.(*Command).execute(0x16cf240, {0xc0000360b0, 0x4, 0x4})
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x16cf240)
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/samcm/ethereum-metrics-exporter/cmd.Execute()
        /home/admin/ethereum-metrics-exporter/cmd/root.go:47 +0x25
created by main.main
        /home/admin/ethereum-metrics-exporter/main.go:16 +0x85

    The expected behaviour is that the exporter should start with the execution plugin disabled, rather than encountering SIGSEGV.

    If execution is disabled in config, the execution property of the exporter is not initialised in the Init function of pkg/exporter/exporter.go. Because of this, the error appears when trying to call the e.execution.URL() method in the Serve function.

    The issue can be avoided by replacing e.execution.URL() with e.config.Execution.URL:

    func (e *exporter) Serve(ctx context.Context, port int) error {
        e.log.
                WithField("consensus_url", e.config.Consensus.URL).
                //WithField("execution_url", e.execution.URL()).
                WithField("execution_url", e.config.Execution.URL).
                Info(fmt.Sprintf("Starting metrics server on :%v", port))
  • 8

    feat(pair): Add consensus_mechanism metric

    Adds a new exporter job, pair, which requires both the execution & consensus modules to be enabled. There is a single metric exported:

    eth_pair_consensus_mechanism{consensus_mechanism="Proof of Stake",consensus_mechanism_short="PoS"} 3
  • 9

    feat(consensus): Create beacon.Node

    Creates a new package consensus/beacon. This package is intended to be the central point of reference for the jobs in the jobs packages.

    This will faciliate us exporting complex metrics, like expected slot/epoch, missed blocks, etc.

  • 10

    feat(consensus): Export state roots as a bounded hash range

    This change adds two new metrics:

    • eth_con_beacon_finality_checkpoint_hash following the head checkpoints for finalized/justified/previous_justified
    • eth_con_beacon_head_slot_hash following the head slot

    These metrics observe the state roots, and consistently convert them in to an integer between 0 and 65536. These metrics can then be used to observe which chain a beacon node is following by comparing this value at the current time to other nodes.

    To work around scrape intervals, a query like this can be used to try and avoid noise: max(eth_con_beacon_head_slot_hash)[1m].

  • 11

    linux/arm64 docker image not compatible

    ethereum-metrics-exporter The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

  • 12

    Nimbus - Failed to fetch peers

    Following error is repeating in the logs:

    Nov 19 18:00:54 Ubuntu-VM ethereum-metrics-exporter[21296]: {"component":"exporter","error":"nats: maximum payload exceeded","level":"error","module":"consensus/beacon","msg":"Failed to fetch peers","time":"2022-11-19T18:00:54-07:00"}

    All other metrics seem to pull in just fine, both for execution and remaining consensus metrics. Also, able to get peers from Nimbus using the metrics exported by Nimbus itself with no problem.

    Attached logfile of the startup of ethereum-metrics-exporter. Nimbus v. 22.10.1 Besu v. 22.10.0 go v. 1.19.3 prometheus v. 2.39.1

    ethereum-metrics-exporter.log

  • 13

    teku metrics exporter error

    teku - 22.9.1 geth - 1.10.23

    {"component":"exporter","error":"fetched block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-09-21T23:46:26Z","topic":"block"}

  • 14

    Failed to request signed beacon block

    Hi,

    Trying out this awesome exporter with Prometheus and Grafana and got it working, but still see an error in the logs. Using Geth (latest) for EL and Prysm (latest) for CL.

    The error in the logs show this:

    ethereum-metrics-exporter[25186]: {"component":"exporter","error":"failed to request signed beacon block: GET failed with status 500: {\"message\":\"GetBlockV2: rpc error: code = NotFound desc = Could not find requested block: signed beac on block can't be nil\",\"code\":500}","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-08-01T11:21:00+02:00","topic":"block"}

    Is this a misconfiguration from my side or could it be something else?

    Flags used: --consensus-url=http://localhost:3500 --execution-url=http://localhost:8545

    Prysm public API: https://docs.prylabs.network/docs/how-prysm-works/prysm-public-api

  • 15

    Add metric for basefee

    The EL blocks contain a field called basefee. This basefee fluctuation is something we'd like to track on shadow forks. It'd be great if we can add this and update the dashboard to visualize the fluctuations.

Popular Tag

Related