Improve mempool metrics

[jevonearth]

Currently we scrape the mempool RPC and count the various kind's of operations. We record this as a GaugeValue.

Using a CounterValue would be better, but we are blocked in implementing this as we don't have a way of getting monotonic counts of mempool operations.

We have opened a feature request on the tezos gitlab issue tracker here:

https://gitlab.com/tezos/tezos/issues/512

[jevonearth]

The /chains/main/mempool/monitor_operations RPC satisfies what we need here.

Crude shell implementation;
while true; do curl -s --no-buffer http://localhost:8732/chains/main/mempool/monitor_operations\?applied | jq '.[].contents[].kind' | sort |uniq -c; done

The RPC server hangs up after every block, which means we would need to reconnect every-time the server disconnects. This means we could miss streamed operations during the reconnect window.

There are four filter args;

• applied
• branch_refused
• refused
• branch_delayed

Which I believe means we need to have four connections/listeners for each filter kind. Having a histogram metric for the re-connection time might be useful to spot long reconnect times.