ZMQ pub/sub has a default limit of 1000 buffered messages. When it hits the limit messages are dropped. This happens sometimes when a new block is found and ~2500messages ( 2.5MB) are send (almost) at once.
50-800 usually got lost on my (cheap) node.
ZMQ_SNDHWM
The PUBlisher limit, SNDHWM, is disabled by setting it to 0. On very slow networks it might fill up memory a bit, but that is hardly noticeable since the queue is usually emptied in a few hundred milliseconds.
Same applies to the SUBscriber (although the bottleneck there is only the local processing, not the network throughput), so it would be good practice to set RCV_HWM to zero as well for some client script, although I was only able to hit the limit on the subscriber side after adding sleep(1) on every rawtx.
I've tested for about a day and had a bunch of subscribers running on different computers, and not a single message got dropped. ( also while using swap and 100% cpu stress --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 1.9;}' < /proc/meminfo)k --vm-keep -m 1 -c 1 -i 2 )
I forced one subscriber to lag about 10000 messages , and they were still not dropped.
I'd appreciate it if others test their performance and load. You can just use this script and check for missing messages while bitcoind is running. (for testing max throughput, stop bitcoind for a while, then restart it; it will send lots of messages while syncing)
subscriber testscript: https://gist.github.com/FeedTheWeb/e6243410cb463759d5b29fae24550f36
