Maybe this exists and didn't see it in the docs but is there a way to get txhash ZMQ notices just for wallet-affecting TXes?
If not I guess this is a feature request. I saw a request for JSON-encoded ZMQ notices that was shot down that was pretty good with the official answer being to decode the raw TX or use RPC, I think this would be great for the latter since it would cut down on the number of RPC calls greatly.
Is this a problem?
For sure, once Bitcoin Core has a sufficient # of addresses and TXes under it's belt RPC is lagged out greatly, so to call gettransaction/getrawtransaction on every TX on the blockchain would be slow as Christmas. But a long polling RPC method would work as well as this.
Even if this was implemented, on the end of the day, you shouldn't rely on ZMQ notifications. What would you do If the subscriber is down or there is some other problem and it loses notifications? — note that at the moment you can detect if messages were lost but you can't recover them.
If not I guess this is a feature request.
I guess you want to have a reliable notification system since we are talking about wallet transactions, that's why I suggest you to close this — shouldn't be the way to go — and help push #13262 forward as it looks the best solution at the moment.
@promag I think this would still provide value. #13262 sounds like a good solution if you are watching a single wallet. If I am wanting to watch 100s of wallets, the ideal solution to me would be to use #13262 to make sure nothing was missed when the watching service comes up, but then use ZMQ messages filtered for just the wallets I care about going forward.
I guess I am wondering if the long polling requests would work for 100s of wallets?
A long poll per wallet would probably be an overkill since each poll requires an RPC connection/thread. It could probably be scaled by adding a multi-wallet listsinceX call where one could specify the wallets-of-interest for a single poll.
i'm new to this project but in general you guys are dead wrong about RPC and network. RPC is trash for what you're trying to do. So much latency with TCP round trips each reconnect, come on
for the usual requirements RPC is good enough
no, because small delays matter; they become combinatorial in a large scale system
it took me forever to rifle through this BS before i could even find importpubkey RPC call
i need notifications on each one of my pubkeys imported
how is RPC going to do that?
Yeah, definitely can't say the current RPC is good enough. Bitcoind RPC slows down more and more as wallets grow and expecting people to decode raw binary Bitcoin TXes across any language/situation isn't really realistic. That's why ZMQ with an already decoded JSON payload would be about ideal.
That's why ZMQ with an already decoded JSON payload would be about ideal.
That's off topic here, it was discussed in #11614 and #14876.
Bitcoind RPC slows down more and more as wallets grow
Well it's not the RPC that slows down, it's the wallet. There's probably room for some optimizations there, but if you need it to grow unbounded then you should consider another approach to scale.
and expecting people to decode raw binary Bitcoin TXes across any language/situation isn't really realistic
Again, that's a different issue.
how is RPC going to do that? @reqshark I don't mean there's awesome support for that, but we should strive for a reliable notification system, and ZMQ PUB/SUB doesn't look like the the right tool.
Suppose your ZMQ subscriber is down, you would miss notifications and then you would have to query for those.
It's exactly on topic, read the 1st post in this ticket :)
RPC/wallet slowing down is effectively the same thing - end result is slow RPC whatever component is to blame.
In my mind you would be using ZMQ persistently and follow up with a listsinceX RPC call every minute or 2 as backup.
I guess I am wondering if the long polling requests would work for 100s of wallets? @pooleja currently a RPC connection is "binded" to one wallet. BTW, I don't see the problem of having one connection per wallet. If you plan to have a lot of wallets then consider having more than one node?
@Crypto2 sorry but the issue is about "ZMQ notices just for wallet-affecting TXes?" which there is no plan to support. Supporting JSON messages is a different issue which doesn't help regarding wallet notifications.
In my mind you would be using ZMQ persistently and follow up with a listsinceX RPC call every minute or 2 as backup.
I say we could do better: in my mind, the best solution is to support websocket in the HTTP server and then use the approach @jonasschnelli used regarding the wallet log. Meanwhile long poll is a great and reliable alternative - you don't have to support 2 "streams".
Websocket would be ideal for sure, my approach just involves less work and minor mods to an existing function.
got going today on a decent workaround for solving my need here, which was:
notifications on each one of my pubkeys imported
that's basically the issue:
notices just for wallet-affecting TXes
there's a config flag to execute a command for wallet-affected TX: -walletnotify=
i'm pretty sure zmq has a similar executable, but if not it's super easy to install nanomsg or nng. i happen to be using nanomsg (as well as zmq). you could also just compile your own simple zmq publisher executable to do run a single shot publish-and-exit operation.
anyway i fixed it by going into the bitcoin.conf file and adding this line:
walletnotify=/usr/local/bin/nngcat --pub --connect tcp://10.0.23.72:49000 -D"%s" -d 1
the command for legacy nanomsg would be
walletnotify=/usr/local/bin/nanocat --pub --connect tcp://10.0.23.72:49000 -D"%s" -d 1
make sure the last param -d 1 is included so that your pub can have time to connect. if you don't include that the publisher will exit before having time to complete the TCP connection
We have used walletnotify before as well, the main issue is unless you are obtaining the TX data from elsewhere (another copy or copies of Bitcoin running with txindex=1, an API like blockchain, etc.) it will still cost you a slow RPC call to get the data from bitcoind.
ya i would love the option to specify type of data from TX update passed to%s
either just plain txid or a nice JSON of that tx,
because exactly what you're saying, guess what my next call is to bitcoind after i know which tx
actually options are complicated, just need -walletnotifytx= and --walletnotifyjson=
The feature request didn't seem to attract much attention in the past. Also, the issue seems not important enough right now to keep it sitting around idle in the list of open issues.
Closing due to lack of interest. Pull requests with improvements are always welcome.