Add -mempoolnotify to call an external script on mempool transactions #6072

1. 
   promag commented at 4:01 pm on April 27, 2015: member

   This pull request adds support for transactions hitting the memory pool notifications. It is similar to -blocknotify and -walletnotify.

   Useful when bitcoind is configured without wallet support and one wants to be notified as soon as the transaction arrives.

2. Add -mempoolnotify to call an external script on mempool transactions 73e603e2ec
3. 
   sipa commented at 4:05 pm on April 27, 2015: member

   Do you really want to call a script several times per second? You’re better off polling getrawmempool, I would think.

   Not saying polling is better than a callback, but the script-execution mechanism is very inefficient. With zeromq notifications, things may be different.

4. 
   promag commented at 4:14 pm on April 27, 2015: member

   I think RPC pooling is not the way to go. Having an incremental notification is definitely the best option.

   I also think having a script-execution mechanism is not that good, but it is the existing solution, therefore I just followed the same implementation.

   As discussed in #1974, I would also prefer a named pipe. This way it should be easier to integrate with different MQs other than zeromq.

5. 
   jgarzik commented at 5:04 pm on April 27, 2015: contributor

   Yes, we really just need to add a notification socket for local listeners that need high volume data like this. It also solves related problems such as notification storms during tx storms etc.
6. 
   sipa commented at 5:07 pm on April 27, 2015: member

   I understand that you just want to continue the existing mechanism, that is generally a very reasonable thing to do.

   However, no matter what, forking off up to several dozen shell scripts per second is just not a good idea. That has nothing to do with not wanting notification, but just with the current notification mechanism sucks (for high volume events, but also for other reasons). As jgarzik says: let’s fix that first.

7. 
   promag commented at 5:15 pm on April 27, 2015: member

   In the case of staying with script-execution, bitcoind could just fork once when starting and then write to stdin of each script (instead of script arguments).
8. 
   jgarzik commented at 8:19 pm on April 27, 2015: contributor

   Sure; piping a high volume stream of events to a single program is reasonable as well. The point is to avoid initiating fresh executions of a script upon each new event.

   This is the same problem as CGI per-hit execution in the mid-1990s. There is a large menu of solutions from which to choose.

9. 
   promag commented at 8:56 pm on April 27, 2015: member

   Piping sounds the best for me as a low term, cheap solution. It has no external dependency, can handle script crash (a plugin like system can’t) and allows great integration in 3rd infrastructures.
10. 
    jgarzik commented at 2:24 am on April 28, 2015: contributor

    Sockets are the easiest, least complex solution. Getting pipes to work correctly on all platforms is not easy.
11. 
    promag commented at 9:08 am on April 28, 2015: member

    In the case of using sockets, should bitcoind listen/accept connections or connect to a given host:port?
12. 
    jgarzik commented at 11:02 am on April 28, 2015: contributor

    @promag The standard pattern is a listen socket for bitcoind. Processes attach to a localhost port and receive events. If that process dies, the socket automatically closes. The admin is responsible for installing whatever watchdog processes.
13. 
    jonasschnelli commented at 11:40 am on April 28, 2015: contributor

    I think ZMQ could be the long term perspective of these *notify features (especially for rapid I/O stuff like mempool).

    ZMQ supports asynchronous I/O over sockets. It runs on POSIX bases OSs and windows. The wallet (process) separation could also be done over ZMQ. IMO it would make more sense to continue at #5303 instead of adding -mempoolnotify.

14. 
    jgarzik commented at 11:45 am on April 28, 2015: contributor

    @jonasschnelli Agreed and #5303 does that.

    However, sockets do not require a library so sometimes that is chosen.

15. laanwj added the label Feature on Apr 29, 2015
16. 
    jgarzik commented at 4:07 pm on May 2, 2015: contributor

    Closing. Consensus is to NAK as-is, as execution storms etc. are possible.

    However, it is OK to provide this event signal via some other means, such as zmq or socket.

17. jgarzik closed this on May 2, 2015

    ---

18. 
    promag commented at 1:39 pm on May 4, 2015: member

    Noticed another PR similar to this one #5328
19. DrahtBot locked this on Sep 8, 2021

Contributors
promag sipa jgarzik jonasschnelli

Labels
Feature