rpc: prevent circular deps by extracting into separate include #18871

pull brakmic wants to merge 1 commits into bitcoin:master from brakmic:prevent-circular-deps changing 42 files +715 −647
  1. brakmic commented at 10:16 pm on May 4, 2020: contributor

    This PR is only a draft.

    The description below will change over time.


    This PR removes a potential circular dependency that was discovered in travis while building binaries for this PR #18827

    The affected sources/includes in that PR are httprpc and rpc/server

    My first “solution” was a forward declare of a single function in rpc/server.cpp. However, that merely defeats the linter and could not count as a proper solution.

    After some discussion #18827#pullrequestreview-404116986 I tried to extract certain classes and function declarations into a new include file, rpc/interfaces.h

    Now, the circular dependency in that PR is gone, but as this constitutes a much bigger change than the PR wanted to make, I think it should better get its own PR.

    The following classes and functions got moved into a new source file pair rpc/interfaces.cpp and rpc/interfaces.h:

     0class RPCTimerBase
     1class RPCTimerInterface
     2class CRPCCommand
     3class CRPCTable
     4
     5void SetRPCWarmupStatus(const std::string& newStatus);
     6void SetRPCWarmupFinished();
     7
     8bool RPCIsInWarmup(std::string *outStatus);
     9bool IsRPCRunning();
    10
    11int64_t GetStartupTime();
    12int RPCSerializationFlags();
    13
    14std::string JSONRPCExecBatch(const JSONRPCRequest& jreq, const UniValue& vReq);
    15
    16void RPCSetTimerInterface(RPCTimerInterface *iface);
    17void RPCSetTimerInterfaceIfUnset(RPCTimerInterface *iface);
    18void RPCUnsetTimerInterface(RPCTimerInterface *iface);
    19void RPCRunLater(const std::string& name, std::function<void()> func, int64_t nSeconds);
    20
    21bool IsDeprecatedRPCEnabled(const std::string& method);
    

    In src/Makefile.am there is a new library defined: LIBBITCOIN_SERVER_INTERFACES that contains the implementation and is linked with bitcoind and bitcoin-qt (test and bench libraries are linked as well).

    The changes reduce the size of rpc/server.cpp substantially. What remains there are pure server management functionalities:

    0namespace RPCServer
    1{
    2    void OnStarted(std::function<void ()> slot);
    3    void OnStopped(std::function<void ()> slot);
    4}
    5
    6void StartRPC();
    7void InterruptRPC();
    8void StopRPC();
    

    Another advantage coming from these changes is that we got rid of forward declarations like class CRPCTable, class CRPCCommand etc.


    Possible future changes in class structure and function grouping

    Although this PR isn’t aiming for more complex engineering changes, like introducing new classes and other structures, there might be some advantage if we’d in future group the “extracted” functions into new classes. Just like we have CRPCTable that carries table-oriented functions, or CRPCCommand, we could introduce a new class called CRPCTimer:

    0class CRPCTimer {
    1public:
    2    /** Set the factory function for timers */
    3    void RPCSetTimerInterface(RPCTimerInterface *iface);
    4    /** Set the factory function for timer, but only, if unset */
    5    void RPCSetTimerInterfaceIfUnset(RPCTimerInterface *iface);
    6   /** Unset factory function for timers */
    7   void RPCUnsetTimerInterface(RPCTimerInterface *iface);
    8};
    

    And server-status functions could be moved into a new class like this:

    0class CRPCServerStatus {
    1public:
    2    void SetRPCWarmupStatus(const std::string& newStatus);
    3    void SetRPCWarmupFinished();
    4    bool RPCIsInWarmup(std::string *outStatus);
    5    bool IsRPCRunning();
    6    int64_t GetStartupTime();
    7private:
    8   [...here come mutexes, guards, atomics etc. ]
    9};
    

    Then we could give access to their instances the same way we do with CRPCtable tableRPC now.

    0extern CRPCTimer timerRPC;
    1extern CRPCServerStatus statusRPC;
    
  2. sipa commented at 10:20 pm on May 4, 2020: member

    This is still just defeating the linter, I’m afraid. It assumes that a .h file and a .cpp with the same name form a unit.

    The question it’s trying to answer is “can you use module X without having module Y or the other way around”, where a module is seen as defined by F.cpp + F.h (if both exist).

  3. brakmic commented at 10:21 pm on May 4, 2020: contributor

    This is still just defeating the linter, I’m afraid. It assumes that a .h file and a .cpp with the same name form a unit.

    The question it’s trying to answer is “can you use module X without having module Y or the other way around”, where a module is seen as defined by F.cpp + F.h (if both exist).

    What if I would create rpc/interfaces.cpp that contains all the implementations?

  4. sipa commented at 10:22 pm on May 4, 2020: member

    What if I would create rpc/interfaces.cpp that contains all the implementations?

    If that works, it’s a solution indeed.

  5. brakmic marked this as a draft on May 4, 2020
  6. fanquake added the label RPC/REST/ZMQ on May 4, 2020
  7. DrahtBot commented at 0:31 am on May 5, 2020: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #18830 (rpc: add user field to getrpcinfo’s json by brakmic)
    • #18827 (rpc: added getrpcwhitelist method by brakmic)
    • #18814 (rpc: Relock wallet only if most recent callback by promag)
    • #18677 (Multiprocess build support by ryanofsky)
    • #18606 (test: checks that bitcoin-cli autocomplete is in sync by pierreN)
    • #18531 (rpc: Assert that RPCArg names are equal to CRPCCommand ones by MarcoFalke)
    • #18452 (qt: Fix shutdown when waitfor* cmds are called from RPC console by hebasto)
    • #18267 (BIP-325: Signet [consensus] by kallewoof)
    • #18130 (Replace uses of boost::trim* with locale-independent alternatives by Empact)
    • #17356 (RPC: Internal named params by luke-jr)
    • #17127 (util: Correct permissions for datadir and wallets subdir by hebasto)
    • #16710 (build: Enable -Wsuggest-override if available by hebasto)
    • #16549 (UI external signer support (e.g. hardware wallet) by Sjors)
    • #16546 (External signer support - Wallet Box edition by Sjors)
    • #16365 (Log RPC parameters (arguments) if -debug=rpcparams by LarryRuane)
    • #15382 (util: add runCommandParseJSON by Sjors)
    • #14810 (qt: Enable tabbing through labels by hebasto)
    • #13686 (ZMQ: Small cleanups in the ZMQ code by domob1812)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  8. brakmic commented at 6:45 pm on May 5, 2020: contributor

    What if I would create rpc/interfaces.cpp that contains all the implementations?

    If that works, it’s a solution indeed.

    I have now extracted CRPCCommand, CRPCTable, various RPCTimer*-structures and Status/WarmUp functions from rpc/server.cpp into the new rpc/interfaces.cpp.

    I’m deliberately using the plural form “interfaces” to indicate that these are interfaces that deal with RPC Server, but could be later split up into even smaller units, if needed.

    Also, the shutdown.h/*.cpp got removed to get rid of this linker error:

    0 AR       libbitcoin_wallet.a
    1  CXXLD    bitcoin-wallet
    2  CXXLD    bitcoind
    3Undefined symbols for architecture x86_64:
    4  "StartShutdown()", referenced from:
    5      stop(JSONRPCRequest const&) in libbitcoin_common.a(libbitcoin_common_a-interfaces.o)
    6ld: symbol(s) not found for architecture x86_64
    7clang: error: linker command failed with exit code 1 (use -v to see invocation)
    

    I think that moving the shutdown functionality into rpc/interfaces is acceptable, because other server-control functions are located there as well, like IsRPCRunning and others.

    While I was working on rpc/interfaces I also got a chance to remove all those forward class declarations class CRPCTable, which btw. provoked warnings like these:

     0Making all in src
     1  CXX      wallet/libbitcoin_wallet_a-rpcwallet.o
     2  AR       libbitcoin_wallet.a
     3  CXXLD    bitcoind
     4  CXXLD    bitcoin-wallet
     5ld: warning: duplicate symbol '_tableRPC' in:
     6    libbitcoin_util.a(libbitcoin_util_a-interfaces.o)
     7    libbitcoin_wallet.a(libbitcoin_wallet_a-rpcdump.o)
     8ld: warning: duplicate symbol '_tableRPC' in:
     9    libbitcoin_wallet.a(libbitcoin_wallet_a-wallet.o)
    10    libbitcoin_wallet.a(libbitcoin_wallet_a-rpcwallet.o)
    11ld: warning: duplicate symbol '_tableRPC' in:
    12    libbitcoin_wallet.a(libbitcoin_wallet_a-rpcwallet.o)
    13    libbitcoin_util.a(libbitcoin_util_a-interfaces.o)
    14ld: warning: duplicate symbol '_tableRPC' in:
    15    libbitcoin_zmq.a(libbitcoin_zmq_a-zmqrpc.o)
    16    libbitcoin_wallet.a(libbitcoin_wallet_a-wallet.o)
    17ld: warning: duplicate symbol '_tableRPC' in:
    18    libbitcoin_zmq.a(libbitcoin_zmq_a-zmqrpc.o)
    19    libbitcoin_wallet.a(libbitcoin_wallet_a-rpcdump.o)
    20ld: warning: duplicate symbol '_tableRPC' in:
    21    libbitcoin_zmq.a(libbitcoin_zmq_a-zmqrpc.o)
    22[...snip...]
    

    rpc/interfaces.cpp is part of libbitcoin_server_a_SOURCES in src/Makefile.am

    And just to make it even more complex for me, I also stumbled upon yet another circular dependency. :)

    0node/transaction -> validation -> rpc/interfaces -> rpc/util -> node/transaction
    

    But after I replaced node/transaction.h with util/error.h in rpc/util.h, it disappeared.

    I hope that this proposed solution makes more sense. There are other circular dependencies still, but those exist independently from the one I was chasing over the last few days.

     0Circular dependency: chainparamsbase -> util/system -> chainparamsbase
     1Circular dependency: index/txindex -> validation -> index/txindex
     2Circular dependency: policy/fees -> txmempool -> policy/fees
     3Circular dependency: qt/addresstablemodel -> qt/walletmodel -> qt/addresstablemodel
     4Circular dependency: qt/bitcoingui -> qt/walletframe -> qt/bitcoingui
     5Circular dependency: qt/recentrequeststablemodel -> qt/walletmodel -> qt/recentrequeststablemodel
     6Circular dependency: qt/sendcoinsdialog -> qt/walletmodel -> qt/sendcoinsdialog
     7Circular dependency: qt/transactiontablemodel -> qt/walletmodel -> qt/transactiontablemodel
     8Circular dependency: txmempool -> validation -> txmempool
     9Circular dependency: wallet/fees -> wallet/wallet -> wallet/fees
    10Circular dependency: wallet/wallet -> wallet/walletdb -> wallet/wallet
    11Circular dependency: policy/fees -> txmempool -> validation -> policy/fees
    
  9. in src/rpc/interfaces.h:21 in 0ce2518428 outdated
    16+#include <univalue.h>
    17+#include <sync.h>
    18+
    19+static const unsigned int DEFAULT_RPC_SERIALIZE_VERSION = 1;
    20+
    21+static std::atomic<bool> fRequestShutdown(false);
    


    sipa commented at 7:48 pm on May 5, 2020:

    Making this static and in a .h file means that every compilation unit (cpp file) that includes it will get its own copy of the variable. If you only want a single copy, it must be declared in a .cpp file, and defined in the corresponding .h file.

    I also think it’s very strange that shutdown is now merged into an RPC-specific module. Does nothing outside of RPC need it?


    brakmic commented at 7:53 pm on May 5, 2020:

    Making this static and in a .h file means that every compilation unit (cpp file) that includes it will get its own copy of the variable. If you only want a single copy, it must be declared in a .cpp file, and defined in the corresponding .h file.

    Thanks for the hint. Will correct it now.

    I also think it’s very strange that shutdown is now merged into an RPC-specific module. Does nothing outside of RPC need it?

    I never wanted to remove shutdown.h/cpp, but as I couldn’t pass the linker error I spoke above, the only way for me was to include shutdown’s three methods into rpc/interface.

    I haven’t encountered any problems with shutdown removed. Functional RPC-tests are passing.

    But I’ll gladly revert it, if there is a way to avoid the linker error.


    brakmic commented at 8:14 pm on May 5, 2020:

    it must be declared in a .cpp file, and defined in the corresponding .h file.

    I must admit that I don’t know how to do this. Especially with const variables and atomics.


    sipa commented at 8:19 pm on May 5, 2020:

    I’m being stupid sorry - it’s a static variable that’s local to shutdown.cpp. It can’t and doesn’t need to go in shutdown.h.

    I think you shouldn’t need to touch this at all. It’s perfectly fine as-as in shutdown.cpp. It also has no dependencies, so touching it shouldn’t be needed for the purpose of breaking circular dependencies.

    The issue you’re seeing while linking must have to do with the organization into libraries and possibly the order in which those are passed to the linker. That should be solvable by potentially moving things between libraries, but it’s hard to say what the exact issue is without tinkering with it myself.


    brakmic commented at 8:22 pm on May 5, 2020:

    I’m being stupid sorry - it’s a static variable that’s local to shutdown.cpp. It can’t and doesn’t need to go in shutdown.h.

    Thanks for the clarification! I was already questioning life and existence. :)

    I think you shouldn’t need to touch this at all. It’s perfectly fine as-as in shutdown.cpp. It also has no dependencies, so touching it shouldn’t be needed for the purpose of breaking circular dependencies.

    The issue you’re seeing while linking must have to do with the organization into libraries and possibly the order in which those are passed to the linker. That should be solvable by potentially moving things between libraries, but it’s hard to say what the exact issue is without tinkering with it myself.

    Ok, then I will bring back shutdown.h/cpp and then try to find a way.

  10. ryanofsky commented at 8:18 pm on May 5, 2020: member

    Have’t looked closely at this but I’d discourage using “prevent circular deps” as the main justification for a change. If a change makes sense you should be able to say what advantages it has without reference to the linter, and I don’t see other advantages cited in the PR description.

    The circular dependencies linter often suggests joining files together which make more sense separately and splitting things up which make more sense together. It’s also rare that it can’t be satisfied by just dropping unnecessary includes and using forward-declared types appropriately.

  11. sipa commented at 8:24 pm on May 5, 2020: member

    @ryanofsky Circular dependency here is not in the meaning of “can code be compiled without a cycle in #includes” but in the meaning of “can this module be used at a semantic level without this other module”. Forward declares do not solve that, they only hide the ability to detect it.

    The sort of circular dependencies we’re talking about here are a symptom of badly organized code, and improving it is a worthwhile goal in my opinion - but not at all costs. Sometimes it requires introducing costly abstractions like installing callbacks/signals, which just aren’t worth it. Often it can be solved by just moving things appropriately so that module dependencies form a DAG.

    I agree that the advantage can be stated without reference to the linter in that case.

  12. ryanofsky commented at 8:43 pm on May 5, 2020: member

    The sort of circular dependencies we’re talking about here are a symptom of badly organized code, and improving it is a worthwhile goal in my opinion - but not at all costs.

    I think we agree. I just think if the code is badly organized and the PR improves the organization, the PR description should actually say what was wrong with the organization before and how it’s improved. I’m definitely willing to believe there’s bad organization here, but I think it’d be good to take a few minutes and spell out what the actual problem is before spending too much time chasing linker and linter errors.

    Sometimes it requires introducing costly abstractions like installing callbacks/signals, which just aren’t worth it. Often it can be solved by just moving things appropriately so that module dependencies form a DAG.

    Definitely agree. I think if someone is introducing virtual classes and callbacks for no other reason than to satisfy the linter they are doing something wrong. I think there are cases like the fee estimation / mempool circular dependency where having modules call back and forth directly, but still be separate modules is a simple and good solution (https://github.com/bitcoin/bitcoin/pull/15638#discussion_r268718912)

  13. brakmic commented at 8:51 pm on May 5, 2020: contributor

    the PR description should actually say what was wrong with the organization before and how it’s improved.

    Yes, the description will be updated as soon as this PR becomes a real one. Currently, it’s only a draft and the only proper description is my first (longer) answer to @sipa regarding latest changes. As this is still a moving target and I am right now trying to overcome the linking error with shutdown.cpp any description in the header of this PR would sooner or later become obsolete. So, better not changing it every few minutes as this would only annoy people.

    I think if someone is introducing virtual classes and callbacks for no other reason than to satisfy the linter they are doing something wrong.

    My goal is to change nothing, that is: no magic via extra calls or structures to circumvent obstacles whatsoever. As @sipa said, it’s about creating modules that could be used independently from each other. Either I will succeed in creating proper binary modules out of existing structures and functions, or I will have to give up.

    So I hope it’ll work.

  14. brakmic commented at 9:34 pm on May 5, 2020: contributor

    @sipa

    shutdown.h/.cpp is back. I’ve removed its functions from rpc/interfaces.h/.cpp accordingly. RPC functional tests are passing. Unit tests as well. Qt code compiles too and its test pass also. This change introduced no new circular dependency that’d affect rpc/interfaces.

    That is, they stayed the same as before:

     0Circular dependency: chainparamsbase -> util/system -> chainparamsbase
     1Circular dependency: index/txindex -> validation -> index/txindex
     2Circular dependency: policy/fees -> txmempool -> policy/fees
     3Circular dependency: qt/addresstablemodel -> qt/walletmodel -> qt/addresstablemodel
     4Circular dependency: qt/bitcoingui -> qt/walletframe -> qt/bitcoingui
     5Circular dependency: qt/recentrequeststablemodel -> qt/walletmodel -> qt/recentrequeststablemodel
     6Circular dependency: qt/sendcoinsdialog -> qt/walletmodel -> qt/sendcoinsdialog
     7Circular dependency: qt/transactiontablemodel -> qt/walletmodel -> qt/transactiontablemodel
     8Circular dependency: txmempool -> validation -> txmempool
     9Circular dependency: wallet/fees -> wallet/wallet -> wallet/fees
    10Circular dependency: wallet/wallet -> wallet/walletdb -> wallet/wallet
    11Circular dependency: policy/fees -> txmempool -> validation -> policy/fees
    

    –EDIT: the only line of code that I had to change was in void SetRPCWarmupFinished() because of this complaint from Travis:

    0* Checking consistency between dispatch tables and vRPCConvertParams
    1CHECK_NONFATAL(condition) should be used instead of assert for RPC code.
    2src/rpc/interfaces.cpp:28:    assert(fRPCInWarmup);
    3^---- failure generated from test/lint/lint-assertions.sh
    

    So I replaced assert with CHECK_NONFATAL.

  15. brakmic commented at 6:33 pm on May 9, 2020: contributor

    What if I would create rpc/interfaces.cpp that contains all the implementations?

    If that works, it’s a solution indeed.

    Meanwhile, I have created a separate shared library that contains the “extracted” functions. There are no forward declarations anymore. I’ve also updated the PR description to reflect all the changes, and when you have time, it’d help me a lot if you could check them.

    Am I on the right track? Did I miss something important? Or is this whole thing simply not worth the effort?

    –EDIT: And I have no clue why AppVeyor is failing. Travis is fine.

  16. rpc: prevent circular deps by extracting into separate include fcd4f6155e
  17. brakmic closed this on May 10, 2020

  18. DrahtBot locked this on Feb 15, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-10-06 22:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me