kernel: Streamline util library #29015

pull ryanofsky wants to merge 13 commits into bitcoin:master from ryanofsky:pr/rmutil changing 121 files +1023 −526
  1. ryanofsky commented at 10:26 pm on December 6, 2023: contributor

    Remove fees.h, errors.h, and spanparsing.h from the util library. Specifically:

    • Move Split functions from util/spanparsing.h to util/string.h, using util namespace for clarity.
    • Move remaining spanparsing functions to script/parsing.h since they are used for descriptor and miniscript parsing.
    • Combine util/fees.h and util/errors.h into common/messages.h so there is a place for simple functions that generate user messages to live, and these functions are not part of the util library.

    Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn’t be called by kernel code or kernel applications. These changes should also improve code organization and make functions easier to discover. Some of these same moves are (or were) part of #28690, but did not help with code organization, or made it worse, so it is better to move them and clean them up in the same PR so code only has to change one time.

  2. DrahtBot commented at 10:26 pm on December 6, 2023: contributor

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Code Coverage

    For detailed information about the code coverage, see the test coverage report.

    Reviews

    See the guideline for information on the review process.

    Type Reviewers
    ACK TheCharlatan, hebasto, achow101
    Concept ACK Sjors

    If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #bitcoin-core/gui/599 (Translate unit names & fix external signer error by luke-jr)
    • #30212 (rename TransactionErrors: MISSING_INPUTS and ALREADY_IN_CHAIN by willcl-ark)
    • #30200 (Introduce Mining interface by Sjors)
    • #30043 (net: Replace libnatpmp with built-in PCP+NATPMP implementation by laanwj)
    • #29473 (refactor: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing by paplorinc)
    • #29432 (Stratum v2 Template Provider (take 3) by Sjors)
    • #29415 (Broadcast own transactions only via short-lived Tor or I2P connections by vasild)
    • #29365 (Extend signetchallenge to set target block spacing by starius)
    • #29346 (Stratum v2 Noise Protocol by Sjors)
    • #29278 (Wallet: Add maxfeerate wallet startup option by ismaelsadeeq)
    • #28710 (Remove the legacy wallet and BDB dependency by achow101)
    • #22838 (descriptors: Be able to specify change and receiving in a single descriptor string by achow101)
    • #21283 (Implement BIP 370 PSBTv2 by achow101)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  3. DrahtBot added the label Validation on Dec 6, 2023
  4. DrahtBot added the label CI failed on Dec 6, 2023
  5. ajtowns commented at 3:13 am on December 7, 2023: contributor

    Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn’t be called by kernel code or kernel applications.

    Is the idea here that util should be for generic code, and that these bits and pieces (like fees and error) are specific bitcoin features that are only relevant to wallet/p2p, so aren’t generic?

  6. ryanofsky force-pushed on Dec 7, 2023
  7. ryanofsky commented at 11:23 pm on December 7, 2023: contributor

    Updated 0e647454b49274ee82978c06b606f3fd9c02b779 -> 778c0214d25e5a012c61251d592257ae19805255 (pr/rmutil.1 -> pr/rmutil.2, compare) fixing CI errors from forgitting to add new files to git

    re: #29015 (comment)

    Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn’t be called by kernel code or kernel applications.

    Is the idea here that util should be for generic code, and that these bits and pieces (like fees and error) are specific bitcoin features that are only relevant to wallet/p2p, so aren’t generic?

    I do think that’s generally a good thing to do, but it’s not really the goal of this PR. The main thing this PR is trying to do is provide a better kernel API and reduce the size of the kernel library.

    Because libbitcoin_kernel depends on libbitcoin_util, any external application using the kernel will also depend on util. So we should try to avoid including functions and types in util that are not likely to be useful to external applications, or are unstable and meant for internal use. The common library is usually a better place for these things.

  8. ajtowns commented at 11:58 pm on December 7, 2023: contributor

    Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn’t be called by kernel code or kernel applications.

    Is the idea here that util should be for generic code, and that these bits and pieces (like fees and error) are specific bitcoin features that are only relevant to wallet/p2p, so aren’t generic?

    I do think that’s generally a good thing to do, but it’s not really the goal of this PR. The main thing this PR is trying to do is provide a better kernel API and reduce the size of the kernel library.

    Does that mean that things should only go in util if they’re going to be used by the kernel library then? That would presumably mean util/bitdeque.h and util/sock.cpp should also eventually go elsewhere, for example? That seems like a bad approach to me (in that it means code has to move into and out of util based on whether it’s relevant to the kernel, rather than due to any changes to the code itself, and users then potentially have to deal with it moving into/outof the util:: namespace).

  9. ryanofsky commented at 0:53 am on December 8, 2023: contributor

    Does that mean that things should only go in util if they’re going to be used by the kernel library then?

    No, I don’t think so, and I don’t think much else is going to move after this PR. I think only code that shouldn’t be used by the kernel or kernel applications should be moved out of util, not just code that isn’t used by the kernel. I agree just moving any code that isn’t used by the kernel would be a bad approach, and wrote basically the same comment as you recently in #27989 (comment), so I think we are in agreement.

  10. ryanofsky force-pushed on Dec 8, 2023
  11. ryanofsky commented at 6:23 pm on December 8, 2023: contributor
    Updated 778c0214d25e5a012c61251d592257ae19805255 -> 8b21f41335dc837b36ea863156605d092f47a0ab (pr/rmutil.2 -> pr/rmutil.3, compare) to fix clang-tidy error https://cirrus-ci.com/task/4602567305461760. Also switched to using declarations more places to avoid creating conflicts and make the PR easier to review Updated 8b21f41335dc837b36ea863156605d092f47a0ab -> 8748d63a07d23237dca41f8a5f91288ce59b1de8 (pr/rmutil.3 -> pr/rmutil.4, compare) to fix another clang-tidy error https://cirrus-ci.com/task/4950555420786688 and making other small cleanups Updated 8748d63a07d23237dca41f8a5f91288ce59b1de8 -> b23409474f6fe874a25374bb978db9e5682954e6 (pr/rmutil.4 -> pr/rmutil.5, compare) to try to fix another clang-tidy error
  12. ryanofsky force-pushed on Dec 8, 2023
  13. ryanofsky force-pushed on Dec 8, 2023
  14. DrahtBot removed the label CI failed on Dec 8, 2023
  15. hebasto commented at 11:20 am on December 10, 2023: member

    @ryanofsky

    #29015#issue-2029493410:

    the util library is a dependency of the kernel, and we should remove functionality from util that shouldn’t be called by kernel code or kernel applications.

    #29015 (comment):

    only code that shouldn’t be used by the kernel or kernel applications should be moved out of util, not just code that isn’t used by the kernel.

    #27989 (comment):

    I think util/ is a good home for general purpose code providing basic data structures and cross platform capabilities, and I think common/ is good home for project-specific code that needs to be shared between bitcoind, bitcoin-cli, and bitcoin-wallet.

    Does it make sense to improve https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md by expressing the idea from the quotes above more clearly? As for now, it reads:

    In general, if it is ever unclear whether it is better to add code to util or common, it is probably better to add it to common unless it is very generically useful or useful particularly to include in the kernel.

  16. TheCharlatan commented at 5:22 pm on December 10, 2023: contributor

    Re #29015 (comment)

    Does it make sense to improve https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md by expressing the idea from the quotes above more clearly? As for now, it reads:

    This seems like the right place to talk about the relationship between the util and kernel library. My work on the kernel library has been focusing on reducing its essence to validation code, pruning external dependencies, and stripping out as much system related functionality as possible. I think if the goal is to have the kernel be capable of running on as many platforms as possible, potentially beyond the number of platforms that Bitcoin Core is capable of today and extending into embedded environments, this last point becomes crucial.

    Looking at the content of util it provides roughly two types of functionality: Utilities for basic data structures and their serialization, and cross-platform system related utilities, like file handling, threading, process handling, etc.

    In my opinion, making the util library a proper dependency of the kernel should not introduce new cross-platform system-related content into the kernel library. Besides potentially limiting use cases of the kernel, providing cross-platform utilities beyond what the kernel strictly requires seems like a potentially unreasonable future maintenance burden. This would be contrary to our incremental process of creating a minimal kernel library.

    The asterisk here is that some of the utilities that are not currently used by the kernel do not introduce new dependencies, since they entirely rely on code that is already used by the kernel. I also generally think that some utilities could remain in the util library even though they are not currently used by validation code. The criteria I propose for evaluating if modules should remain in util after this PR are:

    1. The module is directly used by kernel code.
    2. The module contains basic data structure utilities for interacting with kernel code.
    3. The module does not contain system-related utilities that are not directly used by the kernel.
    4. The module does not contain project-specific code that is not relevant to the kernel.
    5. The module does not introduce new dependencies beyond the current dependencies of the kernel.

    Looking at @hebasto’s comment here, threadinterrupt.cpp and readwritefile.cpp would violate the 3rd criteria in my eyes, while the spanparsing and bytevectorhash modules could potentially remain in util.

    Between this PR and #28690 I think it would be nice if common criteria for util would be defined and documented. All this said, I think it would be acceptable too if things were kept purposefully vague as they are now until the need for more concrete structures arises.

  17. ajtowns commented at 2:00 am on December 11, 2023: contributor

    Would it make sense to just merge common into util; but have a “stripped” version of util available for the kernel, that excludes stuff that doesn’t match the 5 points above? That way the difference is just at the build system level, it doesn’t involve moving files around or moving code between namespaces…

    Then the advice would be: (a) just put things in util, (b) but only add things to the “util-core” section of the build file when they’re needed by the kernel; (c) keep things in the util:: namespace; (d) split stuff into different .h/.cpp files pretty aggressively?

    That would imply a different approach for the util: move fees.h and error.h to common/messages.h commit here.

  18. ryanofsky commented at 4:37 pm on December 11, 2023: contributor

    re: #29015 (comment)

    Would it make sense to just merge common into util; but have a “stripped” version of util available for the kernel

    Idea to build util and util-core (or util-lite?) libraries is interesting, because then it would be easy to add/remove things from util library depending on whether the kernel needs them, and it would never require renaming anything. I could see this being useful in the future if util library started growing a lot and accumulating more features of over time.

    But for now, IMO, just sticking with the current util/common separation and using the cleanup in this PR would organize things better and be less confusing than having two util versions, and it would require moving fewer things. Even though the libraries.md document is currently ambiguous, I think it should be pretty easy to remove the ambiguity here.

    re: #29015 (comment)

    The criteria I propose for evaluating if modules should remain in util after this PR are:

    I think these are good rules if you replace “basic data structure utilities” with “basic utilities that to do not require outside dependencies”, and also drop the 3rd rule. It seems like with the 3rd rule you are trying to make a distinction between “data structure utilities” and “system-related utilities”, and I don’t think it helps to draw a dividing line there.

  19. ryanofsky commented at 6:17 pm on December 11, 2023: contributor

    Added 1 commits b23409474f6fe874a25374bb978db9e5682954e6 -> 6d8cb4d4cd6062ad06cd351439e9139e028995d3 (pr/rmutil.5 -> pr/rmutil.6, compare) updating libraries.md to describe differences between util and common libraries better.

    re: #29015 (comment)

    My work on the kernel library has been focusing on reducing its essence to validation code, pruning external dependencies, and stripping out as much system related functionality as possible. […] In my opinion, making the util library a proper dependency of the kernel should not introduce new cross-platform system-related content into the kernel library.

    Just to address these points directly, I think the goals of making the kernel portable and working on any platform, and making the kernel minimal working and in constrained environments are great goals. I just don’t you need to rm src/util/sock.cpp to achieve these goals. As long as the util library is distributed as a static library, or built as a static library and linked into a dynamic library, the socket functions will not be part of resulting executables if they are not called. And if the kernel is being built for a platform which doesn’t support sockets at all, src/util/sock.cpp can just be stubbed or not compiled at all and there should be no problem for portability either. I do think it’s good to keep the util library small and lightweight and remove things which don’t fit there. But I don’t think it needs to be stripped down in a more extreme way.

  20. TheCharlatan commented at 8:40 pm on December 11, 2023: contributor

    Re #29015 (comment)

    And if the kernel is being built for a platform which doesn’t support sockets at all, src/util/sock.cpp can just be stubbed or not compiled at all and there should be no problem for portability either.

    One of the guiding principles for the kernel project so far has been to prohibit future re-couplings of decoupled modules. The tracking issue says “Any further coupling of our consensus engine with non-consensus modules will result in linker errors, preventing this project from becoming a Sisyphean task of battling coupling regressions.”. How could this fit in with leaving unused content in the library?

  21. ryanofsky commented at 10:08 pm on December 11, 2023: contributor

    One of the guiding principles for the kernel project so far has been to prohibit future re-couplings of decoupled modules.

    I reread the issue you linked to, but I don’t understand what it implies here. I do think you can rely on linker errors to ensure kernel code isn’t calling wallet code or GUI code or P2P code or storing state in global variables. But what else do you want beyond that? And how does removing sock.cpp from the util library help, for example?

  22. in doc/design/libraries.md:91 in 6d8cb4d4cd outdated
    87@@ -87,14 +88,15 @@ class bitcoin-qt,bitcoind,bitcoin-cli,bitcoin-wallet bold
    88 
    89 - *libbitcoin_consensus* should be a standalone dependency that any library can depend on, and it should not depend on any other libraries itself.
    90 
    91-- *libbitcoin_util* should also be a standalone dependency that any library can depend on, and it should not depend on other internal libraries.
    92+- *libbitcoin_crypto* should be a standalone dependency that any library can depend on, and it should not depend on any other libraries itself. (It is not shown in diagram above to save space, but has the exact same connections as *libbitcoin_consensus*, which is shown.)
    


    ajtowns commented at 11:01 pm on December 11, 2023:
    Doesn’t libconsensus necessarily depend on libcrypto (for sha256 at least)?

    ryanofsky commented at 5:19 pm on December 12, 2023:

    re: #29015 (review)

    Doesn’t libconsensus necessarily depend on libcrypto (for sha256 at least)?

    Yes that should be right. I think I just deluded myself because I didn’t want to update the mermaid diagram. Updated now though.

  23. in src/node/types.h:17 in 529be13d20 outdated
    11+//! files.
    12+
    13+#ifndef BITCOIN_NODE_TYPES_H
    14+#define BITCOIN_NODE_TYPES_H
    15+
    16+enum class TransactionError {
    


    ajtowns commented at 11:16 pm on December 11, 2023:

    It seems odd to put this in “node” when TransactionErrorString(TransactionError error) is in “common” ?

    TransactionError is returned by CombinePSBTs() which seems like something that could reasonably be exposed by a command line tool without needing a full node setup.

    So putting this in common seems more sensible to me?


    ryanofsky commented at 5:20 pm on December 12, 2023:

    re: #29015 (review)

    It seems odd to put this in “node” when TransactionErrorString(TransactionError error) is in “common” ?

    TransactionError is returned by CombinePSBTs() which seems like something that could reasonably be exposed by a command line tool without needing a full node setup.

    So putting this in common seems more sensible to me?

    This is sort of a borderline case, and I did consider whether to move it in to common instead of node. The main place where TransactionError codes are actually used, and the place where the type was introduced in #14978 was in node/transaction.h and node/transaction.cpp for the BroadcastTransaction function. But disappointingly I think, the same type was also reused in other places like wallet code and psbt code, with extra enum values added to support these. I think it would be better if the enum were split up so it would be clearer which functions returned which values, so for example, you would not be goaded by compiler into writing error handling code for enum values that could never be returned.

    With this background, I thought for this PR putting TransactionError in node instead of common would be better so it would be more likely to be appropriately used for functions like node::BroadcastTransaction and interfaces::Node::broadTransaction, and less likely to be misused for non-node functions. I could understand the opposite argument, though, that if even if it belongs in node when used ideally, it makes more sense for it to be in common based on the way it is used now.

    More generally, I do think it’s good for public node types to be defined in the node library, even if they may be also used by outside code (such as as GUI code or code in src/interfaces/ or src/common/). It makes the outside code clearer and more self documenting if you can see which libraries types are coming from. CNodeStats, CNodeStateStats, and SynchronizationState are examples of other node types which are used by the GUI outside the node library. AddressPurpose, isminetype, CRecipient, and CCoinControl are examples of wallet types used by the GUI outside the wallet library, and I think in both cases it is helpful for these types to have node:: and wallet:: prefixes when used by outside code.

    But this is just to explain my reasoning. I wouldn’t oppose moving the type to common if you think that is better. I will also look into splitting up the TransactionError type to see if there’s an easy way to clean this up completely.


    ajtowns commented at 8:40 am on December 13, 2023:

    With this background, I thought for this PR putting TransactionError in node instead of common would be better so it would be more likely to be appropriately used for functions like node::BroadcastTransaction and interfaces::Node::broadTransaction, and less likely to be misused for non-node functions. I could understand the opposite argument, though, that if even if it belongs in node when used ideally, it makes more sense for it to be in common based on the way it is used now.

    I was mostly thinking that it doesn’t make sense for TransactionErrorString() in common even though it depends on TransactionError which is in node, and node depends on common, not vice-versa? I wasn’t expecting paragraphs of response :) I mean, it’s just an enum with associated translatable descriptions? Doesn’t really seem worth worrying about the fact that some of the errors only make sense in a node context; aren’t these all just “something went wrong, deal with it, dear user”?

    BTW, I think TransactionError::P2P_DISABLED can be dropped entirely? (I think the last/only use was removed in #15713)


    ryanofsky commented at 6:24 pm on December 13, 2023:

    re: #29015 (review)

    I was mostly thinking that it doesn’t make sense for TransactionErrorString() in common even though it depends on TransactionError which is in node, and node depends on common, not vice-versa? I wasn’t expecting paragraphs of response :)

    Yeah sorry thanks for pinning down the actual concern. Before you just said it “seems odd” so I felt like I needed to justify my entire thought process.

    In any case, both the node and wallet libraries expose public types which I listed above (CNodeStats, CNodeStateStats, and SynchronizationState, AddressPurpose, isminetype, CRecipient, and CCoinControl) which can be considered part of the interfaces to these libraries, but are not part of the libraries themselves. For example, the GUI code is not allowed to call node or wallet functions, but it uses all of these public types, and this is fine. Other code, including code in common, can also use these types and there is no problem with this.

    We do have a choice where about where to put public types which are not really part of any library. We can put them in the interfaces directory, or in common, or in the node and wallet directories. Personally, I think it makes sense to put them in the wallet directory when they are related to wallet functionality and used by wallet code, put them in the node directory when they are related to node functionality and used by node code, put them in the common directory when they are related to common functionality (like PSBT) and used by common code. The comments at the top of {node,wallet,common}/types.h files are meant to explain the idea that these are public types.

    Doesn’t really seem worth worrying about the fact that some of the errors only make sense in a node context; aren’t these all just “something went wrong, deal with it, dear user”?

    Yes if errors are just being propagated to the user level, you don’t care about this at all, and it’s probably a mistake to use an enum to begin with because the error messages won’t be able to contain useful context. In this case, though, the errors are being translated to RPC codes, and RPC callers may actually want to implement different forms of error handling, so it would be good to be clear about what errors are possible, plus it’s easy to do, so I did add a new commit splitting PSBTError from TransactionError.

    BTW, I think TransactionError::P2P_DISABLED can be dropped entirely? (I think the last/only use was removed in #15713)

    Thanks, the new commit drops this code too.

  24. ajtowns commented at 11:26 pm on December 11, 2023: contributor

    Okay, I think I’m on roughly the same page:

    • keeping Split in util make sense
    • moving our string stuff into the util namespace seems fine * moving the spanparsing stuff to script mostly makes sense – though I feel a bit like mixing consensus critical script manipulation and miniscript/descriptor parsing in the same namespace is a bit clunky?
    • moving fees.h/error.h to “common” makes sense; though if we can make “common” just be a broader part of “util” that seems better long term.
    • moving TransactionError into node rather than common doesn’t make sense to me though
  25. TheCharlatan commented at 8:20 am on December 12, 2023: contributor

    Re #29015 (comment)

    But what else do you want beyond that? And how does removing sock.cpp from the util library help, for example?

    Maybe removing sock.cpp is a bad example, since it is already impossible to link against it. sock.cpp uses modules that are already not part of the kernel, so trying to link it in will result in errors. Then again, if sock.cpp uses common modules, why is it in util in the first place? To provide another concrete example, there is thread.cpp, which I was hoping to remove from the kernel in #28960. If it is removed, how could the kernel library be guarded from its future re-introduction if it remains in the util library?

  26. ryanofsky force-pushed on Dec 12, 2023
  27. ryanofsky commented at 7:38 pm on December 12, 2023: contributor

    Updated 6d8cb4d4cd6062ad06cd351439e9139e028995d3 -> 9ba4dc41fd405a008e9badc4092ef1905f534b96 (pr/rmutil.6 -> pr/rmutil.7, compare) fixing crypto descriptions in libraries.md

    re: #29015#pullrequestreview-1776320307

    • moving the spanparsing stuff to script mostly makes sense – though I feel a bit like mixing consensus critical script manipulation and miniscript/descriptor parsing in the same namespace is a bit clunky?

    Yes, actually different files in src/script/ directory are built into different libraries. Script interpreting files are part of the consensus library, script generating files are part of the common library, and the script/sigcache file is part of the node library. It might make sense to untangle this in some way.

    • moving fees.h/error.h to “common” makes sense; though if we can make “common” just be a broader part of “util” that seems better long term.

    I think I probably agree. It makes sense to me if we are distributing external libraries like consensus & kernel that we should have public utilities which external code can use, and private utilities external code shouldn’t use. But names “util” and “common” probably aren’t the best way to describe these.

  28. ryanofsky commented at 7:38 pm on December 12, 2023: contributor

    re: #29015 (comment)

    Maybe removing sock.cpp is a bad example, since it is already impossible to link against it. sock.cpp uses modules that are already not part of the kernel, so trying to link it in will result in errors. Then again, if sock.cpp uses common modules, why is it in util in the first place?

    I’m actually not sure what is this referring to. Running nm ./src/util/libbitcoin_util_a-sock.o --undefined-only I see sock.cpp is using LogInstance, NodeClock, and SysErrorString which should all be part of the util library.

    To provide another concrete example, there is thread.cpp, which I was hoping to remove from the kernel in #28960. If it is removed, how could the kernel library be guarded from its future re-introduction if it remains in the util library?

    thread.h contains TraceThread which seems like a nice utility for the naming threads (which is helpful for monitoring in top) and printing exceptions (which is useful for debugging). I guess kernel code isn’t currently using this function (or #28960 removes the last usage), but as long as kernel code is calling threads, what benefit is to preventing kernel code from calling this function? It seems like a useful function, and if anything more code should use it, I would think.

    I do agree with goals of making the kernel small and portable, and agree we should organize libraries with that in mind. I also think we should try to trigger link errors to enforce API boundaries. It just seems to me we can do these things without moving a lot around.

  29. TheCharlatan commented at 7:55 pm on December 12, 2023: contributor

    Re #29015 (comment)

    I’m actually not sure what is this referring to. Running nm ./src/util/libbitcoin_util_a-sock.o –undefined-only I see sock.cpp is using LogInstance, NodeClock, and SysErrorString which should all be part of the util library.

    Thanks for actually verifying, I just assumed that the common/system.hinclude was actually used :(.

  30. ryanofsky force-pushed on Dec 13, 2023
  31. ryanofsky commented at 6:40 pm on December 13, 2023: contributor
    Updated 9ba4dc41fd405a008e9badc4092ef1905f534b96 -> 414546dcc455fb5f5f8e2c4591b2df1f16863cdc (pr/rmutil.7 -> pr/rmutil.8, compare) adding new commit 18a9f203bba1ad551bdea0d0580b3e07c3715054 to break up TransactionError type and remove PSBT codes Updated 414546dcc455fb5f5f8e2c4591b2df1f16863cdc -> 7ef33b2bad71a230428b653d6d0297df49630d99 (pr/rmutil.8 -> pr/rmutil.9, compare) to fix PSBT unit test error and clang-tidy errors
  32. DrahtBot added the label CI failed on Dec 13, 2023
  33. ryanofsky force-pushed on Dec 19, 2023
  34. DrahtBot removed the label CI failed on Dec 19, 2023
  35. DrahtBot added the label CI failed on Jan 16, 2024
  36. Sjors commented at 1:27 pm on February 5, 2024: member
    Concept ACK. Moving things out of util that the kernel will / should never need, makes sense to me. I don’t have strong feelings on where exactly to move these things to.
  37. TheCharlatan commented at 10:32 am on February 11, 2024: contributor
    Concept ACK
  38. in src/common/types.h:7 in b8900adcd2 outdated
    0@@ -0,0 +1,26 @@
    1+// Copyright (c) 2010-2021 The Bitcoin Core developers
    2+// Distributed under the MIT software license, see the accompanying
    3+// file COPYING or http://www.opensource.org/licenses/mit-license.php.
    4+
    5+//! @file common/types.h is a home for simple enum and struct type definitions
    6+//! that can be used internally by functions in the libbitcoin_common library,
    7+//! but also used externally by node, wallet, wallet and GUI code.
    


    TheCharlatan commented at 3:10 pm on February 11, 2024:
    There is one wallet too many here.

    ryanofsky commented at 7:31 pm on February 20, 2024:

    re: #29015 (review)

    There is one wallet too many here.

    Good catch, fixed

  39. in src/node/types.h:20 in b8900adcd2 outdated
    16@@ -17,16 +17,10 @@ enum class TransactionError {
    17     OK, //!< No error
    18     MISSING_INPUTS,
    19     ALREADY_IN_CHAIN,
    20-    P2P_DISABLED,
    


    TheCharlatan commented at 12:10 pm on February 12, 2024:
    In commit b8900adcd232bcb2b7475be19b5457c91052ab5a: Can you either add a line about its removal in the commit message, or do it in a separate commit?

    ryanofsky commented at 3:35 pm on February 21, 2024:

    re: #29015 (review)

    In commit b8900ad: Can you either add a line about its removal in the commit message, or do it in a separate commit?

    Thanks, I added a note about dropping unused error codes in the commit message

  40. in src/psbt.h:1266 in b8900adcd2 outdated
    1264@@ -1265,7 +1265,7 @@ bool FinalizeAndExtractPSBT(PartiallySignedTransaction& psbtx, CMutableTransacti
    1265  * @param[in]  psbtxs the PSBTs to combine
    1266  * @return error (OK if we successfully combined the transactions, other error if they were not compatible)
    


    TheCharlatan commented at 12:18 pm on February 12, 2024:
    In commit b8900adcd232bcb2b7475be19b5457c91052ab5a: Need to slightly adjust the comment here, how about: * [@return](/bitcoin-bitcoin/contributor/return/) true if we successfully combined the transactions, false if they were not compatible

    ryanofsky commented at 7:31 pm on February 20, 2024:

    re: #29015 (review)

    In commit b8900ad: Need to slightly adjust the comment here, how about: * [@return](/bitcoin-bitcoin/contributor/return/) true if we successfully combined the transactions, false if they were not compatible

    Thanks, used suggestion and tweaked it to match other bool returns in this file

  41. hebasto commented at 1:16 pm on February 15, 2024: member
    Approach ACK 7ef33b2bad71a230428b653d6d0297df49630d99.
  42. ryanofsky force-pushed on Feb 21, 2024
  43. ryanofsky commented at 3:49 pm on February 21, 2024: contributor

    Thanks for the reviews!

    Updated 7ef33b2bad71a230428b653d6d0297df49630d99 -> 486379c10fdce568a64b78cbb2dad1fa52263a79 (pr/rmutil.9 -> pr/rmutil.10, compare) with suggestions

  44. DrahtBot removed the label CI failed on Feb 21, 2024
  45. DrahtBot added the label Needs rebase on Mar 9, 2024
  46. ryanofsky force-pushed on Mar 11, 2024
  47. ryanofsky commented at 5:22 pm on March 11, 2024: contributor
    Rebased 486379c10fdce568a64b78cbb2dad1fa52263a79 -> ffcd3068359ea35f3b7c491df866a0d6924e3b37 (pr/rmutil.10 -> pr/rmutil.11, compare) due to conflict with #28960
  48. DrahtBot removed the label Needs rebase on Mar 11, 2024
  49. TheCharlatan approved
  50. TheCharlatan commented at 9:52 am on March 13, 2024: contributor

    ACK ffcd3068359ea35f3b7c491df866a0d6924e3b37

    Regarding the previous discussions on possibly further discriminating the contents of /util, this can always be revisited at a later point in time. Much more important to keep things moving in the correct direction than getting this immediately correct for all possible (and currently unforeseeable) kernel use cases.

  51. DrahtBot requested review from Sjors on Mar 13, 2024
  52. DrahtBot requested review from hebasto on Mar 13, 2024
  53. DrahtBot added the label Needs rebase on Mar 13, 2024
  54. ryanofsky force-pushed on Mar 27, 2024
  55. DrahtBot removed the label Needs rebase on Mar 27, 2024
  56. TheCharlatan approved
  57. TheCharlatan commented at 9:21 am on March 30, 2024: contributor
    Re-ACK 2d9686fa3052ddc52bc129c7ecc8495a5b6c0688
  58. DrahtBot added the label Needs rebase on Apr 1, 2024
  59. ryanofsky force-pushed on Apr 18, 2024
  60. DrahtBot removed the label Needs rebase on Apr 18, 2024
  61. hebasto added the label Needs CMake port on Apr 29, 2024
  62. hebasto commented at 4:45 pm on April 29, 2024: member

    I’ve tested 5eeafeb302dcf56dd5c16c1e4785a41a132344c7 on Ubuntu 23.10.

    While all changes look correct and reasonable to me, I was expecting that this PR removes some of dependencies mentioned in #28548.

    But that is not the case. The libbitcoin_util still depends on libbitcoin_crypto, libbitcoin_consensus and libbitcoin_common libraries.

    If such an outcome is expected, maybe update the doc/design/libraries.md accordingly?

  63. DrahtBot added the label Needs rebase on Apr 30, 2024
  64. TheCharlatan approved
  65. TheCharlatan commented at 8:57 am on May 6, 2024: contributor
    Re-ACK 5eeafeb302dcf56dd5c16c1e4785a41a132344c7
  66. TheCharlatan commented at 9:05 am on May 6, 2024: contributor

    Re #29015 (comment)

    But that is not the case. The libbitcoin_util still depends on libbitcoin_crypto, libbitcoin_consensus and libbitcoin_common libraries. If such an outcome is expected, maybe update the doc/design/libraries.md accordingly?

    I understand this PR more as providing clarity of what should be in the util library and #28690 as actually realizing the diagram.

  67. ryanofsky force-pushed on May 10, 2024
  68. ryanofsky commented at 9:17 pm on May 10, 2024: contributor

    re: #29015 (comment)

    The libbitcoin_util still depends on libbitcoin_crypto, libbitcoin_consensus and libbitcoin_common libraries.

    Thanks for checking this. Letting the util library depend on the crypto library seems like the easiest path to take because random.cpp, randomenv.cpp, hasher.cpp, and bytevectorhash.cpp are using the crypto hash functions. But I added new commits to the PR to remove the util dependency on crypt, and remove consensus and common dependencies on util. I also added a script which turned up a few other other unexpected dependencies (not related to util), which could be addressed separately. I think the script should be able to run in CI, too, but I did not set that up.


    Rebased 5eeafeb302dcf56dd5c16c1e4785a41a132344c7 -> 6ea5ee5a268cfbb9cd234e0f1381c8a4785ff04c (pr/rmutil.13 -> pr/rmutil.14, compare) adding dependency checking script and removing unexpected dependencies related to util

  69. DrahtBot added the label CI failed on May 10, 2024
  70. DrahtBot commented at 9:47 pm on May 10, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/24838365077

  71. hebasto commented at 5:56 pm on May 11, 2024: member

    I also added a script which turned up a few other other unexpected dependencies (not related to util), which could be addressed separately.

    It would be great to have such a tool at our disposal.

  72. TheCharlatan commented at 12:31 pm on May 12, 2024: contributor

    There is an error in commit fd342cb40b63ca00d23946743a038847673f8726, which can be fixed with:

    0diff --git a/src/warnings.cpp b/src/warnings.cpp
    1index 5b6ace63dd..40043cd8e7 100644
    2--- a/src/warnings.cpp
    3+++ b/src/warnings.cpp
    4@@ -11,0 +12 @@
    5+#include <util/string.h>
    
  73. TheCharlatan commented at 12:54 pm on May 12, 2024: contributor
    The check script is very useful, could you add it as the first commit? I think it could make review a bit easier.
  74. DrahtBot removed the label Needs rebase on May 13, 2024
  75. ryanofsky force-pushed on May 13, 2024
  76. ryanofsky commented at 2:27 pm on May 13, 2024: contributor
    Updated 6ea5ee5a268cfbb9cd234e0f1381c8a4785ff04c -> 43532d9c9470ad00ecb991859f7f0c6a025fa72e (pr/rmutil.14 -> pr/rmutil.15, compare) fixing silent conflict with #29845, fixing shellcheck lint errors, and moving check-deps.sh commit first as suggested so it is easier to see effects of later commits.
  77. hebasto commented at 6:53 pm on May 13, 2024: member

    There is another CI failure in https://cirrus-ci.com/task/6531594385620992:

    0crypto/.libs/libbitcoin_crypto_base.a(crypto_libbitcoin_crypto_base_la-cleanse.o): in function `memory_cleanse(void*, unsigned long)':
    1cleanse.cpp:(.text+0x0): multiple definition of `memory_cleanse(void*, unsigned long)'; /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x22
    2support/.libs/libbitcoinkernel_la-cleanse.o:cleanse.cpp:(.text+0x0): first defined here
    3clang: error: linker command failed with exit code 1 (use -v to see invocation)
    
  78. ryanofsky force-pushed on May 13, 2024
  79. DrahtBot removed the label CI failed on May 14, 2024
  80. in test/util/check-deps.sh:1 in 850c73e250 outdated
    0@@ -0,0 +1,203 @@
    1+#!/usr/bin/env bash
    


    TheCharlatan commented at 11:16 am on May 14, 2024:
    I think rather than test/ this should go into contrib/devtools, which contains functionally similar scripts like symbol-check, or security-check.

    ryanofsky commented at 3:08 pm on May 16, 2024:

    re: #29015 (review)

    I think rather than test/ this should go into contrib/devtools, which contains functionally similar scripts like symbol-check, or security-check.

    Good suggestion, it does fit in more with those tools, moved over there.

  81. in doc/design/libraries.md:58 in 850c73e250 outdated
    53@@ -53,21 +54,30 @@ bitcoin-wallet[bitcoin-wallet]-->libbitcoin_wallet_tool;
    54 libbitcoin_cli-->libbitcoin_util;
    55 libbitcoin_cli-->libbitcoin_common;
    56 
    57+libbitcoin_consensus-->libbitcoin_crypto;
    58+libbitcoin_consensus-->libbitcoin_crypto;
    


    hebasto commented at 2:11 pm on May 14, 2024:
    Duplicated lines?

    ryanofsky commented at 3:51 pm on May 14, 2024:

    re: #29015 (review)

    Duplicated lines?

    Thanks, cleaned up

  82. in doc/design/libraries.md:91 in 850c73e250 outdated
    87@@ -78,22 +88,23 @@ class bitcoin-qt,bitcoind,bitcoin-cli,bitcoin-wallet bold
    88 ```
    89 </td></tr><tr><td>
    90 
    91-**Dependency graph**. Arrows show linker symbol dependencies. *Consensus* lib depends on nothing. *Util* lib is depended on by everything. *Kernel* lib depends only on consensus and util.
    92+**Dependency graph**. Arrows show linker symbol dependencies. *Crypto* lib depends on nothing. *Util* lib is depended on by everything. *Kernel* lib depends only on consensus and util.
    


    hebasto commented at 2:12 pm on May 14, 2024:
    0**Dependency graph**. Arrows show linker symbol dependencies. *Crypto* lib depends on nothing. *Util* lib is depended on by everything. *Kernel* lib depends only on consensus, crypto and util.
    

    ryanofsky commented at 3:51 pm on May 14, 2024:

    re: #29015 (review)

    Good catch, added

  83. in test/util/check-deps.sh:36 in 850c73e250 outdated
    31+    "util crypto"
    32+    "wallet common"
    33+    "wallet crypto"
    34+    "wallet_tool util"
    35+    "wallet_tool wallet"
    36+    "wallet util"
    


    hebasto commented at 2:13 pm on May 14, 2024:
    nit: Sort these lines?

    ryanofsky commented at 3:51 pm on May 14, 2024:

    re: #29015 (review)

    nit: Sort these lines?

    Fixed, thanks. (I actually did sort them, but it seems sort command treats underscore and space characters the same on my system so the wallet and wallet_util dependencies were mixed up)

  84. hebasto approved
  85. hebasto commented at 2:16 pm on May 14, 2024: member

    ACK 850c73e250fe0b248cae80d26561da7047696e3d.

    Thanks for the test/util/check-deps.sh script!

  86. DrahtBot requested review from TheCharlatan on May 14, 2024
  87. ryanofsky force-pushed on May 14, 2024
  88. hebasto approved
  89. hebasto commented at 4:07 pm on May 14, 2024: member
    re-ACK b58156701ac132f87b8ef8da1c7d22158c804a81, only suggested changes since my recent review.
  90. ryanofsky commented at 4:07 pm on May 14, 2024: contributor

    Thanks for the review! Made suggested changes

    Updated 850c73e250fe0b248cae80d26561da7047696e3d -> b58156701ac132f87b8ef8da1c7d22158c804a81 (pr/rmutil.16 -> pr/rmutil.17, compare) with suggested changes (documentation fixes, sorting)

  91. TheCharlatan approved
  92. TheCharlatan commented at 1:44 pm on May 16, 2024: contributor

    Re-ACK b58156701ac132f87b8ef8da1c7d22158c804a81

    Good improvements since my last review and having some form of external verification for the work being done is very nice. If other reviewers or maintainers are not happy with introducing a bash script, I could volunteer some time to translate it.

    The includes could be cleaned up a bit, especially util/string.h and util/strencodings.h are missing in a bunch of places. It would also be nice if the newly introduced files had correct includes. Maybe this PR could also be used as a opportunity to more aggressively clean up includes in general.

  93. test: Add check-deps.sh script to check for unexpected library dependencies ffa27af24d
  94. build: move chainparamsbase from util to common
    Move chainparamsbase from util to common, because util library should not
    depend on the common library and chainparamsbase uses the ArgsManager class in
    common.
    5b9309420c
  95. build: move memory_cleanse from util to crypto
    Move memory_cleanse from util to crypto because the crypto library should not
    depend on other libraries, and it calls memory_cleanse.
    cc5f29fbea
  96. util: move util/message to common/signmessage
    Move util/message to common/signmessage so it is named more clearly, and
    because the util library is not supposed to depend on other libraries besides
    the crypto library. The signmessage functions use CKey, CPubKey, PKHash, and
    DecodeDestination functions in the consensus and common libraries.
    6861f954f8
  97. util: move HexStr and HexDigit from util to crypto
    Move HexStr and HexDigit functions from util to crypto. The crypto library does
    not actually use these functions, but the consensus library does. The consensus
    and util libraries not allowed to depend on each other, but are allowed to
    depend on the cryto library, so the crypto library is a reasonable put these.
    
    The consensus library uses HexStr and HexDigit in script.cpp, transaction.cpp,
    and uint256.cpp.
    
    The util library does not use HexStr but does use HexDigit in strencodings.cpp
    to parse integers.
    23cc8ddff4
  98. util: move spanparsing.h Split functions to string.h
    This will help move the miniscript / descriptor parsing functions out of the
    util library in an upcoming commit, so they are not exposed to libbitcoinkernel
    applications. Moving the Split functions should also make them more
    discoverable since they now close to related functions like Join.
    
    The functions are moved verbatim without any changes.
    6dd2ad4792
  99. util: move spanparsing.h to script/parsing.h
    Move miniscript / descriptor script parsing functions out of util library so
    they are not a dependency of the kernel.
    
    There are no changes to code or behavior.
    9bcce2608d
  100. util: move error.h TransactionError enum to node/types.h
    New node/types.h file is analagous to existing wallet/types.h and is a better
    place to define simple node types that are shared externally with wallet and
    gui code than the util library.
    
    Motivation for this change is to completely remove util/error.h file currently
    holding TransactionError in a followup commit.
    0d44c44ae3
  101. common: Add PSBTError enum
    Add separate PSBTError enum instead of reusing TransactionError enum for PSBT
    operations, and drop unused error codes. The error codes returned by PSBT
    operations and transaction broadcast functions mostly do not overlap, so using
    an unified enum makes it harder to call any of these functions and know which
    errors actually need to be handled.
    
    Define PSBTError in the common library because PSBT functionality is
    implemented in the common library and used by both the node (for rawtransaction
    RPCs) and the wallet.
    02e62c6c9a
  102. util: move fees.h and error.h to common/messages.h
    Move enum and message formatting functions to a common/messages header where
    they should be more discoverable, and also out of the util library, so they
    will not be a dependency of the kernel
    
    The are no changes in behavior and no changes to the moved code.
    680eafdc74
  103. util: add TransactionError includes and namespace declarations
    Add TransactionError to node namespace and include it directly instead of
    relying on indirect include through common/messages.h
    
    This is a followup to a previous commit which moved the TransactionError enum.
    These changes were done in a separate followup just to keep the previous commit
    more minimal and easy to review.
    4d05d3f3b4
  104. util: Move util/string.h functions to util namespace
    There are no changes to behavior. Changes in this commit are all additions, and
    are easiest to review using "git diff -U0 --word-diff-regex=." options.
    
    Motivation for this change is to keep util functions with really generic names
    like "Split" and "Join" out of the global namespace so it is easier to see
    where these functions are defined, and so they don't interfere with function
    overloading, especially since the util library is a dependency of the kernel
    library and intended to be used with external code.
    4f74c59334
  105. doc: Clarify distinction between util and common libraries in libraries.md
    Also describe crypto library which was previously unmentioned
    c7376babd1
  106. ryanofsky force-pushed on May 16, 2024
  107. ryanofsky commented at 3:25 pm on May 16, 2024: contributor

    Updated b58156701ac132f87b8ef8da1c7d22158c804a81 -> 7689dfd6646054a0be8e62ccbf72d5403e28b548 (pr/rmutil.17 -> pr/rmutil.18, compare) moving check-deps script.


    re: #29015#pullrequestreview-2055052953

    The includes could be cleaned up a bit, especially util/string.h and util/strencodings.h are missing in a bunch of places. It would also be nice if the newly introduced files had correct includes. Maybe this PR could also be used as a opportunity to more aggressively clean up includes in general.

    Agree this would be nice. The only issue is I never figured out a good way to run the IWYU tool that didn’t seem to add lot of unrelated changes to PRs, given that existing includes are so messy, and there are so many that could be cleaned up. If you have any advice on how to run IWYU with this PR, I’d be happy to follow. Or you already have a clear idea of how to clean up includes and just want implement changes on a branch, I’d be happy to use that.

  108. DrahtBot added the label CI failed on May 16, 2024
  109. DrahtBot commented at 5:49 pm on May 16, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/25060059418

  110. hebasto commented at 8:03 pm on May 16, 2024: member
    There seems to be a conflict with bitcoin/bitcoin#30098. Maybe rebase?
  111. ryanofsky force-pushed on May 17, 2024
  112. ryanofsky commented at 1:50 am on May 17, 2024: contributor

    There seems to be a conflict with #30098. Maybe rebase?

    Thanks! Rebased now.

    If other reviewers or maintainers are not happy with introducing a bash script, I could volunteer some time to translate it.

    Yes this could be a good idea. The script started out just being a few lines of bash listing symbols exported by one library and imported by another one. But then it grew as it was extended to support multiple libraries and suppress known errors. So now it’s no longer a small script, though it is pretty clean and well commented. Other possible followups besides translating the script could be:

    • Running the script as part of CI
    • Cleaning up unexpected dependencies listed as suppressions in the script.

    Rebased 7689dfd6646054a0be8e62ccbf72d5403e28b548 -> 1c26d10b23bb7a7d0204b4a8b44a50fad89f2a2a (pr/rmutil.18 -> pr/rmutil.19, compare) due to silent conflict with #30098

  113. DrahtBot removed the label CI failed on May 17, 2024
  114. TheCharlatan approved
  115. TheCharlatan commented at 8:33 am on May 17, 2024: contributor
    Re-ACK 1c26d10b23bb7a7d0204b4a8b44a50fad89f2a2a
  116. DrahtBot requested review from hebasto on May 17, 2024
  117. hebasto approved
  118. hebasto commented at 9:12 am on May 17, 2024: member
    re-ACK 1c26d10b23bb7a7d0204b4a8b44a50fad89f2a2a.
  119. maflcko removed the label Validation on May 17, 2024
  120. maflcko added the label Build system on May 17, 2024
  121. maflcko added the label DrahtBot Guix build requested on May 17, 2024
  122. hebasto commented at 12:21 pm on May 21, 2024: member

    My Guix build:

     007318f5568a25cf18d6ec29d8fbaba3adc9ddfa23891fbb62d63e136b5d25c7f  guix-build-1c26d10b23bb/output/aarch64-linux-gnu/SHA256SUMS.part
     1acbc7851322d559b96cca83764ef32f6e7df0f2449b1ca951a54127efea8eaf3  guix-build-1c26d10b23bb/output/aarch64-linux-gnu/bitcoin-1c26d10b23bb-aarch64-linux-gnu-debug.tar.gz
     2c94cdf25eab02a297f5b25fed2211cf0b35f2b1b6c7d48b154f2ed46720b9abb  guix-build-1c26d10b23bb/output/aarch64-linux-gnu/bitcoin-1c26d10b23bb-aarch64-linux-gnu.tar.gz
     32a7aca0c44b234af3f4395591e78e41e06309c9b9a8a9f5e8f68d248b7a45162  guix-build-1c26d10b23bb/output/arm-linux-gnueabihf/SHA256SUMS.part
     4a741202ad3a1df852035111b20a102fa72b411245b735f330826d908490fdd64  guix-build-1c26d10b23bb/output/arm-linux-gnueabihf/bitcoin-1c26d10b23bb-arm-linux-gnueabihf-debug.tar.gz
     5068a9b730c1e59449e9b2d52e87df8009fdd33e210c3b02e52d4ec677b9c5f78  guix-build-1c26d10b23bb/output/arm-linux-gnueabihf/bitcoin-1c26d10b23bb-arm-linux-gnueabihf.tar.gz
     61fbb2927be3f4f52c46f303f71755f3af7617cdb1ca3988a0bfe7827e743b500  guix-build-1c26d10b23bb/output/arm64-apple-darwin/SHA256SUMS.part
     7ab004b44ec36a2cb1cc051eed74e4f36026719a452729a0f2ffc74c52164aded  guix-build-1c26d10b23bb/output/arm64-apple-darwin/bitcoin-1c26d10b23bb-arm64-apple-darwin-unsigned.tar.gz
     898eb503f1b9d97da2478b7e31b6a6ce54c1d9080cf06f45f5aa1de552ce01805  guix-build-1c26d10b23bb/output/arm64-apple-darwin/bitcoin-1c26d10b23bb-arm64-apple-darwin-unsigned.zip
     9b4006d722cd71fa709919e7d05f35e4970270ae8369b7e4dc6fd7ea603e73d84  guix-build-1c26d10b23bb/output/arm64-apple-darwin/bitcoin-1c26d10b23bb-arm64-apple-darwin.tar.gz
    106a099c53eb043f9cf122497ccfb3822acd4d65a04d7110c1aafbc879d3c5856c  guix-build-1c26d10b23bb/output/dist-archive/bitcoin-1c26d10b23bb.tar.gz
    118d9a43ca5f80a71d0c835d61d0beab25565229492fa4fee46e8ec178e00cb658  guix-build-1c26d10b23bb/output/powerpc64-linux-gnu/SHA256SUMS.part
    12a812d1b8de12d93cc569a89e2420ccc368c8c0ba3b715b13f6ef3fb6b1e1a535  guix-build-1c26d10b23bb/output/powerpc64-linux-gnu/bitcoin-1c26d10b23bb-powerpc64-linux-gnu-debug.tar.gz
    13e15e92cf6f4cceaed18546255bc362f52cd9ce35dbe676cb57ebee9c883e3180  guix-build-1c26d10b23bb/output/powerpc64-linux-gnu/bitcoin-1c26d10b23bb-powerpc64-linux-gnu.tar.gz
    145f874fbe6e466e0b153b082f77b051b93856ccdf212f2ebf81229ff5668656cf  guix-build-1c26d10b23bb/output/riscv64-linux-gnu/SHA256SUMS.part
    15b30c16dd59a2152130c4d04a23426c77d73af4db1e68b56f61e74c23cbe0b74e  guix-build-1c26d10b23bb/output/riscv64-linux-gnu/bitcoin-1c26d10b23bb-riscv64-linux-gnu-debug.tar.gz
    16598c942f47a3b32062503fa0a1909eba08603c2437984f681c8948a76eedecd7  guix-build-1c26d10b23bb/output/riscv64-linux-gnu/bitcoin-1c26d10b23bb-riscv64-linux-gnu.tar.gz
    17f6cbd80a3170d9bfe3df6db7ad6c45a7edf5999d56faf5881917bd2046c659b2  guix-build-1c26d10b23bb/output/x86_64-apple-darwin/SHA256SUMS.part
    1886c261ee89e1c61a95ed3b90d6547be579ead16d93f56aba5eebeb8ce282c897  guix-build-1c26d10b23bb/output/x86_64-apple-darwin/bitcoin-1c26d10b23bb-x86_64-apple-darwin-unsigned.tar.gz
    1981b44a879c6982f05a02633a939fd96083f7c59cf34429b3c9d916a32ade1dd9  guix-build-1c26d10b23bb/output/x86_64-apple-darwin/bitcoin-1c26d10b23bb-x86_64-apple-darwin-unsigned.zip
    208b76c441ca2fc11573785b2fae007fff2dcae65c8f2cd2fd517e8c4e9dacc086  guix-build-1c26d10b23bb/output/x86_64-apple-darwin/bitcoin-1c26d10b23bb-x86_64-apple-darwin.tar.gz
    21eae30d8ecc90e29913653fe41634836c36e7fea89c09c22c3b05945c2c30fbbe  guix-build-1c26d10b23bb/output/x86_64-linux-gnu/SHA256SUMS.part
    2230a74515f8078a81968fd97b723b56d6331f45cd9b104a35d73559ee99666940  guix-build-1c26d10b23bb/output/x86_64-linux-gnu/bitcoin-1c26d10b23bb-x86_64-linux-gnu-debug.tar.gz
    23bf59e96c6b5b2033a84c2bfa29b97c3972827cb6edcbc484935dd0e9a7b2c28a  guix-build-1c26d10b23bb/output/x86_64-linux-gnu/bitcoin-1c26d10b23bb-x86_64-linux-gnu.tar.gz
    240241585f8314adfc7a57bd988199bf2bd0b0cdb2f1b91b0aa9c1b03af1dfa6bd  guix-build-1c26d10b23bb/output/x86_64-w64-mingw32/SHA256SUMS.part
    2544556b4c3e7140c5d8df64b07b491daa3eef2851b23d19963eb89a48c66adb50  guix-build-1c26d10b23bb/output/x86_64-w64-mingw32/bitcoin-1c26d10b23bb-win64-debug.zip
    26eb1e6368638f754af7980dd46e3482c9430604df451e819777727b9ad0720fc3  guix-build-1c26d10b23bb/output/x86_64-w64-mingw32/bitcoin-1c26d10b23bb-win64-setup-unsigned.exe
    278e2137350f44640bbff5b97216d37673062dbb89f2b3985cb377effc75327e7f  guix-build-1c26d10b23bb/output/x86_64-w64-mingw32/bitcoin-1c26d10b23bb-win64-unsigned.tar.gz
    2818380b999dfb5eb2b6a8784335553039df4ac3c773b1869e504d22c155ae90bd  guix-build-1c26d10b23bb/output/x86_64-w64-mingw32/bitcoin-1c26d10b23bb-win64.zip
    
  123. DrahtBot commented at 2:39 pm on May 22, 2024: contributor

    Guix builds (on x86_64) [untrusted test-only build, possibly unsafe, not for production use]

    File commit a786fd2041913d82ca90b561de309421bd24e41b(master) commit d1893f68b1d2cdd0108a241142081fee65a53c15(master and this pull)
    SHA256SUMS.part 14f282b5bf7ebcd4... 6b8304534f742d40...
    *-aarch64-linux-gnu-debug.tar.gz b2364625629230d1... 7236fe91743fee41...
    *-aarch64-linux-gnu.tar.gz b05fc0561baa0966... 816af5636f1f4755...
    *-arm-linux-gnueabihf-debug.tar.gz 4545637daf89882c... 71186e35ede72b81...
    *-arm-linux-gnueabihf.tar.gz 27115f4c7e251175... 322e8aa674f6b0aa...
    *-arm64-apple-darwin-unsigned.tar.gz cf3950f9d05fb1c0... 916b86c91919cf3b...
    *-arm64-apple-darwin-unsigned.zip b1c32e004722a3b4... ae490ec387351537...
    *-arm64-apple-darwin.tar.gz f09a38acef4b141b... 6ecf573cf6708073...
    *-powerpc64-linux-gnu-debug.tar.gz 75cfe8d73120b00d... 477191c62a748459...
    *-powerpc64-linux-gnu.tar.gz d4f8c1296886fac4... f081bfeafe7e82b8...
    *-riscv64-linux-gnu-debug.tar.gz a8f56aee14e85f29... 55992dd6815ca3b5...
    *-riscv64-linux-gnu.tar.gz d8a4ae4295e6aa9f... 0d3d859cc90faa23...
    *-x86_64-apple-darwin-unsigned.tar.gz ff9ae232120c85f9... 6b0af8e6122b564c...
    *-x86_64-apple-darwin-unsigned.zip 9917159a6fd59954... 3b4bf085de74d320...
    *-x86_64-apple-darwin.tar.gz 61112274cace86c4... cee59c641c8de378...
    *-x86_64-linux-gnu-debug.tar.gz 557d05bc585a2910... 6a76262d04675700...
    *-x86_64-linux-gnu.tar.gz 4928355769a494d7... 738635ffec7a993a...
    *.tar.gz 49c2acd89613cf65... 297752c33bf58704...
    guix_build.log 88d5371b29f8bb75... a238cef1d7cbedfc...
    guix_build.log.diff b3866f990e369152...
  124. DrahtBot removed the label DrahtBot Guix build requested on May 22, 2024
  125. DrahtBot added the label CI failed on Jun 2, 2024
  126. DrahtBot commented at 5:48 pm on June 2, 2024: contributor

    🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the documentation.

    Possibly this is due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.

    Leave a comment here, if you need help tracking down a confusing failure.

    Debug: https://github.com/bitcoin/bitcoin/runs/25080673770

  127. achow101 commented at 0:00 am on June 5, 2024: member
    CI failure suggests there’s a silent merge conflict.
  128. ryanofsky force-pushed on Jun 5, 2024
  129. TheCharlatan approved
  130. TheCharlatan commented at 9:01 pm on June 5, 2024: contributor
    Re-ACK c7376babd19d0c858fef93ebd58338abd530c1f4
  131. DrahtBot requested review from hebasto on Jun 5, 2024
  132. DrahtBot removed the label CI failed on Jun 5, 2024
  133. hebasto approved
  134. hebasto commented at 11:32 am on June 6, 2024: member

    re-ACK c7376babd19d0c858fef93ebd58338abd530c1f4.

    My Guix build:

     0x86_64
     1f33ae90e6d190245d34712f544cd458f282a2c90890029dfbd88d74e8823fb50  guix-build-c7376babd19d/output/aarch64-linux-gnu/SHA256SUMS.part
     2ee444d54e0b9cebfa6ab0c91fc8b543d414a98338d46d299f3b302273139c81f  guix-build-c7376babd19d/output/aarch64-linux-gnu/bitcoin-c7376babd19d-aarch64-linux-gnu-debug.tar.gz
     3fdc7400c04fd6bfec5d0ab3148b0f4da110974ecda0859184adad29f38422f6f  guix-build-c7376babd19d/output/aarch64-linux-gnu/bitcoin-c7376babd19d-aarch64-linux-gnu.tar.gz
     49ef394d81127cf5ee2eb64508feb0a96f72763073fe0ed9a0901d6008e51b2b8  guix-build-c7376babd19d/output/arm-linux-gnueabihf/SHA256SUMS.part
     540cec39c64338a03e7553bb9734d48061d5d1669ab6a59d6250d0dcd5821fde0  guix-build-c7376babd19d/output/arm-linux-gnueabihf/bitcoin-c7376babd19d-arm-linux-gnueabihf-debug.tar.gz
     6acc8be16daaecab8fdf96763973b8fe7e3e5e9e8014e1e04cf9133cba9f55375  guix-build-c7376babd19d/output/arm-linux-gnueabihf/bitcoin-c7376babd19d-arm-linux-gnueabihf.tar.gz
     7791971a7afaef6827d3a2b4da5f92a10aa28ff906efd372ef524b900c6815ba7  guix-build-c7376babd19d/output/arm64-apple-darwin/SHA256SUMS.part
     8bad59a9c07844fa7b788a8b17063c950fa6e124e046d1bee02959344438350dd  guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin-unsigned.tar.gz
     941de8c1590fc2479ac52946f61affcc592fa7c187abcef02972933e91487ff73  guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin-unsigned.zip
    10b4e834ae5e563d3cf975a0bc5cae1ba9734b19df387eddc45677f910b8bb5043  guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin.tar.gz
    115d2928dc2f3b6c5390a2020221f1f069bb4c0b0acff09c0d2c79b35ef486b5c9  guix-build-c7376babd19d/output/dist-archive/bitcoin-c7376babd19d.tar.gz
    124b8e4f59977136e23ae0dc20816a671257cee4156d80551cbe1c27024857e8f2  guix-build-c7376babd19d/output/powerpc64-linux-gnu/SHA256SUMS.part
    13d5ce1a3593408c9eb7d1f2fc3d61e7d400381335faeca5cc6be11999c798e4c6  guix-build-c7376babd19d/output/powerpc64-linux-gnu/bitcoin-c7376babd19d-powerpc64-linux-gnu-debug.tar.gz
    14e11a741a82205faa1c271aa62f5a20329cf46c30db46bda37ed25dc83cbca662  guix-build-c7376babd19d/output/powerpc64-linux-gnu/bitcoin-c7376babd19d-powerpc64-linux-gnu.tar.gz
    156ddd58c3fe5e4c600a65b5b9e4bb7e0b99bd3cd5f8655366bb0e29e498e480fe  guix-build-c7376babd19d/output/riscv64-linux-gnu/SHA256SUMS.part
    16c35ab4f03e23c3889fac7c1a3a64d979af00cb762f6425599149249e8df7e4fd  guix-build-c7376babd19d/output/riscv64-linux-gnu/bitcoin-c7376babd19d-riscv64-linux-gnu-debug.tar.gz
    17f46bee10789cb0d2f1df0a386b93ebf92ecbd9faaf3446b2df8dc1c5fa77e46c  guix-build-c7376babd19d/output/riscv64-linux-gnu/bitcoin-c7376babd19d-riscv64-linux-gnu.tar.gz
    185ba9da03a94da08d34d20edff2d9dd1de729f4087897d3573d5a17fbb053f0d7  guix-build-c7376babd19d/output/x86_64-apple-darwin/SHA256SUMS.part
    194b0a6d20c60ff4bfb85bb143b13b0ad28f9c65d536cdbda96b695abf03a0d268  guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin-unsigned.tar.gz
    209525e7082ae7c691381af6abff1a1733b7a2efd772ffbe9bbd98cb522cd8609d  guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin-unsigned.zip
    21a62eb1faf7a413267aab397e0c687016a2055f225f5a07d2780eaa7678e0260e  guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin.tar.gz
    2270ec027a5d59eefeedf1943c96504d39739753f367442211d0353512b1ac798d  guix-build-c7376babd19d/output/x86_64-linux-gnu/SHA256SUMS.part
    23793c670bbd8724271c4966cbcffae13a9c876821aeee9ede7449e705acd463e7  guix-build-c7376babd19d/output/x86_64-linux-gnu/bitcoin-c7376babd19d-x86_64-linux-gnu-debug.tar.gz
    244c711b6709ee78686f83296bb1e2596772d103cf4fc9959b50085a7d62652921  guix-build-c7376babd19d/output/x86_64-linux-gnu/bitcoin-c7376babd19d-x86_64-linux-gnu.tar.gz
    253562a253c9bc448238aad4bb612c73f4fef6899ef297967b4a77b46fab40e73d  guix-build-c7376babd19d/output/x86_64-w64-mingw32/SHA256SUMS.part
    263ae8eb90a5dd6fb50caf0f3df16bc0cb60646d1a126a45b89a70bc5d12d00ec4  guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-debug.zip
    2723f8f28cc9bb4c63d4c4420d8ccf5c0b2547c34b0ebf4b73b10d3553e7fc3681  guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-setup-unsigned.exe
    28c709e3dbcf3267a6f18ca984b8308bf7f05d58ddbbae83ae9e024db84693a861  guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-unsigned.tar.gz
    292cf7011773d95db8398fdb538a5650eb3abd443c2fc18ea091ab07059c778eaf  guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64.zip
    
  135. achow101 commented at 8:52 pm on June 12, 2024: member
    ACK c7376babd19d0c858fef93ebd58338abd530c1f4
  136. achow101 merged this on Jun 12, 2024
  137. achow101 closed this on Jun 12, 2024

  138. hebasto commented at 3:42 pm on July 4, 2024: member

    It seems that libbitcoin_consensus still depends on libbitcoin_util via the ParseHex() call in pubkey.cpp.

    However, the "consensus util" dependency is not allowed in contrib/devtools/check-deps.sh.

  139. ryanofsky commented at 3:17 pm on July 8, 2024: contributor

    It seems that libbitcoin_consensus still depends on libbitcoin_util via the ParseHex() call in pubkey.cpp.

    However, the "consensus util" dependency is not allowed in contrib/devtools/check-deps.sh.

    Nice find! This seems to point to a bug in the check-deps.sh script for exported template functions. In this case, the fact that pubkey.cpp depends on TryParseHex function is detected in generated consensus_imports.txt file:

    0_Z11TryParseHexIhESt8optionalISt6vectorIT_SaIS2_EEESt17basic_string_viewIcSt11char_traitsIcEE libbitcoin_consensus.a:libbitcoin_consensus_a-pubkey.o:
    

    And the same symbol is exported according to nm ./src/util/libbitcoin_util_a-strencodings.o

    00000000000000000 W _Z11TryParseHexIhESt8optionalISt6vectorIT_SaIS2_EEESt17basic_string_viewIcSt11char_traitsIcEE
    

    but it is a weak symbol, so the script does not detect it. Following fix for the script seems to work:

     0--- a/contrib/devtools/check-deps.sh
     1+++ b/contrib/devtools/check-deps.sh
     2@@ -78,7 +78,7 @@ extract_symbols() {
     3     local temp_dir="$1"
     4     for lib in "${!LIBS[@]}"; do
     5         for lib_path in ${LIBS[$lib]}; do
     6-            nm -o "$lib_path" | grep ' T ' | awk '{print $3, $1}' >> "${temp_dir}/${lib}_exports.txt"
     7+            nm -o "$lib_path" | grep ' T \| W ' | awk '{print $3, $1}' >> "${temp_dir}/${lib}_exports.txt"
     8             nm -o "$lib_path" | grep ' U ' | awk '{print $3, $1}' >> "${temp_dir}/${lib}_imports.txt"
     9             awk '{print $1}' "${temp_dir}/${lib}_exports.txt" | sort -u > "${temp_dir}/${lib}_exported_symbols.txt"
    10             awk '{print $1}' "${temp_dir}/${lib}_imports.txt" | sort -u > "${temp_dir}/${lib}_imported_symbols.txt"
    

    Producing the following output:

    0Error: libbitcoin_consensus_a-pubkey.o depends on libbitcoin_util_a-strencodings.o symbol 'std::optional<std::vector<unsigned char, std::allocator<unsigned char> > > TryParseHex<unsigned char>(std::basic_string_view<char, std::char_traits<char> >)', can suppess with:
    1    SUPPRESS["libbitcoin_consensus_a-pubkey.o libbitcoin_util_a-strencodings.o _Z11TryParseHexIhESt8optionalISt6vectorIT_SaIS2_EEESt17basic_string_viewIcSt11char_traitsIcEE"]=1
    2Error: Unexpected dependencies were detected. Check previous output.
    

    As for the removing the consensus library depenency on util:

    https://github.com/bitcoin/bitcoin/blob/a83f050dbe1392fc6b1b6c2a140c7346653b40d3/src/pubkey.cpp#L193

    Maybe #30377 could be a good fix for that, since ParseHex is just being used to parse a constant.

  140. ryanofsky referenced this in commit 5a8b9432cd on Jul 9, 2024
  141. hebasto commented at 6:47 am on July 14, 2024: member
    Ported to the CMake-based build system in https://github.com/hebasto/bitcoin/pull/264, except for the contrib/devtools/check-deps.sh script.
  142. hebasto removed the label Needs CMake port on Jul 14, 2024
  143. hebasto added the label Needs CMake port on Jul 14, 2024
  144. ryanofsky referenced this in commit 114b2a406e on Jul 24, 2024
  145. ryanofsky referenced this in commit 5dddbbce38 on Jul 26, 2024
  146. luke-jr referenced this in commit 7376fef46d on Aug 1, 2024
  147. maflcko commented at 5:14 am on August 29, 2024: member
    This is tagged with “Needs CMake port”, so if anything needs to be done, now would probably be a good time.
  148. hebasto commented at 10:41 am on August 29, 2024: member

    This is tagged with “Needs CMake port”, so if anything needs to be done, now would probably be a good time.

    Related to #30415.

  149. ryanofsky referenced this in commit 0c23041591 on Sep 4, 2024
  150. ryanofsky referenced this in commit 3a90dd0df6 on Sep 4, 2024
  151. hebasto removed the label Needs CMake port on Sep 4, 2024

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-25 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me