How to define custom parameters in an interface data definition (*.capnp) ? #51

I've successfully added and built API/data definitions for my new Validation interface

    namespace interfaces {
    
    //! Inteface giving clients access to the validation engine.
    class Validation
    {
    public:
        virtual ~Validation() {}
    
        // Check if headers are valid.
        virtual bool validateHeaders(const BlockHeader& header) = 0;
    };
    
    
    } // namespace interfaces

    using Cxx = import "/capnp/c++.capnp";
    $Cxx.namespace("ipc::capnp::messages");
    
    using Proxy = import "/mp/proxy.capnp";
    $Proxy.include("interfaces/validation.h");
    $Proxy.include("ipc/capnp/validation.capnp.h");
    
    interface Validation $Proxy.wrap("interfaces::Validation") {
        destroy [@0](/bitcoin-core-multiprocess/contributor/0/) (context :Proxy.Context) -> ();
        validateHeaders [@1](/bitcoin-core-multiprocess/contributor/1/) (context :Proxy.Context, header: BlockHeader) -> (result: Bool);
    }

I would like to add a custom parameters to my interface :

struct BlockHeader $Proxy.wrap("BlockHeader")
{
    nVersion [@0](/bitcoin-core-multiprocess/contributor/0/) :Int32;
    hashPrevBlock [@1](/bitcoin-core-multiprocess/contributor/1/) :Data;
    hashMerkleRoot [@2](/bitcoin-core-multiprocess/contributor/2/) :Data;
    nTime [@3](/bitcoin-core-multiprocess/contributor/3/) :UInt32;
    nBits [@4](/bitcoin-core-multiprocess/contributor/4/) :UInt32;
    nNonce [@5](/bitcoin-core-multiprocess/contributor/5/) :UInt32;
}

But so far, I'm hitting libmultiprocess compilations errors which are beyond my understanding for now.

AFAIU by observing the interfaces in #10102, you have two options to add a custom parameters :

• wrapping a struct duplicated in the API definition (e.g struct WalletTxOut $Proxy.wrap("interfaces::WalletTxOut"))
• declaring a custom reader/builder in "types files" linking API/data definitions (e.g CustomReadMessage/CustomBuildMessage)

I should rebase my branch on top of #10102, as this branch includes a lot more of common helpers.

Feel free to correct the terminology I used, I'm progressively catching up on libmultiprocess usage :)

I'm happy to help debug if you post a git tag. I think probably for your case, you don't need to build on https://github.com/bitcoin/bitcoin/pull/10102 and just https://github.com/bitcoin/bitcoin/pull/19160 should be sufficient.

Everything you posted and wrote is right. To use custom C++ types in interfaces, you need to either use $Proxy.wrap and list all fields in the type, or you need implement CustomReadField/CustomWriteField/CustomPassField overloads to read/write the C++ type as a capnp type.

The CustomReadMessage/CustomBuildMessage overloads you mentioned are added in https://github.com/bitcoin/bitcoin/pull/10102. They are a simpler alternative to CustomReadField/CustomWriteField/CustomPassField and can be defined in .cpp instead of .h files since they aren't template functions. https://github.com/bitcoin/bitcoin/pull/10102 also adds CustomReadField/CustomWriteField/CustomPassField overload that detect if a type has Serialize/Unserialize methods and will use those if them if they are available.

Thanks to propose to help on the debug, here the git tag : https://github.com/ariard/bitcoin/commit/ddcfd078b6b61d80872e90521384c616091831f2. Note, you need to pass the following configuration flag --enable-altnet on top of the --enable-multiprocess one.

I'm trying to follow the first path for my custom C++ types, i.e $Proxy.wrap and list all fields in the type. I'm encountering the following build error :

/home/user/Bitcoin/bitcoin/depends/x86_64-pc-linux-gnu/include/mp/proxy-types.h:942:30: error: no matching function for call to ‘BuildPrimitive(mp::InvokeContext&, const uint256&, mp::TypeList<capnp::Data::Builder>)’
     output.set(BuildPrimitive(invoke_context, std::forward<Value>(value), TypeList<decltype(output.get())>()));                                                                                                  
                ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~       

AFAICT, such BuildPrimitive() is present in bitcoin/bitcoin#10102 to build the node interface custom types (in src/ipc/capnp/node-types.h). Though it's just returning a ::capnp::Void, I'll try to do the same for my Validation interface.

The BuildPrimitive error indicates that no Build overload is present that can convert const uint256& C++ values to capnp::Data values. (Sorry about the confusing error message. I want to improve these errors by collapsing Build* overloads with C++17 if constexpr expressions in #46. This error happens because BuildPrimitive is the lowest priority build function that gets called when no other build functions can be applied the data type).

Because uint256 is a serializeable type, the fix for this is to supply a CustomBuildField function that can serialize it, like: https://github.com/ryanofsky/bitcoin/blob/db776ff2e92eb51fd51f56f9557546d4484a44d9/src/ipc/capnp/common-types.h#L248-L262 added in https://github.com/bitcoin/bitcoin/pull/10102

I'm building your branch and can post a patch soon.

Following commit should fix the BuildPrimitive error: https://github.com/ryanofsky/bitcoin/commit/e53c7a9778ec9b1703c593ab18f60d6588f8c474. It just pulls in common-types.h functions from https://github.com/bitcoin/bitcoin/pull/10102 to be able to handle serializable types.

There are still some other build errors on the branch, which I can try to help with, so feel free to ask any followups!

Thanks for the answer on this, I finally fixed the other build errors by rebasing and tweaking few others files. I can build the data definitions I want and making progress on my altnet branch.

I don't have other libmultiprocess questions for now :)

I don't have other libmultiprocess questions for now :)

Will close this issue, but feel free to reopen or post questions or work-in-progress code in new issues. Happy to help!

Sure, fyi the WIP branch is here: https://github.com/ariard/bitcoin/commits/2021-07-altnet-lightning, where I'm able to chain multiple processes : bitcoin-node -> bitcoin-altnet -> altnet-lightning but give me few more days to rebase/cleanup code/test with lightning driver then I think I'll have more advanced questions on the code generator itself!