[22.0rc2] There are cases where the fee calculation is incorrect #22670

1. 
   fujicoin commented at 2:37 am on August 10, 2021: none

   Please read this, thinking that there may be such a possibility. I fork each version of Bitcoin to make Fujicoin. I am currently testing Fujicoin made from v22.0rc2. https://github.com/fujicoin/fujicoin

   I often create large transactions with more than 350 inputs, but in v22.0, I fail to transfer money due to lack of fee about once every few times and get into a “not in memory pool” state. This phenomenon sometimes occurred in previous versions. It rarely occurred in v0.21.x, but it seems that the frequency increased in v22.0.

   Specific explanation

   The following image is the dialog at the time of remittance.

   The following images show the remittance result and its details.

   

   I made a program with python and calculated fee from raw Tx. In my calculation the fee was 2.57729334. Therefore, it is correct to be out of memoly pool due to insufficient remittance fee. In this case, the pre-remittance fee calculation is incorrect.

   As you can see from the figure, this phenomenon can be reproduced many times if the wallet state is the same. Unfortunately, I don’t have any more parsing ability, so I can’t identify what kind of transaction causes a calculation error.

   Fortunately, I’ve never experienced this with regular size transactions. Transactions that could not be remitted can be canceled, so there is no actual harm.

   I think it’s pretty difficult to identify this problem, but I think it’s very good if it improves.

2. fujicoin added the label Bug on Aug 10, 2021
3. 
   fujicoin commented at 4:58 am on August 10, 2021: none

   In the wallet state where this error is surely reproduced, stop v22.0 and start v0.21.1. And when I execute the same transaction, no error occurs. v0.21.1 calculates fee correctly. Similar confirmations were made for v0.21.0 and v0.20.0. These calculate the fee correctly.

   From the above test results, it can be said that the calculation of fee before remittance of v22.0 has changed significantly from the past version.

4. 
   fujicoin commented at 7:27 am on August 10, 2021: none

   For reference, attach the debug.log of the relevant part when an error occurs. debug.log
5. 
   ghost commented at 8:06 am on August 10, 2021: none

   I made a program with python and calculated fee from raw Tx. In my calculation the fee was 2.57729334. Therefore, it is correct to be out of memoly pool due to insufficient remittance fee. @fujicoin Did you try to abandon the transaction using abandontransaction and use a higher fee rate?

   There are few questions with similar issues in which low fee rate was used and transaction timed out: https://bitcoin.stackexchange.com/search?q=not+in+memory+pool

   Similar thing is mentioned in your log file:

   02021-08-10T00:24:04Z [default wallet] Submitting wtx c6281e449ad08aa694d28c32cc545b8ddb3d01f74e3b0eff4f81ce9e9c6e616d to mempool for relay
   12021-08-10T00:24:04Z [default wallet] CommitTransaction(): Transaction cannot be broadcast immediately, min relay fee not met, 257729334 < 527440000
   

   I often create large transactions with more than 350 inputs

   Not sure what all changes you made in your altcoin, there are some size limits on transactions too in Bitcoin. For example someone was getting an error in NBitcoin recently for using lot of inputs: https://gitter.im/MetacoSA/NBitcoin?at=60eef4327473bf3d7825ab85

   Don’t think there are any issues in Bitcoin Core so this issue is off-topic IMO.

6. 
   fujicoin commented at 10:21 am on August 10, 2021: none

   @prayank23 I’m not having such a low-level discussion. In the first image, the Total Amount is 5005.27790000 and the fee is 5.27790000. The remittance amount is 5000. In the third image, Netamount is 5002.57729334 and fee is 2.57729334. Don’t you think this is strange?
7. 
   achow101 commented at 6:20 pm on August 10, 2021: member

   This is likely due to the coin selection changes that were merged for 22.0. I’ll have a closer look.

   Your debug.log indicates that either -paytxfee was set or you set a custom feerate when making the transaction. Which did you set, and what was it set to?

   Are you able to recreate this issue using just Bitcoin Core on regtest?

   ---

   The failure should come from an error is size estimation. It appears to be estimating that the size of the transaction is less than half of the actual size of the transaction. Edit: No, it’s estimating the right size, the output value is being set incorrectly. Or maybe its input values are calculated incorrectly.

   ---

   As an experiment, can you try using raw transaction RPCs? fundrawtransaction will return the estimated transaction fee, and after you do sendrawtransaction, you can lookup the actual transaction fee in the wallet.

8. 
   fujicoin commented at 6:33 pm on August 10, 2021: none

   @achow101 I have set 0.1 FJC / kB as the Custom feerate. This value corresponds to the minimum feerate.
9. 
   achow101 commented at 6:54 pm on August 10, 2021: member

   @fujicoin Can you provide the output of getrawtransaction c6281e449ad08aa694d28c32cc545b8ddb3d01f74e3b0eff4f81ce9e9c6e616d true or gettransaction c6281e449ad08aa694d28c32cc545b8ddb3d01f74e3b0eff4f81ce9e9c6e616d true true?
10. 
    fujicoin commented at 6:58 pm on August 10, 2021: none

    I tested with a transaction that failed to transfer 3000 FJC. The following image is the information at the time of remittance. The fee is 3.172.

    When I checked this transaction with fundrawtransaction, I got the following response. The fee rate was specified as 0.1 FJC/kB. fundrawtransaction "[RAW TX]" "{\"feeRate\": 0.1}"

    0"fee": 3.18670000,
    1  "changepos": 0
    

    The result of sendrawtransaction is as follows:

    0min relay fee not met, 253605577 < 316870000 (code -26)
    

11. 
    achow101 commented at 8:14 pm on August 10, 2021: member

    I believe the issue is in the effective value calculation for inputs. Since the fee is being underpaid, that means that the effective value of inputs is calculated to be higher than it actually is, which means that the size is being calculated smaller than it actually is. I don’t see anything in our code that would cause this issue that would not be apparent in smaller transactions. It is possible that the wallet may be corrupted and contain some data that causes the size calculation of inputs to be incorrect, however I don’t really know what would cause that (the size calculation is only dependent on scriptPubKeys and private keys).

    Possibly the issue is that there are some inputs which are causing the problem and so you are seeing the issue with only very large txs because they happen to include those inputs. This could be tested by using fundrawtransaction with only one input and making sure that only that input is in the funded tx. Then use testmempoolaccept (instead of sendrawtransaction so no transactions are actually sent) to see the tx works. If this hypothesis is correct, then some txs will fail and those inputs would be the problem.

    Otherwise, I can’t tell what the issue is without having a test case to experiment with myself.

12. 
    fujicoin commented at 9:13 pm on August 10, 2021: none

    @achow101 Thank you for your answer. I am trying to reproduce with bitcoin v22.0 rc2 windows-qt regtest. But for now, even if I make a big transaction, the problem doesn’t occur. I will consider the peculiar input UTXO based on your suggestions.
13. 
    fujicoin commented at 7:28 am on August 11, 2021: none

    The wallet currently being tested is collecting coins newly generated by mining. A rescan was performed, but the situation has not changed.

    Subsequent investigations revealed that: ・I checked 24 transactions that failed to transfer money, but all of them have one “vout”. There is no change vout. 　Although the fee is insufficient, the total amount will not be less than the remittance amount. It’s as if wallet considers fee to be zero. ・Transactions that fail to remit use the output of change. ・Successful large transactions do not use change outputs.

    ・When all the change were locked, a big transaction will succeed. Conversely, unlocking will fail. ・Change is bech32 address by default. I made a change and tested it by specifying “legacy” or “p2sh-segwit” with -changetype. ・If I unlock only the “legacy” change and send money, it will fail. Also, even if I unlock only the change of “p2sh-segwit” and send the money, it will fail. In short, using change fails a big transaction.

    A similar test was performed with fujicoin v0.21.0 qt. Unlock only bech32 change. Large transactions succeed. The transaction input contains the change. Similar results were obtained by testing with “legacy” or “p2sh-segwit” change.

    What I can say for sure is that the v22.0rc2 and v0.21.0 wallets behave differently.

14. 
    achow101 commented at 5:39 pm on August 11, 2021: member

    @fujicoin If you unlock on bech32 change on 22.0rc2, does it also fail?
15. 
    fujicoin commented at 7:13 pm on August 11, 2021: none

    @achow101 Yes, it also fail. I’m still considering it with bitcoin v22.0rc2 regtest. I found something interesting and will report it soon.
16. 
    fujicoin commented at 8:10 pm on August 11, 2021: none

    With bitcoin v22.0rc2 regtest, even if I blindly perform a reproduction test on a large transaction, it does not reproduce. So I decided to create an environment similar to fujicoin’s wallet. My fuji coin wallet is a collection of generated coins of about 13.99 FJC. The minimum fee for fujicoin is 10000sat / byte. On the other hand, bitcoin is 1sat / byte. So I made the bitcoin wallet a set of 0.001399 BTC. Next, I made 0.0002, 0.0003, 0.0004, 0.0005 change. Then, I sent 0.3BTC and 0.5BTC, but no error occurred as a result.

    What is interesting is the change generated at this time. Although the wallet is a set of 0.001399 BTC, large changes of 0.00974425 and 0.01009771 were generated.

    On the other hand, in fujicoin, if the transaction is successful, a small change of about 3FJC is generated for the set of 13.99FJC.

    This means that bitcoin has more transaction inputs. Of course, this difference is considered to be due to the difference in the fee-related parameters of bitcoin and fujicoin.

    The fee-related parameters of fujicoin are as follows.

     0DEFAULT_MIN_RELAY_TX_FEE	10000000
     1DEFAULT_TRANSACTION_MAXFEE		10 * COIN
     2HIGH_TX_FEE_PER_KB		1 * COIN
     3HIGH_MAX_TX_FEE		100 * HIGH_TX_FEE_PER_KB
     4DEFAULT_BLOCK_MIN_TX_FEE	10000000
     5DEFAULT_INCREMENTAL_RELAY_FEE	10000000
     6DUST_RELAY_TX_FEE	3000
     7DEFAULT_PAY_TX_FEE	0
     8DEFAULT_FALLBACK_FEE	20000000
     9DEFAULT_DISCARD_FEE	100000000
    10DEFAULT_TRANSACTION_MINFEE	10000000
    11WALLET_INCREMENTAL_RELAY_FEE	50000000
    

    Which parameters are affecting the number of inputs in a transaction? It is thought that the reason why the parameters have an influence on the creation of transactions is that the transaction-related programs have been significantly rewritten in v22.0.

    ★ postscript: This seems to be irrelevant as it is the same for bitcoin v0.21.0 and fujicoin v0.21.0.

17. 
    fujicoin commented at 10:27 pm on August 11, 2021: none

    In the first place, whether the address is for change or not knows only the wallet that has the address. Other wallets don’t know if the address is for change and it doesn’t matter. Why does the change in the input address affect when creating a transaction? Where is the part that is judged to be change and processed? This problem may not be a miscalculation of fee.
18. 
    achow101 commented at 11:50 pm on August 11, 2021: member

    I have found the source of the problem, and I’m able to replicate the error on regtest when I set the minrelayfee to be similarly high. The culprit is that there are some places where we use an outputs real value rather than its effective value. I think the issue is much more apparent at high feerates because the difference between the effective value and the real value of an output is much larger so using the incorrect value means the calculation error can become big enough to be an issue.

    I’ll have a fix shortly.

    ---

    The problem with the change outputs probably has to do with the size of the outputs rather than the fact that they are change.

19. 
    fujicoin commented at 2:04 am on August 12, 2021: none

    @achow101 You are right. I have now succeeded in reproducing the problem with bitcoin’s regtest. I finally succeeded in reproducing the phenomenon with bitcoin v22.0rc3 regtest. Create a wallet situation that is almost the same as fujicoin. ・set -maxtxfee=10 ・Send 13.99 BTC 400 times to one legacy address of an empty wallet-A. ・Advance one block. ・Select one UTXO of wallet-A and send 11BTC to the other wallet. ・Select one UTXO of wallet-A and send 10.5 BTC to the other wallet. ・Advance one block. ・set Custom tx fee to 0.1 BTC/kB ・Send 3000 BTC to another wallet. This method reproduces the problem. The transaction before remittance and the actual transaction are different.

    Thank you and expect a fix.

20. 
    achow101 commented at 2:16 am on August 12, 2021: member

    Fix in #22686
21. 
    fujicoin commented at 4:01 am on August 12, 2021: none

    @achow101 I created a binary with guix build to reflect the content of your PR. As a result of remittance of 3000FJC with two bech32 changes on the wallet, it was confirmed that the remittance can be done without any problem. When I decoded and confirmed the transaction, the input did not contain any change. I think this is the result of a change in the behavior of the coin selection. I would like to continue testing as soon as the funds are collected. Thank you for your prompt response.
22. 
    fujicoin commented at 8:25 pm on August 12, 2021: none

    I repeated testing large transactions with a modified binary. Transactions with change in input are also normal. The coin selection feels good. I will close this case because I think the problem has been resolved.
23. fujicoin closed this on Aug 12, 2021

    ---

24. fanquake referenced this in commit 820129aee9 on Aug 16, 2021
25. MarcoFalke referenced this in commit 92f3a4b4d0 on Aug 19, 2021
26. SmartArray referenced this in commit 5e3a8f8c13 on Apr 6, 2022
27. SmartArray referenced this in commit 69ea42be26 on Apr 6, 2022
28. DrahtBot locked this on Aug 18, 2022

Contributors
fujicoin ghost achow101

Labels
Bug