rpc: make loadtxoutset async #28620

1. 
   Sjors commented at 1:20 pm on October 9, 2023: member

   Please describe the feature you’d like to see added.

   loadtxoutset should return more quickly, perhaps just checking that the file exists. There should be another RPC to monitor the status.

   Is your feature related to a problem, if so please describe it.

   1. loadtxoutset takes too long, even requiring -rpctimeout=0. This makes it hard for potential integrations like Umbrel, BTCPay server or Specter desktop to show progress.

   2. Progress can only be followed via the debug log, which is too noisy during IBD.

   Describe the solution you’d like

   It could follow the same pattern as scanblocks, with start, status and abort. The latter two won’t need an argument. This would not be backwards compatible, but imo that’s fine as long as it lands before mainnet support.

   Describe any alternatives you’ve considered

   No response

   Please leave any additional context

   No response

2. Sjors added the label Feature on Oct 9, 2023
3. 
   pablomartin4btc commented at 1:45 am on October 10, 2023: contributor

   Concept ACK.

   eg running it on mainnet:

   0./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset ${AU_DATADIR}/utxo-800000.dat
   1error: timeout on transient error: Could not connect to the server 127.0.0.1:8332 (error code 0 - "timeout reached")
   2
   3Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
   

   I got the above error message around 50% of processing (checking bitcoind logs):

   2023-10-09T19:35:35Z [httpworker.0] [snapshot] 56000000 coins loaded (50.21%, 7186 MB)

   And if you try again at least you can have a clue (double checking in the logs that it’s stil processing it):

   0./src/bitcoin-cli -datadir=${AU_DATADIR} loadtxoutset ${AU_DATADIR}/utxo-800000.dat
   1error code: -1
   2error message:
   3Fatal LevelDB error: IO error: .../chainstate_snapshot/LOCK: already held by process
   

4. 
   Sjors commented at 7:34 am on October 11, 2023: member

   @pablomartin4btc meanwhile you should use -rpctimeout=0 when calling this.
5. 
   pablomartin4btc commented at 1:13 pm on October 11, 2023: contributor

   @Sjors

   Error parsing command line arguments: Invalid parameter -rpctimeout=0

   I found that the parameter is actually called -rpcclienttimeout, so I’m re-trying loadtxoutset with it now as you recommended (-rpcclienttimeout=0). Thanks!

6. 
   pablomartin4btc commented at 4:29 pm on October 11, 2023: contributor

   -rpcclienttimeout=0 worked fine:

   0./src/bitcoin-cli -datadir=${AU_DATADIR} -rpcclienttimeout=0 loadtxoutset ${AU_DATADIR}/utxo-800000.dat
   1{
   2  "coins_loaded": 111535121,
   3  "tip_hash": "00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054",
   4  "base_height": 800000,
   5  "path": "/home/xyz/.test_utxo_2/utxo-800000.dat"
   6}
   

7. 
   hernanmarino commented at 5:38 pm on October 11, 2023: contributor

   This would be nice, is anyone working on this ? I might take it
8. 
   hernanmarino commented at 3:17 pm on October 17, 2023: contributor

   I created a (partial) fix for the issue here #28659 This PR only implements the start beahavior (making it mandatory.) Since the changes are simple enough, if there is interest in implementing this, it could be merged in this release cycle while enabling the possibility of implementing status abort or similar parameters (without chaging the RPC syntax) in follow-ups, even in the next release.

Contributors
Sjors pablomartin4btc hernanmarino

Labels
Feature