Fatal performance when importing 151M private keys

pbies commented at 2:55 am on September 9, 2020: none

Earlier I was importing several thousands of private keys to Bitcoin Core and it took much time, especially rescan of the blockchain. Few hours, >3 surely.

But now, when I am importing 151M private keys (10 GB file with keys) the import process is now taking 2 days and I have 65% for now.

How badly Bitcoin Core is written is unbelivable. There is no other way to scan such an amount of keys, because other apps are for private, limited use and do not offer scanning the blockchain with such a large amount of keys.

Will this be fixed or even rewritten?

pbies added the label Feature on Sep 9, 2020

MarcoFalke added the label Wallet on Sep 9, 2020

MarcoFalke added the label Resource usage on Sep 9, 2020

MarcoFalke renamed this:
~~Fatal Bitcoin Core 0.20.1 performance~~
Fatal performance when importing 151M private keys
on Sep 9, 2020

TheBlueMatt commented at 2:06 pm on September 9, 2020: member

How are you importing these? Are you using the importmulti RPC?

promag commented at 2:16 pm on September 9, 2020: member

Looks like it can take around >3 hours times 1000 right? Do you know the “birth” timestamp of those keys?

pbies commented at 4:40 pm on September 9, 2020: none

I am importing via importwallet. Timestamp is not known so whole blockchain must be scanned.

pstratem commented at 7:49 pm on September 9, 2020: contributor

@pbies scanning with the block filter index can be much faster, still working on it though, https://github.com/bitcoin/bitcoin/pull/19116

pstratem commented at 5:38 am on September 10, 2020: contributor

btw there’s 1506642276 transaction outputs, so checking against 150million keys is 225 quadrillion operations, if that ever finishes it’s because computers are magic

edit: it’s been pointed out that it’s not O(m * n) but rather O(m log n) for m transaction outputs and n keys (or something close to that)

pbies commented at 8:13 am on September 10, 2020: none

I wonder if making index for keys would improve the scan process. So if the key begins with few characters it would be faster to find it when all the keys beginning with the same characters would be indexed. All keys would be marked in index. It would be long to make index but much faster to find newly added keys.

pbies commented at 11:38 am on September 10, 2020: none

Multithreading also would help. Seems like for 12 thread CPU Bitcoin Core is using maximally 10% of all CPUs.

jonasschnelli commented at 12:04 pm on September 10, 2020: contributor

IMO, the Bitcoin Core wallet is not designed to hold 151M private keys. It does probably work if you have the necessary RAM and disk requirements, but it’s certainly not optimised for that case.

How badly Bitcoin Core is written is unbelivable. There is no other way to scan such an amount of keys, because other apps are for private, limited use and do not offer scanning the blockchain with such a large amount of keys.

Will this be fixed or even rewritten?

This is up to you. It looks like you have strong interest that large amounts of keys work smoothly. I suggest you either start benchmarking and optimising Core or that you’ll find and pay someone that is willing to do that.

pbies commented at 12:55 pm on September 10, 2020: none

Surely if someone would have to do that would be payed for that work.

But in case Bitcoin Core is still in development, it would be nice to think earlier about implementing multithreading to make use of multicore CPUs.

Just a little advice now, when it is not too late.

sipa commented at 6:15 pm on September 10, 2020: member

It’s well known that there are improvements possible to the rescanning speed, and there is some work towards that as @phantomcircuit mentions. However, this is open source software, and people work on what is important to them. At some point that may mean faster rescanning, but a situation with 151M keys seems so unrealistic that I doubt anyone will specifically target such a use case.

But in case Bitcoin Core is still in development,

That’s not relevant. If you want someone to work on what feels important to you, you’ll need convince them somehow, through financial means or otherwise. Starting off by insulting people who write software you’re using for free and claiming their software is badly written is unlikely to help you with the latter, though.

here is no other way to scan such an amount of keys, because other apps are for private

There are tons of other open-source applications that can index the blockchain and query it, some of which are specifically written for that purpose.

pbies commented at 6:19 pm on September 10, 2020: none

Statement of facts is not insulting people. Software should be efficient in all cases and this is one of them.

pbies closed this on Sep 10, 2020

promag commented at 6:51 pm on September 10, 2020: member

Should? That’s like saying software should have no bugs, ever.

DrahtBot locked this on Feb 15, 2022

Fatal performance when importing 151M private keys #19921