Significantly Improve Wallet Load times #2733

pull pstratem wants to merge 2 commits into bitcoin:master from pstratem:keyfix changing 4 files +97 −9
  1. pstratem commented at 11:57 PM on June 3, 2013: contributor

    Extremely large wallets take a very long time to load.

    After some testing I found that most of the time was spent in EC_KEY_check_key

    The fix this I have merged two of the checks into using a single CECKey instance, replaced EC_KEY_check_key with a faster operation (thanks to sipa for helping me there), and appended a hash of the private key to "key" entries.

    On my system loading one private key took approximately 1500 usec.

    CKey::Load reduced this to 900 usec replacing EC_KEY_check_key reduced this to 500 usec checking just the hash reduced this to 40 usec

  2. improve wallet load time by removing duplicated checks in EC_KEY_check_key and adding a hash for "key" entries in wallet.dat b5ba77b77a
  3. Merge branch 'master' of git://github.com/bitcoin/bitcoin into keyfix ab139c964e
  4. BitcoinPullTester commented at 12:50 AM on June 4, 2013: none

    Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/ab139c964e32b3ec5279eea0e749e0fe4a31a2dd for binaries and test log. This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/ Contact BlueMatt on freenode if something looks broken.

  5. in src/key.cpp:None in ab139c964e
     190 | +            if (!pkey || !group || !pub_key || !priv_key)
     191 | +                goto err;
     192 | +            
     193 | +            order = BN_new();
     194 | +            
     195 | +            if ((ctx = BN_CTX_new()) == NULL)
    


    gavinandresen commented at 4:34 AM on June 4, 2013:

    Looks like ctx and point are memory leaked in the return true case...


    vinniefalco commented at 12:53 PM on June 4, 2013:

    Do I need my glasses or is that a "goto" statement?


    jgarzik commented at 1:25 PM on June 4, 2013:

    "goto" use is just fine. It is efficient for both the compiler and human eyes.


    vinniefalco commented at 1:27 PM on June 4, 2013:

    No, goto is not okay. http://stackoverflow.com/questions/46586/goto-still-considered-harmful

    It's true that there are exceedingly rare cases where a goto is useful but this is not one of them. Especially this case, error handling.


    sipa commented at 1:31 PM on June 4, 2013:

    This is code that I assume was copied from the OpenSSL sources, which uses such error handling extensively. Without RAII-like features (as it was copied from C), I also know of no less contrived way of dealing with this.


    vinniefalco commented at 1:39 PM on June 4, 2013:

    Then roll a quick RAII wrapper (don't add more dependence on boost). Here's one for you http://codepad.org/gr8M2LE6

    Look at how easy it was to introduce a memory leak. I know that you guys know better than this!


    jgarzik commented at 1:47 PM on June 4, 2013:

    ~90 LOC just to avoid a goto?

    I think that speaks for itself :)


    vinniefalco commented at 1:52 PM on June 4, 2013:

    90 LOC to have the right tool. Note, you could use boost::auto_ptr or whatever RAII wrapper they have around dynamically created objects. I prefer not to create additional dependencies on boost. That Bitcoin source code is sloppy and breaks numerous "best practices" endorsed by luminaries of C++ should not even be up for debate - I know that you guys know better! The attitude needs to change or else it will remain a difficult to maintain mess, becoming progressively harder to implement changes without breaking things and tons of extra work.


    jgarzik commented at 2:19 PM on June 4, 2013:

    NAK that logic


    laanwj commented at 2:24 PM on June 4, 2013:

    Goto is fine in C code (in many cases the only sane way to do error handling with proper cleanup), but in C++ we can indeed do better with RAII. Using boost would be preferable to rolling our own RAII wrappers.


    ValleZ commented at 3:47 PM on June 13, 2013:

    I'm not a c++ guy, but I believe there are RuntimeExceptions, right? And maybe they allows to pass description of error. It would be much better than to use goto.


    GilesBathgate commented at 8:14 AM on July 3, 2013:

    Rather than creating 90 Lines of RAII wrappers, You could probably just rewrite it like this:

    bool SetPrivKeyCleanup(BN_CTX *ctx,EC_POINT *point)
    {
     if (ctx   != NULL)
      BN_CTX_free(ctx);
     if (point != NULL)
      EC_POINT_free(point);
     return false;
    }
    

    Then just ammend:

                if (!pkey || !group || !pub_key || !priv_key)
                    return SetPrivKeyCleanup(ctx,point);
    
                order = BN_new();
    
                if ((ctx = BN_CTX_new()) == NULL)
                    return SetPrivKeyCleanup(ctx,point);
    
                if (!EC_GROUP_get_order(group, order, ctx))
                    return SetPrivKeyCleanup(ctx,point);
    
                // check if generator * priv_key == pub_key 
                if (BN_cmp(priv_key, order) >= 0)
                    return SetPrivKeyCleanup(ctx,point);
    
                if ((point = EC_POINT_new(group)) == NULL)
                    return SetPrivKeyCleanup(ctx,point);
    
                if (!EC_POINT_mul(group, point, priv_key, NULL, NULL, ctx))
                    return SetPrivKeyCleanup(ctx,point);
    
                if (EC_POINT_cmp(group, point, pub_key, ctx) != 0)
                    return SetPrivKeyCleanup(ctx,point);
    
                return true;
    

    Since I didn't test this there is probably a good reason why it wouldn't work, but you get the basic idea.


    laanwj commented at 8:51 AM on July 3, 2013:

    I still prefer RAII. Make the compiler do the work of making it leak proof. Safer and more convenient.


    GilesBathgate commented at 10:17 AM on July 3, 2013:

    Indeed making an EC_POINT class that implements EC_POINT_new in the constructor and EC_POINT_free in the destructor, as well as operator overloads for EC_POINT_mul and EC_POINT_cmp, is much more in the spirit of C++. I was really just addressing the comment:

    Do I need my glasses or is that a "goto" statement?

    Its not really an issue worth making a fuss about really. Perhaps I should have kept quiet.

  6. in src/walletdb.cpp:None in ab139c964e
     295 | +            
     296 | +            bool fSkipCheck = false;
     297 | +            
     298 | +            if (hash != 0)
     299 | +            {
     300 | +                if (Hash(pkey.begin(), pkey.end()) != hash)
    


    sipa commented at 7:41 AM on June 4, 2013:

    The hash should be calculated over both private and public key, as the reason for the test is guaranteeing that we have matching keypairs.

  7. sipa commented at 3:55 PM on June 15, 2013: member

    Would you mind if I rewrote this a bit, by simultaneously changing how keys are stored in the wallet file (using a serialized CKey, instead of CPrivKey, and combining it with the new CKeyMeta)?

  8. pstratem commented at 8:17 PM on June 15, 2013: contributor

    That sounds like a good idea.

  9. pstratem commented at 9:19 PM on July 4, 2013: contributor

    I'm closing this pull request.

    sipa's suggestion is a better long term solution.

  10. pstratem closed this on Jul 4, 2013

  11. Bushstar referenced this in commit 9eb0ca7040 on Apr 5, 2019
  12. DrahtBot locked this on Sep 8, 2021

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2026-04-19 00:16 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me