Update libsecp256k1 subtree to latest master #21573

pull sipa wants to merge 4 commits into bitcoin:master from sipa:202104_secp256k1 changing 96 files +4871 −2461
  1. sipa commented at 5:12 pm on April 2, 2021: member

    This updates our src/secp256k1 subtree to the latest upstream master. The changes include:

    • The introduction of safegcd-based modular inverses, reducing ECDSA signing time by 25%-30% and ECDSA verification time by 15%-17%.
    • Removal of libgmp as an (optional) dependency (which wasn’t used in the Bitcoin Core build)
    • CI changes (Travis -> Cirrus)
    • Build system improvements
  2. MarcoFalke added the label Needs gitian build on Apr 2, 2021
  3. MarcoFalke added the label Needs Guix build on Apr 2, 2021
  4. MarcoFalke added the label Validation on Apr 2, 2021
  5. sipa commented at 7:03 pm on April 2, 2021: member
    @sipsorcery Is there a way of disabling C4146 (at least inside src/secp256k1, and at least it being treated as an error)? All these instances of negating a positive number are well-defined and intentional…
  6. sipsorcery commented at 7:12 pm on April 2, 2021: member

    @sipa yes the warning should be disabled by adding <DisableSpecificWarnings>4146</DisableSpecificWarnings> to build_msvc\libsecp256k1\libsecp256k1.vcxproj.

    Probably clearer putting it this way: replace the ClCompile XML node with the below:

    0<ClCompile>   
    1<PreprocessorDefinitions>ENABLE_MODULE_ECDH;ENABLE_MODULE_RECOVERY;ENABLE_MODULE_EXTRAKEYS;ENABLE_MODULE_SCHNORRSIG;%(PreprocessorDefinitions)</PreprocessorDefinitions>
    2  <AdditionalIncludeDirectories>..\..\src\secp256k1;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
    3  <DisableSpecificWarnings>4146</DisableSpecificWarnings>
    4</ClCompile>
    
  7. sipa commented at 7:20 pm on April 2, 2021: member
    @sipsorcery Thanks!
  8. sipa force-pushed on Apr 2, 2021
  9. Disable certain false positive warnings for libsecp256k1 msvc build cabb566123
  10. sipa force-pushed on Apr 2, 2021
  11. fanquake added the label Upstream on Apr 3, 2021
  12. DrahtBot commented at 1:28 am on April 3, 2021: member

    The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

    Conflicts

    Reviewers, this pull request conflicts with the following ones:

    • #21859 (Add minisketch subtree and integrate in build/test by sipa)

    If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

  13. DrahtBot commented at 10:41 am on April 4, 2021: member

    Gitian builds

    File commit 9565dafed5973b35d9b242ff5352ee7abdc5799b(master) commit 4c060c83b26828996b6be528e406eab2990a72eb(master and this pull)
    *-aarch64-linux-gnu-debug.tar.gz c0046768eeb849d7... c49e0ffa987280f5...
    *-aarch64-linux-gnu.tar.gz 1580ba8ff2495a87... c28e8c4629e52694...
    *-arm-linux-gnueabihf-debug.tar.gz 91183e9932c2e107... 6067411af5bfb57f...
    *-arm-linux-gnueabihf.tar.gz 6b7e8e5049846b30... 64bd6c077194d59a...
    *-osx-unsigned.dmg b0d2e99986e8ac94... c54896065868dfa1...
    *-osx64.tar.gz 88c06b4f009d7892... eb576531a1819d44...
    *-powerpc64-linux-gnu-debug.tar.gz 55f612a413399c33... 4d6a2da7b0250b15...
    *-powerpc64-linux-gnu.tar.gz f9e81e514b8a33d1... 11160996fd682d38...
    *-powerpc64le-linux-gnu-debug.tar.gz 14862e5f2925c0b8... 83988ca58c3b54cf...
    *-powerpc64le-linux-gnu.tar.gz 9df6f7c7a73f7ea8... afa04f89e42b7a22...
    *-riscv64-linux-gnu-debug.tar.gz 4c7ebb4b0b29dae2... 38d9f719bc49a20d...
    *-riscv64-linux-gnu.tar.gz 3ecd5c46eb4e7efe... bb732eccd2f3df60...
    *-win64-debug.zip 8a1a6858530ff8a8... 8896b1ad643edc58...
    *-win64-setup-unsigned.exe 1a7a08827e187490... 2b41daa5cc0d9b67...
    *-win64.zip 551e458426e9c99f... 7c1f830ce1fc6605...
    *-x86_64-linux-gnu-debug.tar.gz 6d5d800afe9370d0... 706f6cb38498e72f...
    *-x86_64-linux-gnu.tar.gz b3e3816046e6073c... 4661f5e10f7cb111...
    *.tar.gz 1c2514829bd0e839... 06ad5107d364330a...
    bitcoin-core-linux-22-res.yml c55772b82aae9480... 90fb8e3982dbb83d...
    bitcoin-core-osx-22-res.yml 493ec22f2f5371da... 54b103d2bcf4b120...
    bitcoin-core-win-22-res.yml 9d11fd74afb57ec0... 7592117df7dc7260...
    linux-build.log 974cce601787cdfe... 4cbb577dbbe594c7...
    osx-build.log eb340cf8538d6aca... 609fdd5e264b3f21...
    win-build.log 1bf905c67e8aa459... 750063037088b595...
    bitcoin-core-linux-22-res.yml.diff a91d3a57dee8afe0...
    bitcoin-core-osx-22-res.yml.diff 5cc71879fa088cca...
    bitcoin-core-win-22-res.yml.diff ff1aa350d5cab3ca...
    linux-build.log.diff f7deb97a6cbe8115...
    osx-build.log.diff 410297d8e553bfb2...
    win-build.log.diff 0eaa2ae0560c3294...
  14. DrahtBot removed the label Needs gitian build on Apr 4, 2021
  15. fanquake deleted a comment on Apr 4, 2021
  16. DrahtBot commented at 8:50 am on April 6, 2021: member

    Guix builds

    File commit 590e49ccf2af27c6c1f1e0eb8be3a4bf4d92ce8b(master) commit 58cdde629c80dcc9e47d51177f3f8ee09c514d40(master and this pull)
    *-aarch64-linux-gnu-debug.tar.gz f0a68eb9e2279c63... 3bc511fc72b3cf6f...
    *-aarch64-linux-gnu.tar.gz 13199419b3b64e98... 0bfb9e168d59c1c5...
    *-arm-linux-gnueabihf-debug.tar.gz ffe1971af4043eb6... ebd4c61b3e74753b...
    *-arm-linux-gnueabihf.tar.gz d82d03881fb3c3ca... f6e70939d3156356...
    *-osx-unsigned.dmg 0cad1e816f1150d5... 21836ea3bcfcd28c...
    *-osx-unsigned.tar.gz acaf0c3f66486b00... bd8dfb7dd03064d9...
    *-osx64.tar.gz dc4d6a00bd9a1527... e9b5872a0c68a696...
    *-powerpc64-linux-gnu-debug.tar.gz a196518428c4430e... 69f0b746650e18ab...
    *-powerpc64-linux-gnu.tar.gz 7fb19f766e4fe244... 9ddbcd91e25131ea...
    *-powerpc64le-linux-gnu-debug.tar.gz 4e2b6b4c0c7f80ba... ff3e743e6724a43d...
    *-powerpc64le-linux-gnu.tar.gz 950a04d5cc040b61... 95554fc5022bff2a...
    *-riscv64-linux-gnu-debug.tar.gz 476ceb4fbf92e658... 91fb9a3bc6479c5e...
    *-riscv64-linux-gnu.tar.gz cd74b7396c2f6d18... aba29c78fecb8b67...
    *-win-unsigned.tar.gz 4b5618315e50cd6f... 90fdf989d7f17c69...
    *-win64-debug.zip b6c0751aafd352ea... bf0a802e7d770589...
    *-win64-setup-unsigned.exe 8a36329599e8236c... 67f8d8b0072b52aa...
    *-win64.zip 0dbbf7adedb432b7... 5da5d19b59b89bf1...
    *-x86_64-linux-gnu-debug.tar.gz c900123a9ba582fc... 6d75f1c24ea4f9e1...
    *-x86_64-linux-gnu.tar.gz 93803385e1e2d2d8... c36c0688d5530ea8...
    *.tar.gz 6d6abb344d7d6d41... 21d39dcc72c60152...
    guix_build.log 5a52bb8b0e0fe6b1... df2aeb115f64e94c...
    guix_build.log.diff 147f8410d22f3589...
  17. DrahtBot removed the label Needs Guix build on Apr 6, 2021
  18. fanquake commented at 5:01 am on April 7, 2021: member
    Concept ACK. I’ve completed one (pruned) sync using this branch. @jamesob want to do some benchmarking?
  19. Squashed 'src/secp256k1/' changes from 3967d96bf1..efad3506a8
    efad3506a8 Merge #906: Use modified divsteps with initial delta=1/2 for constant-time
    cc2c09e3a7 Merge #918: Clean up configuration in gen_context
    07067967ee add ECMULT_GEN_PREC_BITS to basic_config.h
    a3aa2628c7 gen_context: Don't include basic-config.h
    be0609fd54 Add unit tests for edge cases with delta=1/2 variant of divsteps
    cd393ce228 Optimization: only do 59 hddivsteps per iteration instead of 62
    277b224b6a Use modified divsteps with initial delta=1/2 for constant-time
    376ca366db Fix typo in explanation
    1e5d50fa93 Merge #889: fix uninitialized read in tests
    c083cc6e52 Merge #903: Make argument of fe_normalizes_to_zero{_var} const
    6e898534ff Merge #907: changed import to use brackets <> for openssl
    4504472269 changed import to use brackets <> for openssl as they are not local to the project
    26de4dfeb1 Merge #831: Safegcd inverses, drop Jacobi symbols, remove libgmp
    23c3fb629b Make argument of fe_normalizes_to_zero{_var} const
    24ad04fc06 Make scalar_inverse{,_var} benchmark scale with SECP256K1_BENCH_ITERS
    ebc1af700f Optimization: track f,g limb count and pass to new variable-time update_fg_var
    b306935ac1 Optimization: use formulas instead of lookup tables for cancelling g bits
    9164a1b658 Optimization: special-case zero modulus limbs in modinv64
    1f233b3fa0 Remove num/gmp support
    20448b8d09 Remove unused Jacobi symbol support
    5437e7bdfb Remove unused scalar_sqr
    aa9cc52180 Improve field/scalar inverse tests
    1e0e885c8a Make field/scalar code use the new modinv modules for inverses
    436281afdc Move secp256k1_fe_inverse{_var} to per-impl files
    aa404d53be Move secp256k1_scalar_{inverse{_var},is_even} to per-impl files
    08d54964e5 Improve bounds checks in modinv modules
    151aac00d3 Add tests for modinv modules
    d8a92fcc4c Add extensive comments on the safegcd algorithm and implementation
    8e415acba2 Add safegcd based modular inverse modules
    de0a643c3d Add secp256k1_ctz{32,64}_var functions
    4c3ba88c3a Merge #901: ci: Switch all Linux builds to Debian and more improvements
    9361f360bb ci: Select number of parallel make jobs depending on CI environment
    28eccdf806 ci: Split output of logs into multiple sections
    c7f754fe4d ci: Run PRs on merge result instead of on the source branch
    b994a8be3c ci: Print information about binaries using "file"
    f24e122d13 ci: Switch all Linux builds to Debian
    ebdba03cb5 Merge #891: build: Add workaround for automake 1.13 and older
    3a8b47bc6d Merge #894: ctime_test: move context randomization test to the end
    7d3497cdc4 ctime_test: move context randomization test to the end
    99a1cfec17 print warnings for conditional-uninitialized
    3d2cf6c5bd initialize variable in tests
    f329bba244 build: Add workaround for automake 1.13 and older
    24d1656c32 Merge #882: Use bit ops instead of int mult for constant-time logic in gej_add_ge
    e491d06b98 Use bit ops instead of int mult for constant-time logic in gej_add_ge
    f8c0b57e6b Merge #864: Add support for Cirrus CI
    cc2a5451dc ci: Refactor Nix shell files
    2480e55c8f ci: Remove support for Travis CI
    2b359f1c1d ci: Enable simple cache for brewing valgrind on macOS
    8c02e465c5 ci: Add support for Cirrus CI
    659d0d4798 Merge #880: Add parens around ROUND_TO_ALIGN's parameter.
    b6f649889a Add parens around ROUND_TO_ALIGN's parameter. This makes the macro robust against a hypothetical ROUND_TO_ALIGN(foo ? sizeA : size B) invocation.
    a4abaab793 Merge #877: Add missing secp256k1_ge_set_gej_var decl.
    5671e5f3fd Merge #874: Remove underscores from header defs.
    db726782fa Merge #878: Remove unused secp256k1_fe_inv_all_var
    b732701faa Merge #875: Avoid casting (void**) values.
    75d2ae149e Remove unused secp256k1_fe_inv_all_var
    482e4a9cfc Add missing secp256k1_ge_set_gej_var decl.
    2730618604 Avoid casting (void**) values. Replaced with an expression that only casts (void*) values.
    fb390c5299 Remove underscores from header defs. This makes them consistent with other files and avoids reserved identifiers.
    f2d9aeae6d Merge #862: Autoconf improvements
    328aaef22a Merge #845: Extract the secret key from a keypair
    3c15130709 Improve CC_FOR_BUILD detection
    47802a4762 Restructure and tidy configure.ac
    252c19dfc6 Ask brew for valgrind include path
    8c727b9087 Merge #860: fixed trivial typo
    b7bc3a4aaa fixed typo
    33cb3c2b1f Add secret key extraction from keypair to constant time tests
    36d9dc1e8e Add seckey extraction from keypair to the extrakeys tests
    fc96aa73f5 Add a function to extract the secretkey from a keypair
    98dac87839 Merge #858: Fix insecure links
    07aa4c70ff Fix insecure links
    b61f9da54e Merge #857: docs: fix simple typo, dependecy -> dependency
    18aadf9d28 docs: fix simple typo, dependecy -> dependency
    2d9e7175c6 Merge #852: Add sage script for generating scalar_split_lambda constants
    dc6e5c3a5c Merge #854: Rename msg32 to msghash32 in ecdsa_sign/verify and add explanation
    6e85d675aa Rename tweak to tweak32 in public API
    f587f04e35 Rename msg32 to msghash32 in ecdsa_sign/verify and add explanation
    329a2e0a3f sage: Add script for generating scalar_split_lambda constants
    8f0c6f1545 Merge #851: make test count iteration configurable by environment variable
    f4fa8d226a forbid a test iteration of 0 or less
    f554dfc708 sage: Reorganize files
    3a106966aa Merge #849: Convert Sage code to Python 3 (as used by Sage >= 9)
    13c88efed0 Convert Sage code to Python 3 (as used by Sage >= 9)
    0ce4554881 make test count iteration configurable by environment variable
    9e5939d284 Merge #835: Don't use reserved identifiers memczero and benchmark_verify_t
    d0a83f7328 Merge #839: Prevent arithmetic on NULL pointer if the scratch space is too small
    903b16aa6c Merge #840: Return NULL early in context_preallocated_create if flags invalid
    1f4dd03838 Typedef (u)int128_t only when they're not provided by the compiler
    ebfa2058e9 Return NULL early in context_preallocated_create if flags invalid
    29a299e373 Run the undefined behaviour sanitizer on Travis
    7506e064d7 Prevent arithmetic on NULL pointer if the scratch space is too small
    e89278f211 Don't use reserved identifiers memczero and benchmark_verify_t
    
    git-subtree-dir: src/secp256k1
    git-subtree-split: efad3506a8937162e8010f5839fdf3771dfcf516
    bdca9bcb6c
  20. Update libsecp256k1 subtree to latest upstream master a5a447a352
  21. libsecp256k1 no longer has --with-bignum= configure option 5c7ee1b2da
  22. sipa force-pushed on Apr 23, 2021
  23. sipa commented at 7:19 pm on April 23, 2021: member
    Updated to also include the now-merged https://github.com/bitcoin-core/secp256k1/pull/906.
  24. jamesob commented at 7:23 pm on April 23, 2021: member

    @jamesob want to do some benchmarking?

    Apologies, missed this notification. Will get some going in the next day or so.

  25. DrahtBot commented at 9:32 am on May 3, 2021: member

    🕵️ @practicalswift has been requested to review this pull request as specified in the REVIEWERS file.

  26. Sjors commented at 12:30 pm on May 5, 2021: member

    I’m running a full IBD with 5c7ee1b2da6b on mainnet with -assumevalid=0 on macOS 11.3.1 (Intel). It’s definately not a benchmark, because I’m using an external USB drive for -blocksdir which seems to dramatically slow it down. But at least it’s a consensus check.

    0System: macOS 11.3, x86_64-little_endian-lp64
    1Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
    2Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
    3Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
    4Script verification uses 15 additional threads
    

    Update 2021-05-07: that took a while, but it reached the tip

  27. laanwj commented at 5:01 pm on May 12, 2021: member
    Added this to my FreeBSD node for testing. Configure phase and compilation goes without hitch at least.
  28. jamesob commented at 2:13 pm on May 17, 2021: member

    Apologies, missed this notification. Will get some going in the next day or so.

    Still planned, but debugging some build weirdness on the bench machines (bad interaction between ancient Debian and c++17 requirements).

  29. jamesob commented at 5:01 pm on May 19, 2021: member

    Happy to report that I’m seeing 1.07x speedup here.

    ibd local range 500000 510000

    commands index

    bench name command
    ibd.local.range.500000.510000 bitcoind -dbcache=300 -debug=coindb -debug=bench -listen=0 -connect=0 -addnode=127.0.0.1:8888 -prune=9999999 -printtoconsole=0 -assumevalid=0

    #21573 vs. $mergebase (absolute)

    bench name x #21573 $mergebase
    ibd.local.range.500000.510000.total_secs 2 1805.3634 (± 3.2389) 1938.7935 (± 2.3256)
    ibd.local.range.500000.510000.peak_rss_KiB 2 1701526.0000 (± 19670.0000) 1720014.0000 (± 10298.0000)

    #21573 vs. $mergebase (relative)

    bench name x #21573 $mergebase
    ibd.local.range.500000.510000.total_secs 2 1 1.074
    ibd.local.range.500000.510000.peak_rss_KiB 2 1 1.011
  30. sipa commented at 6:08 pm on May 19, 2021: member
    @jamesob Any possibility of measuring CPU time (as opposed to wall clock time)?
  31. jamesob commented at 2:32 pm on May 20, 2021: member

    Here’s the second benchmark run, reporting CPU seconds:

    #21573 vs. $mergebase (absolute)

    bench name x #21573 $mergebase
    ibd.local.range.500000.510000.total_secs 2 1809.3671 (± 3.3923) 1958.0218 (± 10.2106)
    ibd.local.range.500000.510000.peak_rss_KiB 2 1690334.0000 (± 6038.0000) 1703732.0000 (± 16204.0000)
    ibd.local.range.500000.510000.cpu_kernel_secs 2 315.8800 (± 0.5400) 328.8000 (± 5.0200)
    ibd.local.range.500000.510000.cpu_user_secs 2 10365.2350 (± 3.6150) 11663.7200 (± 12.4200)

    #21573 vs. $mergebase (relative)

    bench name x #21573 $mergebase
    ibd.local.range.500000.510000.total_secs 2 1 1.082
    ibd.local.range.500000.510000.peak_rss_KiB 2 1 1.008
    ibd.local.range.500000.510000.cpu_kernel_secs 2 1 1.041
    ibd.local.range.500000.510000.cpu_user_secs 2 1 1.125
  32. sipa commented at 1:06 am on June 7, 2021: member
    Anything left to do here?
  33. laanwj commented at 3:02 pm on June 7, 2021: member
    I don’t think so. Tested ACK 5c7ee1b2da6bf783d27034fca9dfd3a64ed525cb
  34. laanwj merged this on Jun 7, 2021
  35. laanwj closed this on Jun 7, 2021

  36. sidhujag referenced this in commit 35224f3f5a on Jun 9, 2021
  37. UdjinM6 referenced this in commit bc61867454 on Aug 10, 2021
  38. 5tefan referenced this in commit dd45c616b6 on Aug 12, 2021
  39. gwillen referenced this in commit 3c8e88e7cf on Jun 1, 2022
  40. hebasto referenced this in commit 1d1546e4c2 on Jun 28, 2022
  41. MarcoFalke referenced this in commit 72d6469ab4 on Jun 29, 2022
  42. sidhujag referenced this in commit 30fc0ceef5 on Jun 29, 2022
  43. janus referenced this in commit 4e0802c858 on Aug 4, 2022
  44. DrahtBot locked this on Aug 18, 2022

github-metadata-mirror

This is a metadata mirror of the GitHub repository bitcoin/bitcoin. This site is not affiliated with GitHub. Content is generated from a GitHub metadata backup.
generated: 2024-12-21 15:12 UTC

This site is hosted by @0xB10C
More mirrored repositories can be found on mirror.b10c.me