l0rinc
commented at 12:52 pm on February 24, 2024:
contributor
To mitigate the quadratic complexity inherent in the sequential byte division of the Base58 encoding process, this update aggregates characters into groups of 7 and 9, fitting them into 64-bit long integers.
This refinement utilizes the inherent efficiency of native division and modulo operations on 64-bit architectures, significantly optimizing computational overhead.
The benchmarks indicate a ~4.4x speedup for Base58Encode:
make && ./src/bench/bench_bitcoin –filter=Base58Encode –min-time=1000
before:
ns/byte
byte/s
err%
total
benchmark
61.00
16,393,083.32
0.2%
1.07
Base58Encode
after:
ns/byte
byte/s
err%
total
benchmark
13.79
72,502,616.20
0.1%
1.10
Base58Encode
The same was applied to decoding, resulting in a ~2x speedup:
make && ./src/bench/bench_bitcoin –filter=Base58Decode –min-time=1000
l0rinc renamed this:
WIP optimization: Enhance Base58 conversion through intermediary Base56 conversion
optimization: Enhance Base58 conversion through intermediary Base56 conversion by 300%
on Feb 24, 2024
l0rinc renamed this:
optimization: Enhance Base58 conversion through intermediary Base56 conversion by 300%
optimization: Speed up Base58 conversion through intermediary Base56 conversion by 300%
on Feb 24, 2024
l0rinc marked this as ready for review
on Feb 24, 2024
l0rinc force-pushed
on Feb 24, 2024
DrahtBot added the label
CI failed
on Feb 24, 2024
DrahtBot
commented at 6:12 pm on February 24, 2024:
contributor
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.
Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
Leave a comment here, if you need help tracking down a confusing failure.
137+ std::string result;
138+ result.reserve(leadingZeros + size);
139+ result.assign(leadingZeros, '1');
140+
141+ auto inputAsLongs = ConvertToLongs(input, leadingZeros);
142+ for (auto i{0U}; i < inputAsLongs.size();) {
optout21
commented at 5:40 am on February 25, 2024:
This construct with the two nested loops is not evident. Maybe some comment would be useful. The inner loop does not depend on the value of i. Is it repeated for as long as all inputAsLongs elements are reduced to 0?
Fantastic comment, the internal loop should skip the leading zeros, indeed.
l0rinc marked this as a draft
on Feb 25, 2024
l0rinc force-pushed
on Feb 25, 2024
l0rinc force-pushed
on Feb 25, 2024
DrahtBot removed the label
CI failed
on Feb 25, 2024
l0rinc force-pushed
on Feb 25, 2024
l0rinc force-pushed
on Feb 25, 2024
DrahtBot added the label
CI failed
on Feb 25, 2024
DrahtBot
commented at 3:24 pm on February 25, 2024:
contributor
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.
Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
Leave a comment here, if you need help tracking down a confusing failure.
DrahtBot removed the label
CI failed
on Feb 25, 2024
l0rinc requested review from optout21
on Feb 25, 2024
l0rinc requested review from Empact
on Feb 25, 2024
l0rinc marked this as ready for review
on Feb 25, 2024
l0rinc force-pushed
on Feb 26, 2024
l0rinc renamed this:
optimization: Speed up Base58 conversion through intermediary Base56 conversion by 300%
optimization: Speed up Base58 conversion through intermediary Base56 conversion by 400%
on Feb 26, 2024
l0rinc force-pushed
on Feb 26, 2024
Empact
commented at 4:59 am on February 27, 2024:
contributor
I have some questions about this change, but to start I would encourage you to make the change a minimal one. I.e. avoid strictly unnecessary changes, such as those in the last commit. Merge the variable name changes with the commit they are introduced, and don’t change other lines in the file for conventions.
The goal, again, is to ease review and avoid cause for controversy. Leaving the code unchanged is a good default if not significantly justified.
l0rinc renamed this:
optimization: Speed up Base58 conversion through intermediary Base56 conversion by 400%
optimization: Speed up Base58 encoding with 64-bit packing by 400%
on Feb 27, 2024
l0rinc force-pushed
on Feb 27, 2024
fanquake
commented at 10:45 am on February 27, 2024:
member
Pleas stop leaving endless self review comments on PRs. It’s not only unecessary, but everytime you comment here, it sends 4000 people an email. Some commentary / responding to review comment is obviously fine, but what you are currently doing is just turning the thread into a spammy/confusing mess, and I’d assume, making things potentially harder for reviewers.
l0rinc renamed this:
optimization: Speed up Base58 encoding with 64-bit packing by 400%
optimization: Speed up Base58 encoding by 400% by 64-bit preliminary byte packing
on Feb 27, 2024
DrahtBot renamed this:
optimization: Speed up Base58 encoding by 400% by 64-bit preliminary byte packing
optimization: Speed up Base58 encoding by 400% by 64-bit preliminary byte packing
on Feb 27, 2024
Empact
commented at 5:25 pm on February 27, 2024:
contributor
@paplorinc agree with @fanquake, a better place to articulate your motivations and reasoning is in the commit messages, that way they’ll be retained for future reference via git blame, etc.
l0rinc renamed this:
optimization: Speed up Base58 encoding by 400% by 64-bit preliminary byte packing
optimization: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
l0rinc force-pushed
on Mar 6, 2024
DrahtBot added the label
CI failed
on Mar 6, 2024
DrahtBot
commented at 5:52 pm on March 6, 2024:
contributor
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.
Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
Leave a comment here, if you need help tracking down a confusing failure.
l0rinc marked this as ready for review
on Mar 7, 2024
DrahtBot removed the label
CI failed
on Mar 7, 2024
l0rinc force-pushed
on Mar 7, 2024
l0rinc force-pushed
on Mar 7, 2024
l0rinc renamed this:
optimization: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing
refactor: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing
on Mar 8, 2024
DrahtBot added the label
Refactoring
on Mar 8, 2024
l0rinc force-pushed
on Mar 8, 2024
l0rinc force-pushed
on Mar 9, 2024
l0rinc force-pushed
on Mar 9, 2024
in
src/test/data/base58_encode_decode.json:16
in
ffb81c978aoutdated
I’ve added the random test cases that failed during development to make sure they’re covered deterministically.
Also added ones which are testing the boundaries of powers of 58, since those cases happen less frequently in the randomized tests.
Yes, I meant you can open a new pull request with the added tests, if you want.
l0rinc force-pushed
on May 7, 2024
l0rinc force-pushed
on May 8, 2024
l0rinc marked this as a draft
on May 8, 2024
l0rinc marked this as ready for review
on May 8, 2024
l0rinc force-pushed
on May 11, 2024
l0rinc force-pushed
on May 29, 2024
l0rinc force-pushed
on May 29, 2024
DrahtBot added the label
Needs rebase
on Jun 12, 2024
l0rinc force-pushed
on Jun 13, 2024
DrahtBot removed the label
Needs rebase
on Jun 13, 2024
achow101 referenced this in commit
fcc3b653dc
on Jun 13, 2024
l0rinc renamed this:
refactor: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing
optimization: Speed up Base58 encoding/decoding by 400%/200% via preliminary byte packing
on Jun 23, 2024
l0rinc
commented at 10:45 am on July 1, 2024:
contributor
@Sjors, this is the optimization I’ve mentioned to you in person, I’d appreciate a review if you think it’s worthwhile:
maflcko
commented at 12:48 pm on July 1, 2024:
member
I don’t think this is used in a critical path, to warrant a speedup?
l0rinc
commented at 12:58 pm on July 1, 2024:
contributor
I don’t think this is used in a critical path, to warrant a speedup?
maflcko
commented at 1:05 pm on July 1, 2024:
member
What you link to is supporting my point, is it not? #12704 (comment)
Stop wasting time on discussing the performance. This does not matter. Decoding an address could take 50 us and I don’t think anyone would notice.
If the resulting code looks better, go for it. Otherwise, don’t.
-0
l0rinc
commented at 1:09 pm on July 1, 2024:
contributor
What’s the point of the benchmarks in that case exactly?
Anyway, I can make the code slightly simpler and similarly performant, would that be welcome?
maflcko
commented at 1:09 pm on July 1, 2024:
member
Ok, the reason for #7656#issue-139438690 was an improvement in listunspent. Seems fine, if this is still the case. But this will need to be checked first.
@paplorinc The reason you’re getting some pushback on your PRs is because review is the biggest bottleneck for our project, which makes contributors and maintainers weary about proposed changes that demand review (which could be spent elsewhere) but don’t result in observable benefits (even if they improve microbenchmarks). I hope you don’t conclude from this that we don’t care about performance, but it may take some experience with the codebase and the project to figure out what people consider important. In that sense, reviewing things like #28280 are far more impactful, so thank you for that.
As long as it doesn’t introduce a lot of complexity, I don’t think a pure performance improvement is an inherently bad change that we should reject, but it is a reason why you may not get much attention from reviewers. So (speaking just for myself), this isn’t “don’t make such changes!”, but a “be warned that people may not care about this enough for it go anywhere”.
l0rinc
commented at 1:47 pm on July 1, 2024:
contributor
Thanks @sipa, appreciate your comment!
How do I find small changes that are needed, welcome, not so controversial, and don’t need 3 years of up-front studying?
That’s why I like optimizations, since the desired behavior is already in the code and test and benchmarks and history, and I can have visible progress while learning about the project.
maflcko
commented at 1:59 pm on July 1, 2024:
member
How do I find small changes that are needed, welcome, not so controversial, and don’t need 3 years of up-front studying?
You can spend time on in-depth review of open pull requests and compare your review with the review done by others. This may be time-consuming (Yes, some pull requests may need up-front studying, but there are many that do not). Ideally the process will teach you relevant skills for this project and for yourself to apply elsewhere. Moreover, on some pull requests there remain small follow-ups that were found during review, so you’ll naturally find stuff to work on if you start out by reviewing pull requests done by others.
andrewtoth
commented at 2:00 pm on July 1, 2024:
contributor
How do I find small changes that are needed, welcome, not so controversial, and don’t need 3 years of up-front studying?
@paplorinc I know you are past your first PR and issue, but most of the issues with this label would fit that criteria https://github.com/bitcoin/bitcoin/labels/good%20first%20issue.
Also, browsing open issues I’m sure you could find others that fulfill most of the criteria as well.
l0rinc
commented at 2:24 pm on July 1, 2024:
contributor
@andrewtoth, that was the first thing I checked, couldn’t find any reasonable “good first issue” - or other issue that piqued my interest. The problem has to be personal, otherwise the solution won’t be honest.
@maflcko, you already suggested that multiple times, but my question wasn’t how to review PRs.
I’ll check if this PR improved listunspent, as suggested, if not, I’ll just close this.
andrewtoth
commented at 2:28 pm on July 1, 2024:
contributor
@paplorinc#28945 was abandoned and I don’t think @fjahr has taken it over. Perhaps that would be interesting to you. Maybe check in that PR if anyone is working on it if you are.
fjahr
commented at 2:46 pm on July 1, 2024:
contributor
andrewtoth
commented at 2:48 pm on July 1, 2024:
contributor
Ahh sorry thanks.
l0rinc force-pushed
on Jul 1, 2024
l0rinc
commented at 8:37 pm on July 1, 2024:
contributor
Ok, the reason for #7656#issue-139438690 was an improvement in listunspent. Seems fine, if this is still the case. But this will need to be checked first.
I’ve created 10k blocks in regtest and measured the performance of listunspent before and after:
i.e. 55% faster listunspent for the given usecase.
l0rinc force-pushed
on Jul 1, 2024
l0rinc force-pushed
on Jul 1, 2024
l0rinc force-pushed
on Jul 1, 2024
l0rinc force-pushed
on Jul 1, 2024
l0rinc force-pushed
on Jul 2, 2024
maflcko
commented at 7:37 am on July 2, 2024:
member
I presume, if you wanted to speed up listunspent in this case of a bip-32 base58 encoded descriptor string, a higher speed-up could be achieved by caching the descriptor string results, because the expectation is that there are generally only a few (or one) per wallet. But I am not sure how easy that is, or whether it is worth it.
l0rinc
commented at 7:42 am on July 2, 2024:
contributor
I was following the benchmarks to decide what to work on, but since you’ve mentioned this is an important usecase, I measured it as well.
Is there anything else you’d like me to measure or change in this pull request?
maflcko
commented at 8:04 am on July 2, 2024:
member
I don’t think I said that “this is an important usecase”. I simply said that the reason for the improvement in #7656#issue-139438690 was listunspent. That pull request was created before bech32(m) addresses and descriptor wallets existed. Considering that bech32 is the default address type and that descriptor wallets are the default as well, the only expected remaining use of base58 should be in the bip-32 descriptor string. My guess is that, if it was important to optimize that for speed, a higher speedup could be achieved by caching the generated descriptor string.
However, this is just background information and a guess. I don’t know if it is true, how easy it is to implement, whether it is worth it to review, or worth it at all.
l0rinc
commented at 11:48 am on July 12, 2024:
contributor
The algorithm used in libbase58 is different, so not sure this even applies. Kinda doubt it would be worth the time to port/review either, unless a library consumer cares about performance or the other criteria explained already here.
DrahtBot removed the label
CI failed
on Jul 18, 2024
achow101 removed review request from optout21
on Oct 15, 2024
achow101 requested review from achow101
on Oct 15, 2024
achow101
commented at 11:05 pm on November 4, 2024:
member
As this is basically a complete rewrite of the base58 encoding and decoding code, I don’t think the review effort required to review this is worth it relative to the unimportance of these functions.
l0rinc
commented at 10:16 am on November 5, 2024:
contributor
Thanks for checking it out @achow101.
I could make the diff minimal, but given that it’s not that important, I will leave it as is - until someone starts complaining that listunspent is slow :)
This is a metadata mirror of the GitHub repository
bitcoin/bitcoin.
This site is not affiliated with GitHub.
Content is generated from a GitHub metadata backup.
generated: 2024-11-23 09:12 UTC
This site is hosted by @0xB10C More mirrored repositories can be found on mirror.b10c.me