Feature request: Make Bitcoin libFuzzer-friendly and consider integration into the OSS-Fuzz project

practicalswift commented at 8:53 pm on May 8, 2017: contributor

Making Bitcoin ready for inclusion into the OSS-Fuzz project should be fairly straightforward:

Have a look at test_bitcoin_fuzzy.cpp and read the Bitcoin fuzzing documentation.
Make test_bitcoin_fuzzy.cpp libFuzzer-friendly by implementing LLVMFuzzerTestOneInput. See the libFuzzer documentation for details.
Create the three configuration files project.yaml, Dockerfile and build.sh as described in the OSS-Fuzz new project guide and submit an OSS-Fuzz pull request.

Today Google announced that they offer $1k to $20k in rewards for properly integrated project.

Have a nice vacation and enjoy the warm fuzzy feeling of having helped secure the Bitcoin network! :-)

practicalswift renamed this:
~~Feature request: Make Bitcoin libFuzzer-friendly and request inclusion into the OSS-Fuzz project ($1k to $20k bounty)~~
Feature request: Make Bitcoin libFuzzer-friendly and request integration into the OSS-Fuzz project ($1k to $20k bounty)
on May 8, 2017

TheBlueMatt commented at 9:08 pm on May 8, 2017: member

If we wanted to do this we’d have to get them to exempt us from their usual disclosure policies - their timelines for disclosure are absurdly agressive to risk the safety of users rather significantly.

practicalswift commented at 9:27 pm on May 8, 2017: contributor

@TheBlueMatt Are you referring to the 90-day deadline?

I assume that bitcoind is currently being actively fuzzed by both black hats and white hats, so even in the presence of a strict 90-day deadline I think that the net effect of adding OSS-Fuzz/ClusterFuzz to the mix would simply be an increase the probability that the next critical bug is found by white hats (in the form of core team members).

Think of it as adding essentially free hash rate to the good guys :-)

TheBlueMatt commented at 9:28 pm on May 8, 2017: member

@practicalswift more fuzzing is good, but we do not publicly announce bugs even after they have been fixed for some time. IIRC the OSS-Fuzz shit announces bugs a week or two after a public fix.

practicalswift commented at 9:42 pm on May 8, 2017: contributor

@TheBlueMatt These are the bug disclosure guidelines:

Deadline. After notifying project authors, we will open reported issues to the public in 90 days, or 30 days after the fix is released (whichever comes earlier).
Weekends and holidays. If a deadline is due to expire on a weekend, the deadline will be moved to the next normal work day.
Grace period. We have a 14-day grace period. If a 90-day deadline expires but the upstream engineers let us know before the deadline that a patch is scheduled for release on a specific day within 14 days following the deadline, the public disclosure will be delayed until the availability of the patch.

sipa commented at 9:49 pm on May 8, 2017: member

I think this discussion is premature.

At this point, our fuzzing infrastructure is nearly worthless. It’s a proof of concept that only tests a tiny part of the serialization/deserialization routines. I’m all for improving that, but the order should be:

Incorporate meaningful fuzzing tests (for example, including things like validating blocks received by a fuzzer with the cryptographically hard parts disabled).
Add buildsystem support for building against libfuzzer. This is useful independently from the OSS-fuzz project, as AFL and libfuzzer tend to have different strategies, and may find different problems.
Then we can have a discussion about submitting to OSS-Fuzz and whether their policy is acceptable.

TheBlueMatt commented at 9:53 pm on May 8, 2017: member

@practicalswift yes, announcing bugs with exploit guidelines 30 days after a fix is released would put a ton of our users at massive risk.

RHavar commented at 8:58 pm on May 9, 2017: contributor

@practicalswift yes, announcing bugs with exploit guidelines 30 days after a fix is released would put a ton of our users at massive risk.

Sure. However it’s still a hell of a lot better than being willfully ignorant of vulnerabilities, that could be independently found anyway. But yeah, it’d be great if they extended that deadline for bitcoin but if not, I don’t see the big deal. And the damages of public disclosure can be heavily mitigated by a few days prior widely publizing the need to update.

TheBlueMatt commented at 9:08 pm on May 9, 2017: member

@RHavar I’m not advocating that we dont run fuzzing, as sipa pointed out, we have a framework for it that does get a bunch of CPU hours (despite it not being so useful), but the nature of Bitcoin is somewhat different than your standard software (and the OSS-Fuzz guys seem to have a very google/large-provider-centric view of the OSS world).

practicalswift commented at 9:06 am on May 24, 2017: contributor

FWIW, I’ve been running some libFuzzer-based bitcoin fuzzing for a while using the code in #10440 so it should be ready for review :-)

practicalswift renamed this:
~~Feature request: Make Bitcoin libFuzzer-friendly and request integration into the OSS-Fuzz project ($1k to $20k bounty)~~
Feature request: Make Bitcoin libFuzzer-friendly and consider integration into the OSS-Fuzz project
on May 24, 2017

practicalswift commented at 8:35 pm on May 26, 2017: contributor

test_bitcoin_fuzzy currently supports fuzz testing all the deserialization routines used in ProcessMessage(…, vRecv [tainted], …) with three exceptions. These three are missing:

BlockTransactions deserialization
BlockTransactionsRequest deserialization
CBlockHeaderAndShortTxIDs deserialization

I’ve submitted PR #10409 which adds support for fuzzing these three as well.

laanwj referenced this in commit 9e8ef9d991 on Oct 5, 2017

fanquake added the label Tests on Sep 6, 2018

MarcoFalke added the label Brainstorming on Dec 25, 2018

MarcoFalke commented at 6:43 pm on December 25, 2018: member

I’ve submitted PR #10409 which adds support for fuzzing these three as well.

The pr is merged, so closing this.

MarcoFalke closed this on Dec 25, 2018

MarkusTeufelberger commented at 11:56 pm on December 26, 2018: none

Fuzzing these 3 deserializations is not the topic of this issue though. It is not resolved at all and not much has moved forward in 2018 on the fuzzing front within this repository.

MarcoFalke commented at 5:27 pm on December 27, 2018: member

Make the fuzzers nice and libFuzzer-friendly is separate from integration into OSS-Fuzz and should not be mixed up in the same issue. If there is anything left to discuss or track, please open an issue (or better a pull request to improve the fuzzers).

MarkusTeufelberger commented at 7:00 pm on December 27, 2018: none

This issue is perfectly fine and it hasn’t been resolved, so please reopen this one.

MarcoFalke reopened this on Dec 27, 2018

practicalswift commented at 9:17 am on January 26, 2019: contributor

Somewhat related: The Solidity compiler is now continuously fuzzed with OSS-Fuzz (https://twitter.com/ibags/status/1088869900993613824).

arvidn commented at 12:45 pm on February 8, 2019: none

somewhat on topic. I just started to familiarise myself with the fuzzing framework. It seems quite odd to me to have a single fuzzing binary and input corpus.

Any modern fuzzer (at least libfuzzer) is taking code coverage into account when generating new input. By joining all APIs to be fuzzed into a single call, and use the first 32 bit to switch between them, I’m sure we make genetic algorithm looking for inputs less efficient.

As far as I can tell, it would be a lot more efficient to use separate binaries for each API to be fuzzed, and to use a separate corpus for each of them. It doesn’t make much sense to test all inputs against all APIs.

practicalswift commented at 1:04 pm on February 8, 2019: contributor

@arvidn We’re no longer using a single binary. This was fixed by @MarcoFalke in #15043 :-)

practicalswift commented at 1:06 pm on February 8, 2019: contributor

@arvidn Out of curiosity: what are your fuzzing plans? Are you considering adding Bitcoin Core to OSS-Fuzz? That would be truly awesome!

MarkusTeufelberger commented at 1:33 pm on February 8, 2019: none

FYI, the back-end to OSS-Fuzz has been released yesterday, so it would now be possible for the Bitcoin project to (sorta) self-host it. There’s still relatively tight integration with Google services, but they can be replaced over time: https://opensource.googleblog.com/2019/02/open-sourcing-clusterfuzz.html

Directly using OSS-Fuzz likely is not going to happen, since Bitcoin maintainers disagree with that project’s disclosure policy.

arvidn commented at 2:02 pm on February 8, 2019: none

@practicalswift I’m working on transaction compression, specifically for FIBRE. So I’m looking to add a fuzzer for that code. It may be some ways out to be up-streamed.

But it seems I should rebase on top of latest bitcoin core

practicalswift commented at 2:19 pm on February 8, 2019: contributor

@MarkusTeufelberger I’m not sure that the consensus opinion is that OSS-Fuzz is a bad idea. @TheBlueMatt raised his concerns while @sipa thought the discussion was premature.

Personally I’m strongly in favour:

Given how many CPU hours that have already been put into fuzzing Core it is highly unlikely that OSS-Fuzz will find any existing issues. However it will make sure we catch any newly introduced issues in a timely manner before they are included in a release and risk being exploited by adversaries (who can safely be assumed to be fuzzing heavily without disclosing their findings at all!).

Please point out any flaws in my thinking here.

practicalswift commented at 2:34 pm on February 8, 2019: contributor

@arvidn That’s excellent! Looking forward to seeing it upstreamed!

yevgenypats commented at 10:35 am on June 5, 2019: none

Hi @arvidn, @practicalswift and @MarcoFalke ,

I’m the founder of Fuzzit. We are an OSSFuzz like solution though we don’t have disclosure policies and you make your own rules. of course we are free for open-source.

I’ll be happy to contribute a pull-request.

Cheers, Yevgeny

MarkusTeufelberger commented at 11:15 am on June 5, 2019: none

So you are a proprietary platform from the middle east in alpha with no public documentation that promises to keep bugs found in Bitcoin code secret?

yevgenypats commented at 11:21 am on June 5, 2019: none

Hi @MarcoFalke, It was an offer. If the best thing for you is having an on-premise/self-hosted solution because of the high sensitivity of the project is your way to go then you can just say it.

Cheers, Yevgeny

MarcoFalke commented at 11:28 am on June 5, 2019: member

I think we are primarily lacking contributions for meaningful fuzz targets, not necessarily the resources to run them.

I do appreciate the offer to fuzz for free for open-source. And complaining about the trustworthiness of a platform might not help that much, because eventually any (potentially shady) person can fork the source code and set up their own account with whatever fuzz service they want to.

yevgenypats commented at 11:54 am on June 5, 2019: none

@MarcoFalke Thanks for the response. We have quite an experience with writing fuzz targets. I can try and add a pull-request for a meaningful fuzz target. I don’t know if this is the right place or you are the right person but I’ll be happy for a retweet of-course after the contribution is approved&merged. This will help a small startup like ours.

PastaPastaPasta referenced this in commit a58f3ecefa on Jan 31, 2020

PastaPastaPasta referenced this in commit 50a6d9409f on Jan 31, 2020

PastaPastaPasta referenced this in commit 187c4d0015 on Feb 4, 2020

PastaPastaPasta referenced this in commit fe333e188a on Feb 9, 2020

MarcoFalke commented at 0:35 am on April 17, 2020: member

I am closing this issue for now. It has several threads of unrelated discussion going on, which are either no longer applicable today or remain questionable and unsolved.

Discussion about a test harness https://github.com/bitcoin/bitcoin/blob/master/src/test/test_bitcoin_fuzzy.cpp: We now have a full fuzz suite and this harness no longer exists in that form.
Discussion about the libfuzzer integration. This has already happened https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md we support afl and libfuzzer. Support for more fuzz engines is welcome.
Discussion about the disclosure policy of oss-fuzz: https://github.com/bitcoin-core/secp256k1/issues/739#issuecomment-614831028 explains why the strict public disclosure policy is unsuitable for this project.

MarcoFalke closed this on Apr 17, 2020

ckti referenced this in commit 7ccabae116 on Mar 28, 2021

malbit referenced this in commit 7ee2b75353 on Nov 5, 2021

DrahtBot locked this on Feb 15, 2022

Feature request: Make Bitcoin libFuzzer-friendly and consider integration into the OSS-Fuzz project #10364