Add `blockfilterindex = p2wpkh`

nopara73 commented at 1:00 pm on February 28, 2020: none

I would like to have a p2wpkh blockfilterindex type to outsource Wasabi’s filter generation to Bitcoin Core, as our filter generation is orders of magnitude slower and we would like to bring this to the client side too, so our users with full nodes would be able to validate that our server does not cheat.

I previously created a blockfilterindex extensibility proposal to enable filter type combinations: #18221. So the followup PR of this issue will make sure it adheres to it. Since the PR is ready already, I’ll omit to go into the implementation details in this issue, rather concentrate on the goal, the benefits of end users and how we ruled out the alternative solutions.

The Goal

We would like to enable our users to upgrade Wasabi to be a full node with one click.

This feature is already implemented and rolled out to production (shipping with Core, fee estimation, tx broadcasting, mempool serving etc..) except the (most important) validation part of it is yet to be added.

Alternative Solutions

I’ve been long struggling with how to enable this for our users and we ended up working together with a brilliant dev, @dangershony, who maintained the C# Bitcoin full node for years. Unfortunately we had to rule out all the possible alternatives.

One alternative would be using importmulti, but we would not like to rely on the wallet feature of Bitcoin Core, rather stick to the node functionality.

Another alternative could be to slowly replace our filters with Bitcoin Core’s filters on the client side, but that would require reindexing all of our wallets, which in some cases could take half a day. Also mixing filter types feels dirty and can result in unexpected issues.

We also considered bringing our own filter generation algorithm to the client side, but this is one of the oldest brain class of our codebase that serves a critical functionality so touching this could be problematic. It currently takes 2-3 days for us to create our filters on a powerful server, even though we are using p2wpkh filters and only building them from segwit activation. Bitcoin Core creates these filters within 2 hours. Why we are so slow is out of the scope of this issue, but here’s the clue: input scripts aren’t part of a transaction.

Further we considered just blindly going through all the blocks and process all the transactions and make sure the server doesn’t lie. The performance here is also unacceptable.

Finally we would also don’t like to use bloom filters, as it is on the edge of obsolation and having multiple filters to achieve the same purpose feels dirty from a complexity point of view.

Benefits to the End User

@gmaxwell has suggested to articulate the benefits to the end user for changes to the codebase, thus this section.

Adding new filter types and filter type combinations (https://github.com/bitcoin/bitcoin/issues/18221) would enable a privacy oriented light wallet surge to be built on top of Bitcoin Core.

Maybe more importantly Bitcoin Core providing the filters to Wasabi enables a clean architecture and a seamless UX for users willing to run a full node that comes with Wasabi.

It is quite unique in a sense that no other popular light wallets went to the extent of building a full node into their software, especially not in a way that only requires a a single click configuration, so if our feature gets out of “work in progress” phase there may be an influx of full node users, which is good for the network.

What we stand to lose is that this full node feature will be delayed into the future so far that I may not be here when the next clean solution presents itself and maybe the Wasabi team won’t be willing to work on it as frankly it has little business value.

What’s Next?

This issue described our motivation and the context of the feature request, our pull request is coming up soon, so we discuss the specifics.

nopara73 added the label Feature on Feb 28, 2020

MarcoFalke commented at 1:53 pm on February 28, 2020: member

Can you explain why the existing filter does not work with wasabi? If the only concern is that wasabi currently uses a different (their own) filter type, and that updating a wasabi wallet to the BIP 158 filter type takes too much time (half a day), it could make sense to just have two wasabi wallet versions. New wallets can use the new version and old wallets can be upgraded by the user after educating them it might take a day.

nopara73 commented at 2:03 pm on February 28, 2020: none

It’s a matter of ~5GB vs ~200MB filters. This difference presents itself on the user’s computer in storage, initial sync time (+the accompanying bandwidth usage) and filter processing time.

MarcoFalke commented at 2:17 pm on February 28, 2020: member

The blockfilterindex is constructed locally in a background thread, so there shouldn’t be any ibd slowdown or increased bandwidth usage. Also, if the user needs the BIP 158 filter anyway, they might be better off with just that one filter in terms of disk usage.

nopara73 commented at 2:23 pm on February 28, 2020: none

I thought you suggested that Wasabi start serving Bitcoin Core filters, so I reflected to that idea. Your suggestion seems to be the one where Wasabi serves its own filters to the users and users with full nodes are slowly replacing Wasabi filters with Core filters and as Core filters take over it would work with that instead of Wasabi filters. Sorry, I wasn’t clear about the hybrid nature of our full node mode implementation. Wasabi is a light wallet until the full node catches up.

I described this scenario in my post, so let me quote myself:

Another alternative could be to slowly replace our filters with Bitcoin Core’s filters on the client side, but that would require reindexing all of our wallets, which in some cases could take half a day. Also mixing filter types feels dirty and can result in unexpected issues.

Sjors commented at 2:50 pm on February 28, 2020: member

How much bigger would a p2wpkh + p2wsh filter be? Privacy wise it’s nice for wallets that need p2wsh to piggy-back on a commonly used filter.

nopara73 commented at 3:04 pm on February 28, 2020: none

@Sjors I agree. Please review this extensibility proposal on how to combine the two filter types: #18221

If you would suggest pre-defined combinations however I think it may not be a good idea due to the exponentially growing possibilities in these combinations (or permutation without order?) and I would like to avoid trying to predict what combinations are the most likely to be used together in the future. Smells like a can of worms that I prefer not to open :)

Sjors commented at 3:12 pm on February 28, 2020: member

I don’t expect exponential growth here. p2wpkh + p2wsh covers all of SegWit v0, which is great for wallets that don’t want to deal with pre-SegWit legacy. The next filter I would expect is for SegWit v1 by the time that comes out. Again I can see how future wallets would use SegWit v1 and not bother with v0 or legacy. Having access to smaller size filters, and with it a lower false positive rate, is a nice incentive to adopt something more modern (especially on mobile).

Too many filters wouldn’t be good, because DNS seeds have to announce them, and because it adds fingerprint bits.

nopara73 commented at 3:44 pm on February 28, 2020: none

If a wallet supports segwitv0 then it will still want to be able to still serve segwit0 even if segwitv1 is introduced to ensure backwards compatibility. The best approach until the transition period could be to serve segwitv0+segwitv1 combinations, in fact wrappedsegwit wouldn’t be an unreasonable request either, so add that one to the mix. With this we are already at a huge complexity with 7 different filter types: segwitv0, segwitv1, wrappedsegwit, segwitv0+segwitv1, segwitv0+wrappedsegwit, segwitv1+wrappedsegwit and segwitv0+segwitv1+wrappedsegwit. And this is just how much we could predict within a few minutes of conversation while completely disregarding any future innovation or other usecases, like what if I just want to filter my own keys on my own laptop in a performant way like what we want to do with Wasabi. It’s really a can of worms… #18221 explains how to avoid it.

Too many filters wouldn’t be good, because DNS seeds have to announce them, and because it adds fingerprint bits.

I thought it is not going to be added to the P2P protocol? Also can you elaborate on this? I’m not sure I understand.

MarcoFalke commented at 4:16 pm on February 28, 2020: member

I thought it is not going to be added to the P2P protocol? Also can you elaborate on this? I’m not sure I understand.

BIP 157 explains how the filters can be fetched over a P2P connection. Currently, BIP 157 uses one byte to uniquely identify the filter type. The bits in the byte are currently not interpreted separately. Also, to make it possible for wallets that depend on the BIP 158 filter type, nodes supporting it will set a service bit.

Since there are already nodes running BIP 157+158, it is not trivial to just add a new filter type to the existing BIPs. It might be required to write a new BIP for the new filter type and signal it on a new service bit.

However, introducing a bunch of new filter types (e.g. each script type for each segwit version) needs to be weighted against the downsides (cost of service bit, cost on full nodes to create the index).

Again, it would be useful to know why the existing BIP 158 filters (that work on the raw scriptPubKey, thus work for any segwit version and script type) are not sufficient for Wasabi’s use case.

Sjors commented at 4:18 pm on February 28, 2020: member

I guess you’re focussed on the RPC level here (especially given that p2p support in #16442 is not merged yet). I was thinking p2p earlier. In that case I’m fine with adding p2wphk, though I agree with @MarcoFalke that it doesn’t seem very useful at first glance, especially without pruning.

Bitcoin DNS seeds announce which services are supported by nodes, and they allow querying for specific features, e.g. only nodes that serve block filters: https://github.com/sipa/bitcoin-seeder/pull/77/files (try dig -t A x49.seed.bitcoin.sprovoost.nl).

If there’s plethora of different block filters out there, then either nodes have to play a trial and error game based on everything returned by x49, or we have to introduce additional P2P flags, that the DNS seeds can then filter. Either way doesn’t really scale to more than a small number of filters. That’s in additional to the privacy drawback, because which block filter(s) a wallet asks or a node offers tells something.

Being able to combine filters doesn’t solve the above problems, though tailer made filters may be useful for a local wallet talking to a local node. Under the hood and at the RPC level we could have more fine grained filters like you suggest in #18221, but at the p2p level I suggest serving a very limited set.

dangershony commented at 4:31 pm on February 28, 2020: contributor

I tend to agree with the logic behind a segwit0 filter type, I can compile an alternative PR that will build the segwit0 indexes if that has a better chance to be accepted.

nopara73 commented at 5:31 pm on February 28, 2020: none

I see. If P2P is planned to be supported then the filter combination idea is not viable, in this case we must be careful with not overinflating the number of filter types, so ACK for segwit0 filters those combine P2WPKH and P2WSH makes perfect sense.

@MarcoFalke Again, it would be useful to know why the existing BIP 158 filters (that work on the raw scriptPubKey, thus work for any segwit version and script type) are not sufficient for Wasabi’s use case.

Sorry but which use case are you talking about? I addressed two in my previous reply to you:

Backend: Wasabi Filters -> Client: Core Filters
Backend: Core Filters -> Client: Core Filters

(And this PR is asking for Backend: Core Segwit Filters -> Client: Core Segwit Filters)

MarcoFalke commented at 5:38 pm on February 28, 2020: member

Yes, it wasn’t clear from your previous reply why everyone, the Wasabi Backend, the Core Backend, and the client can not simply use the BIP 158 BASIC filter type.

You mentioned disk size is a problem, but I don’t see how that is true given that the wasabi wallet may just drop filters that are before the wallet’s day of birth.

schildbach commented at 11:44 pm on February 28, 2020: contributor

FWIW from the perspective of Bitcoin Wallet, currently we’d need the following combinations:

P2PK, P2PKH & P2WPKH for really old wallets (we used P2PK for the 2013 key rotations unfortunately)
P2PKH & P2WPKH for almost all current wallets
P2WPKH for a future wallet type (but when this can be started is unknown, given the snail-speed SegWit adoption)

And then, of course, a new script type (SegWit v1, Schnorr, whatever) would add one more item to all of the above lists.

Actually we planned to cover everything with the generic “basic” filter that matches on pubKeys being spent.

dangershony commented at 12:21 pm on February 29, 2020: contributor

@schildbach the combinations you mention are already available with the basic filter type. New wallets that only deal with segwit have no reason to fetch all the legacy data in a filter.

For comparison the size of basic filters for block 619357 is 24.1kb while p2wpkh is 3.9kb (I will later provide the size of the combination for witness scripts)

nopara73 commented at 1:41 pm on March 1, 2020: none

@MarcoFalke Thanks for bringing up the filter from wallet creation idea. We did not consider it this time. Back then this is how I implemented Hidden Wallet (predecessor of Wasabi) but in Wasabi I dropped the concept because others were bringing up privacy concerns. I personally don’t think there are any leaks that wouldn’t be just nitpicking, yet I dropped the concept in Wasabi with the thinking P2PWPKH filters from SegWit activation will just be fine to create a light wallet feeling and hopefully the filters’ size will grow slower than the tech improves and if not then we implement it.
At this point I don’t feel like implementing it again as long as there is a chance for having a new filter type, but if it is rejected, that could be a way forward. @Sjors to answer for your very first question. The P2WPKH vs P2WPKH+P2WSH filter sizes are 249MB vs 251MB. @schildbach that’d be awesome if there’d be a mobile wallet that uses client side filters. Finally a mobile wallet I could recommend for privacy! @dangershony It seems to be there’s a consensus here that P2WPKH+P2WSH filters is preferable over P2WPKH filters, would you mind updating your PR?

Sjors commented at 5:12 pm on March 1, 2020: member

Given that the basic type has a BIP (158) this one should as well. I suggest proposing a service bit for it, but see how that discussion goes. Perhaps it’s overkill to have a service bit for a filter that, with increasing SegWit adoption over time, only offers a short term benefit. But if the filter is only used locally by a wallet, then it’s not worth a BIP. Maybe the service bit can have a sunset block.

You should post the v0 filter BIP on the list as a separate thread from the filter combinations discussion, even though of course they are interlinked because of the service bit.

In your PR you use the same M = 784931, P = 19 params as the basic filter. BIP158 says:

Empirical analysis also shows that was chosen as these parameters minimize the bandwidth utilized, considering both the expected number of blocks downloaded due to false positives and the size of the filters themselves

But it sadly doesn’t link to said analysis. That’s food some some mailinglist discussion. The right numbers might depend on SegWit adoption, but we can’t predict that, so may it’s not a factor. But SegWit v0 transactions also have different typical input and output sizes; I don’t know if that matters.

schildbach commented at 8:09 pm on March 1, 2020: contributor

@schildbach the combinations you mention are already available with the basic filter type. New wallets that only deal with segwit have no reason to fetch all the legacy data in a filter.

For comparison the size of basic filters for block 619357 is 24.1kb while p2wpkh is 3.9kb (I will later provide the size of the combination for witness scripts)

I understand what you’re saying, but won’t this difference disappear with SegWit adoption? If e.g. 90% of txns were native SegWit, it should not matter much wether you’re using the basic filter (which covers everything) or “just” the P2WPKH filter (which I assume would then only be ~10% smaller).

If I’m right with the above, I would suggest to consider specialized filters only as temporary and not allocate any scarce resources (technical debt?) to it.

gmaxwell commented at 9:37 pm on March 1, 2020: contributor

The basic filter type covers this application. The small bandwidth savings comes exclusively from the diminished usage of the P2W* outputs and will vanish as adoption patterns change, it comes at an expenses of an (eventual) doubling of disk space usage for filters, and (if used over the network) a reduction in the querying user’s anonymity set.

Additional filters also reduce the prospect of ever having the filters in committed form that could be securely queried by clients that do not trust the server, as the additional cost (delay for computation) for computing multiple filters weighs against a nearly constant benefit that is achieved by the basic filter type.

If the current size of the filters is a killer for an application, the application will end up getting screwed in a couple years regardless– because the shrunk filters will very likely exceed the current size by a healthy margin by then (due to changing usage patterns as well as simple cumulative growth).

Proliferation of filter types was one of the arguments being made against support for these filters in the first place, and I’m saddened to see those concerns proving prescient.

schildbach commented at 9:51 pm on March 1, 2020: contributor

Additional filters also reduce the prospect of ever having the filters in committed form that could be securely queried by clients that do not trust the server

That’s what I’m also very concerned about.

MarcoFalke removed the label Feature on Mar 1, 2020

MarcoFalke added the label Brainstorming on Mar 1, 2020

nopara73 commented at 11:13 pm on March 1, 2020: none

To sum up this conversation: P2P cannot/should not handle additional filter types if it’s about scripts, maybe additional filter types are better reserved for completely different filter types, like txindex filters.

However no issues had been raised with RPC only filters so far. In fact #18221 blockfilterindex Extensibility Proposal could be completely implemented with PRC only mode in mind?

dangershony commented at 11:28 pm on March 1, 2020: contributor

Additional filters also reduce the prospect of ever having the filters in committed form that could be securely queried by clients that do not trust the server, as the additional cost (delay for computation) for computing multiple filters weighs against a nearly constant benefit that is achieved by the basic filter type.

Perhaps this can be solved by making some filter types RPC only?

In this specific case wassabi codebase is building the segwit filters, but that comes at a technical cost of maintaining a full segwit utxo set (plus rewind data) which is what bitcoin core is doing very well already.

I understand what you’re saying, but won’t this difference disappear with SegWit adoption?

The basic filter type covers this application. The small bandwidth savings comes exclusively from the diminished usage of the P2W* outputs and will vanish as adoption patterns change

I do agree on this observation, I would expect though that the same requirements will be put forward on future v1,v2… etc program releases, making bitcoin core more flexible to interact with is something to consider.

adamjonas commented at 4:35 pm on August 3, 2022: member

Closing this given this hasn’t moved since 2020 and related #18221/#18223/https://github.com/bitcoin/bips/pull/896 were closed due to lack of support for additional filters. Can re-open if there is future interest.

adamjonas closed this on Aug 3, 2022

bitcoin locked this on Aug 3, 2023

Add blockfilterindex = p2wpkh #18222

The Goal

Alternative Solutions

Benefits to the End User

What’s Next?

Add `blockfilterindex = p2wpkh` #18222