AI for fee estimation and other parts of the system #11064

Just wondering, instead constantly tweaking fee estimation algos which will never be perfect why not to deploy AI to do the work?

AI won't be perfect either. Additionally, it comes with a performance burden and the uncertainty of a black box.

None of this is desirable for fee estimation.

On Aug 16, 2017 11:09 AM, "deremeda" notifications@github.com wrote:

Just wondering, instead constantly tweaking fee estimation algos which will never be perfect why not to deploy AI to do the work?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/11064, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGmvzQpdyedVqdrIZN6jPTH9CTZMw0Uks5sYrG7gaJpZM4O4oGx .

@fanquake why close this quik? I'd like to at least hear others opinions, I'm genuinely interested in what people have say about it before I commit more energy into exploring this subject further. @MarcoFalke Nothing is ever perfect, it would just need to be better than what we have now. Performance is just and engineering problem, distributed AI fee estimation, etc. whereas the black box argument I don't buy. If it's safe to fly a fighter jet or drive a car it's safe to estimate a fee surely. Besides how much random is /dev/random, black box too. I don't mean this to be back and forth discussion. Just wanted to hear peoples gut reaction to the idea.

As long as the problem is possible to solve with "conventional" approaches, I don't see a reason to throw AI at it.

Of course you are still more than welcome to do research in that direction.

Btw, are you aware that fee estimation was reworked for the upcoming 0.15.0?

On Aug 16, 2017 12:38 PM, "deremeda" notifications@github.com wrote:

@fanquake https://github.com/fanquake why close this quik? I'd like to at least hear others opinions, I'm genuinely interested in what people have say about it before I commit more energy into exploring this subject further.

@MarcoFalke https://github.com/marcofalke Nothing is ever perfect, it would just need to be better than what we have now. Performance is just and engineering problem, distributed AI fee estimation, etc. whereas the black box argument I don't buy. If it's safe to fly a fighter jet or drive a car it's safe to estimate a fee surely. Besides how much random is /dev/random, black box too. I don't mean this to be back and forth discussion. Just wanted to hear peoples gut reaction to the idea.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bitcoin/bitcoin/issues/11064#issuecomment-322731695, or mute the thread https://github.com/notifications/unsubscribe-auth/AGGmvxf9a-fBX-1meqH8kV8JymVppsA_ks5sYsaOgaJpZM4O4oGx .

What kind of AI? There are many kinds of machine learning. Which one would be suited? What would be the inputs and outputs? What dataset would you train it on? How would you test it? How would it cope with changing circumstances on the network? As it is, this issue is akin to saying "why not use computation to do the work".

Of course you are still more than welcome to do research in that direction.

Exactly. This is research-paper level material, creating a github issue achieves nothing.

@deremeda there's a new estimaterawfee RPC in v0.15. That allows you to extract the raw data that bitcoind is using in its fee estimation algorithm. You can then feed that data into whatever ML you want for your AI fee estimation. See #10543 for details of the API.

Definitely interested to see what you come up with. Fee estimation is a difficult problem so the more approaches the better, but as @MarcoFalke and @laanwj have pointed out, this function should live outside bitcoind.

That allows you to extract the raw data that bitcoind is using in its fee estimation algorithm.

If you want more raw data then estimaterawfee provides, you can run bitcoin with #10443 which logs information about individual transactions. I can also share data from a bitcoin node I've been running with #10443 (a 4.6G log file with transactions going back to Jun 25).

Keep in mind the fee estimate needs to not only be accurate it must be resistant to manipulation (e.g. there shouldn't be a way to make it go up except by transacting more and paying more fees)

@deremeda there's a new estimaterawfee RPC in v0.15. That allows you to extract the raw data that bitcoind is using in its fee estimation algorithm. You can then feed that data into whatever ML you want for your AI fee estimation. See #10543 for details of the API.

Definitely interested to see what you come up with. Fee estimation is a difficult problem so the more approaches the better, but as @MarcoFalke and @laanwj have pointed out, this function should live outside bitcoind.

I'm curious if anyone know's of anyone attempting to do this. I also feel there is an opportunity to improve fee estimation using ML techniques. All the data is is there, it just needs to be crunched appropriately...

Currently I'm using Blockstreams API and also I really like the Mempool.space fee estimations: https://mempool.space/api/v1/fees/recommended

But a well trained narrow AI could likely do it even better.