doc: Outdated block size in rest.md

DesWurstes commented at 11:58 am on April 19, 2020: contributor

The HTTP request and response are both handled entirely in-memory, thus making maximum memory usage at least 2.66MB (1 MB max block, plus hex encoding) per request.

https://github.com/bitcoin/bitcoin/blob/b470c758470fd97ccb42d0348a32942afa2a3ec8/doc/REST-interface.md#L33

Now with SegWit up to 4 MB, is it 10.66 MB?

DesWurstes added the label Bug on Apr 19, 2020

fanquake added the label Docs on Apr 19, 2020

yozaam commented at 6:28 pm on April 19, 2020: none

Sorry if it’s a noob question I understand block size is 4MB now but how did you get 10.66MB as size of the total usage? Is it (4*2.66) ?

DesWurstes commented at 6:44 am on April 20, 2020: contributor

Yes I multiplied by 4; might be a bad estimate.

jonasschnelli added the label good first issue on Apr 20, 2020

michaelfolkson commented at 12:46 pm on April 20, 2020: contributor

I haven’t looked into whether the witness data is also handled in-memory (presumably it is). But for anyone who picks this up it is better to refer to “block weight” rather than “block size” in the docs. The “block size” is still 1MB. https://medium.com/@jimmysong/understanding-segwit-block-size-fd901b87c9d4

yozaam commented at 12:52 pm on April 20, 2020: none

https://medium.com/@jimmysong/understanding-segwit-block-size-fd901b87c9d4

Loved your article 50👏 , It was very informative!

BHouwens commented at 2:24 pm on April 21, 2020: none

I’m happy to make this change if it’s still required? Looking to make a first contribution. It may even be a good idea to note the difference between legacy node “weight” expectation vs segwit nodes?

fjahr commented at 0:09 am on April 22, 2020: member

I think there are a few more things to consider here. First of all, I can’t really make sense of the 2.66MB number. From my understanding, it should be 3 MB if we are talking about Hex. 1 MB is the full block that gets loaded into memory before the processing. Then, it might get processed into hex which is the less efficient encoding format where 1 byte binary takes 2 bytes in hex. I think this is right because HexStr is also reserving twice the space here: https://github.com/bitcoin/bitcoin/blob/b6a5dc90bfd4640cf9f914e59bf8e21cd265b51e/src/util/strencodings.h#L128

But then there is also the JSON option and that should add way more overhead than a hex encoding. I just did a quick check if I am missing something with a recent block to make sure I am not missing something and this looks right:

0$ ls -hla | grep 5694ce
1-rw-r--r--@   1 XXX  XXX   1.2M Apr 22 01:57 00000000000000000011fbefe97f501a0160a2c86201ef5ef79e1cd2ce5694ce.bin
2-rw-r--r--@   1 XXX  XXX   2.4M Apr 22 01:56 00000000000000000011fbefe97f501a0160a2c86201ef5ef79e1cd2ce5694ce.hex.txt
3-rw-r--r--@   1 XXX  XXX   8.0M Apr 22 01:53 00000000000000000011fbefe97f501a0160a2c86201ef5ef79e1cd2ce5694ce.json

@michaelfolkson mentioned block weight, but I think this would not be the right term to talk about here because the weight is a theoretical term used to calculate fees and when a block is considered full but it only has an indirect influence on the space used in memory. I would say the term weight was actually introduced to create a disconnect between the space that the block uses in memory and measuring when a block is full. (Edit: I should probably call it a direct influence but the point is you can’t tell the block size in memory from the block weight)

But this disconnect also makes it much harder to give a good estimate of what size to expect. 4 MB is the theoretical max size of a Segwit block, but even with increasing Segwit adoption, I would say a 2 MB block would be huge. We are currently at 1.2 -1.3 MB usually: https://bitcoinvisuals.com/chain-block-size.

So what number to take? If I had to make a guess it would be ~10 MB worst case for the foreseeable future (big Segwit block with JSON). But really I would rather suggest removing that number, I am not sure how much relevance it has. If I am right and the number was off all this time, then I guess nobody noticed it because of that ;) But I am also not a regular rest interface user and looked at that code for the first time today, so don’t take my word for it.

Anyway, if anyone wants to open a PR for this I would suggest looking for more inconsistencies in the docs and the rest code and put them together. Looking at rest.cpp this error string seems to be missing the json option for example: https://github.com/bitcoin/bitcoin/blob/b6a5dc90bfd4640cf9f914e59bf8e21cd265b51e/src/rest.cpp#L209

MarcoFalke commented at 11:21 am on April 22, 2020: member

But really I would rather suggest removing that number, I am not sure how much relevance it has.

Agree with @fjahr that the number should be removed. No documentation is better than wrong documentation.

laanwj referenced this in commit afed2e98b0 on Apr 30, 2020

laanwj commented at 9:46 am on April 30, 2020: member

Closed by #18810

laanwj closed this on Apr 30, 2020

sidhujag referenced this in commit a39824f75d on May 2, 2020

Fabcien referenced this in commit f4ccadb3d3 on Jan 26, 2021

PastaPastaPasta referenced this in commit cec8c02966 on Jun 27, 2021

PastaPastaPasta referenced this in commit 35bef38798 on Jun 28, 2021

PastaPastaPasta referenced this in commit 094c97512e on Jun 29, 2021

PastaPastaPasta referenced this in commit 2f20207a8f on Jul 1, 2021

PastaPastaPasta referenced this in commit 6142956682 on Jul 1, 2021

PastaPastaPasta referenced this in commit f1e0b21c8e on Jul 14, 2021

DrahtBot locked this on Feb 15, 2022

doc: Outdated block size in rest.md #18703