kanzure commented at 2:14 pm on November 9, 2024: contributor

This pull request updates lists.linuxfoundation.org urls to gnusha.org/url.

Background

Unfortunately, lists.linuxfoundation.org is no longer hosting the bitcoin-dev static email archives, nor any others for that matter. This has caused many links around the web to become broken.

Redirect web service

gnusha.org/url is a web service that redirects old style archive urls to a new location where the content can be viewed.

For example, a bitcoin-dev mailing list archive url like:

https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

using https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-June/014472.html

is redirected to:

https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/

The redirect mappings were generated from my archives and you can review the scripts I made to do that here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b

Alternative

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

Those links would look like:

0https://gnusha.org/pi/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/
1https://mailing-list.bitcoindevs.xyz/bitcoindev/CABaSBaxDjj6ySBx4v+rmpfrw4pE9b=JZJPzPQj_ZUiBg1HGFyA@mail.gmail.com/

If this is the preferred solution for the BIPs, then let me know how I can help. Take a look here.

Finally, one other alternative is to wait and do nothing. Maybe someone can convince Linux Foundation to put up the static HTML archives for posterity?

More background

For more information see:

update lists.linuxfoundation.org urls to gnusha.org/url

Unfortunately, lists.linuxfoundation.org is no longer hosting the
bitcoin-dev mailing list static archives nor any other mailing list
static archives.

gnusha.org/url is a web service that redirects old style archive urls to
a new location where the content can be viewed.

For more information see:
https://gnusha.org/url/
https://x.com/kanzure/status/1853779672514826334
https://github.com/bitcoin/bitcoin/pull/29782#issuecomment-2460974096
https://github.com/bitcoin/bitcoin/pull/31240
https://github.com/bitcoinops/bitcoinops.github.io/pull/2018

55a92a59c1

jlopp commented at 7:59 am on November 10, 2024: contributor

tACK

jonatack commented at 2:52 am on November 11, 2024: member

Nice, concept ack.

Looks like the http urls need updating.

0bip-0099.mediawiki:11:  Post-History: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/008936.html
1bip-0101.mediawiki:90:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-June/008820.html Simulations show] that with the current peer-to-peer protocol, miners behind high-latency or low-bandwidth links are at a disadvantage compared to miners connected to a majority of hashpower via low-latency, high-bandwidth links. Larger blocks increase the advantage of miners with high-bandwidth connections, although that advantage can be minimized with changes to the way new blocks are announced (e.g. http://bitcoinrelaynetwork.org/ ).
2bip-0106.mediawiki:55:These two proposals have been derived after discussion on [https://bitcointalk.org/index.php?topic=1154536.0 BitcoinTalk] and [http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010285.html bitcoin-dev mailing list]. The original idea and its evolution in the light of various arguments can be found [http://upalc.com/maxblocksize.php here].
3bip-0111.mediawiki:30:to no privacy<ref>http://eprint.iacr.org/2014/763</ref>, as well as being a large DoS risk on some nodes<ref>[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2013-July/003044.html] is one example where the issues were found, though others independently discovered issues as well. Sample DoS exploit code available at https://github.com/petertodd/bloom-io-attack.</ref>.
4bip-0112.mediawiki:381:[http://lists.linuxfoundation.org/pipermail/lightning-dev/2015-July/000021.html HTLCs using OP_CHECKSEQUENCEVERIFY/OP_LOCKTIMEVERIFY and revocation hashes]
5bip-0112.mediawiki:389:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010396.html Softfork deployment considerations]
6bip-0113.mediawiki:119:[http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-August/010396.html Softfork deployment considerations]
7bip-0124.mediawiki:13:  Post-History: http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-November/011795.html

jonatack added the label Bug fix on Nov 11, 2024

also update http:// linuxfoundation urls.

See 55a92a59c1e5ddb7c165327d5af4fd0c93d2a2a9.

9834559b3c

kanzure commented at 12:36 pm on November 11, 2024: contributor

@jonatack, fixed that. thanks.

I am seeking feedback on gnusha.org/url/ vs resolving everything to urls like https://gnusha.org/pi/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/

With the resolved urls, the IDs are more verbose– instead of an integer ID for the message, it’s the Message-ID from the actual email. Then in the future someone can use that Message-ID to find the email on a mirror, like https://mailing-list.bitcoindevs.xyz/bitcoindev/ogoa6o$o4s$1@blaine.gmane.org/.

This seems to be the likely direction that bitcoin.git is going in. They only have four urls in the codebase.

kanzure commented at 12:42 pm on November 11, 2024: contributor

I’ll elaborate on gnusha.org/url vs gnusha.org/pi/bitcoindev style urls.

It is nice and simple to use a find-and-replace with gnusha.org/url/ because anyone can easily repeat that exercise and verify the git diff before merging the pull request. However, the actual routing is not something that is verified by just checking that the git diff looks sane.

If I was to provide my mapping (I haven’t uploaded this yet, but I did publish the scripts I used to make it), then someone could write a script and/or cross-verify that all of the replacements are valid according to the mapping, if we were to use the destination urls instead of the redirect service for this pull request. Then someone would be able to verify that kind of pull request.

There are other further issues though, like content malleability, wrong redirect link injection, service provider compromise… instead of referencing by Message-ID maybe in the future we can reference by email hash…. Just dreaming here.

kanzure commented at 7:49 pm on November 11, 2024: contributor

Instead of using gnusha.org/url, another option would be to resolve all of the archive urls to their email Message-ID redirects for the bitcoin-dev mailing list and use those urls instead. In fact, either gnusha.org/pi/bitcoindev or https://mailing-list.bitcoindevs.xyz/bitcoindev/ could be used for this purpose.

this is implemented here: https://gist.github.com/kanzure/4e7bcc58344ceaa1a668e65a434adb2b#file-resolver-py

LMK if that’s the preference instead of the current version of this pull request (gnusha.org/url inserts).

jonatack commented at 9:45 pm on November 13, 2024: member

Thanks @kanzure. No strong opinion, but I think I mildly prefer the destination urls over the redirects.

murchandamus commented at 4:34 pm on November 14, 2024: contributor

LGTM.

I also don’t have a strong opinion regarding the two approaches. I’d prefer a quicker reestablishing functionality over a delayed perfect solution.

kanzure commented at 5:15 pm on November 14, 2024: contributor

There would be no delay. Either will work now. See the gist.

ajtowns commented at 6:49 am on November 19, 2024: contributor

The lists.linuxfoundation.org site seems to redirect automatically now:

0$ curl -I https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html
1HTTP/1.1 301 Moved Permanently
2Content-length: 0
3Location: https://gnusha.org/url/https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2012-June/001511.html

(also applies to lightning-dev mails)

Is there still a good reason to churn a bunch of bip texts, rather than just restrict this to updating the mailing list location in bip 2?

kanzure commented at 6:13 pm on November 19, 2024: contributor

The lists.linuxfoundation.org site seems to redirect automatically now:

I haven’t announced this yet. It’s a strange situation because LF has not agreed to keep these redirects (perhaps they don’t even know at the moment about these redirects), and LF is not running that server. Given the strangeness I haven’t been sure what to say or announce precisely.

Fix broken lists.linuxfoundation.org URLs #1698

Background

Redirect web service

Alternative

More background