contrib: skip missing binaries in gen-manpages

andremralves commented at 9:37 pm on September 26, 2024: contributor

Instead of stopping the execution of gen-manpages.py when a binary is not found, continue generating manpages for the available binaries and skip the missing ones.

A new argument, --skip-missing-binaries, has been added to enable this behavior.

0➜  bitcoin git:(fix-gen-manpages) ✗ ./contrib/devtools/gen-manpages.py --help
1usage: gen-manpages.py [-h] [-s]
2
3options:
4  -h, --help            show this help message and exit
5  -s, --skip-missing-binaries
6                        skip generation for binaries that are not found

closes #30985

This PR also includes an error prompt if no binaries are found in the build path.

DrahtBot commented at 9:38 pm on September 26, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/30986.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	laanwj, achow101
Concept ACK	BrandonOdiwuor
Stale ACK	rkrux, willcl-ark

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#29457 (doc: Fix gen-manpages to check build options by BrandonOdiwuor)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Scripts and tools on Sep 26, 2024

DrahtBot added the label CI failed on Sep 29, 2024

DrahtBot removed the label CI failed on Sep 29, 2024

willcl-ark commented at 8:47 am on October 1, 2024: member

Concept ACK.

I wonder if it’s not easier to minimize what is being caught in the IOError try block and just remove missing bins from the BINARIES list? This list is used later on in the script too so in my opinion that is probably the better option. Otherwise we will end up referencing non-existing binaries/manpages in the existing manpages. Something like:

 0diff --git a/contrib/devtools/gen-manpages.py b/contrib/devtools/gen-manpages.py
 1index 92acd9a4037..0e0d31807bd 100755
 2--- a/contrib/devtools/gen-manpages.py
 3+++ b/contrib/devtools/gen-manpages.py
 4@@ -39,7 +39,8 @@ for relpath in BINARIES:
 5         r = subprocess.run([abspath, "--version"], stdout=subprocess.PIPE, check=True, text=True)
 6     except IOError:
 7         print(f'{abspath} not found or not an executable', file=sys.stderr)
 8-        sys.exit(1)
 9+        BINARIES.remove(relpath)
10+        continue
11     # take first line (which must contain version)
12     verstr = r.stdout.splitlines()[0]
13     # last word of line is the actual version e.g. v22.99.0-5c6b3d5b3508

I also notice running this post-cmake without specifying BUILDDIR results in a slightly incorrect error due to the new binary locations:

 0₿ ./contrib/devtools/gen-manpages.py
 1/home/will/src/bitcoin/src/bitcoind not found or not an executable. Skipping...
 2/home/will/src/bitcoin/src/bitcoin-cli not found or not an executable. Skipping...
 3/home/will/src/bitcoin/src/bitcoin-tx not found or not an executable. Skipping...
 4/home/will/src/bitcoin/src/bitcoin-wallet not found or not an executable. Skipping...
 5/home/will/src/bitcoin/src/bitcoin-util not found or not an executable. Skipping...
 6/home/will/src/bitcoin/src/qt/bitcoin-qt not found or not an executable. Skipping...
 7Traceback (most recent call last):
 8  File "/home/will/src/bitcoin/./contrib/devtools/gen-manpages.py", line 63, in <module>
 9    footer.write('\n'.join(versions[0][2]).strip())
10                           ~~~~~~~~^^^
11IndexError: list index out of range

Whilst you are in here, perhaps we could return early if versions is empty, optionally prompting folks to set BUILDDIR?

We could change the default paths, but as people may use different build dirs I’m not sure if this is worth it.

andremralves force-pushed on Oct 2, 2024

andremralves commented at 2:21 pm on October 2, 2024: contributor

Thanks for the feedback!

I wonder if it’s not easier to minimize what is being caught in the IOError try block and just remove missing bins from the BINARIES list? This list is used later on in the script too so in my opinion that is probably the better option. Otherwise we will end up referencing non-existing binaries/manpages in the existing manpages.

I agree and have updated the commit to reflect your suggestions. I just needed to make a copy of BINARIES so that it’s possible to remove items from it while iterating over the copied list. Thanks.

Whilst you are in here, perhaps we could return early if versions is empty, optionally prompting folks to set BUILDDIR?

I also think it’s a good idea.

We could change the default paths, but as people may use different build dirs I’m not sure if this is worth it.

I think it’s worth it since other scripts/docs have been updated to use /build as the default path, so we can do the same here.

I’ve added a few more commits addressing the latter two topics. Could you review them as well? @willcl-ark

laanwj commented at 12:01 pm on October 24, 2024: member

i’m divided on this. There’s a use-case, for sure, but this script is mainly used to generate the manpages for a release. And we’ve had exactly the opposite issue where conditional behavior resulted in incomplete manual pages. (Prompting me to open #17506) So not sure it should be the default. As an option, sure.

andremralves commented at 1:19 pm on October 24, 2024: contributor

The main issue this PR addresses is that currently, no man pages are generated if a single binary is missing. The goal is to skip the missing binary and generate man pages for the others, as was done in the previous shell script.

If I understand correctly, you want to ensure no information is lost in the man pages, right? In that case, I can simply remove the following line:

042          BINARIES.remove(relpath)

This way, the existing binaries will have their man pages generated, and all the other man pages will still be referenced at the bottom, even if the binaries are missing.

andremralves force-pushed on Oct 24, 2024

laanwj commented at 1:31 pm on October 24, 2024: member

Kind of. The problem is that the command will still finish successfully in case of an incomplete build (or build configuration), while the before a release it’s critical to update all the pages.

This is why i suggested to add this as an option eg --skip-missing-binaries.

This way, the existing binaries will have their man pages generated, and all the other man pages will still be referenced at the bottom, even if the binaries are missing.

That is an improvement, but doesn’t entirely address the concern.

andremralves force-pushed on Oct 27, 2024

andremralves commented at 4:27 am on October 28, 2024: contributor

I probably don’t have enough understanding of the release process to know exactly how this change could affect it, but I assume that having gen-manpages.py exit with an error is important somewhere during the process.

I implemented your suggestion by adding the –skip-missing-binaries option to the command.

Thanks for the feedback! What do you think of it now?

fanquake commented at 12:07 pm on October 29, 2024: member

I wonder if a better approach here would be to further integrate the manpage generation into the build system; given that the two issues of “which binaries exist” and “where do they exist” are already solved in that case. This should also remove the dep on git.

laanwj commented at 1:51 pm on October 29, 2024: member

I implemented your suggestion by adding the –skip-missing-binaries option to the command.

Yes, thanks!

I wonder if a better approach here would be to further integrate the manpage generation into the build system; given that the two issues of “which binaries exist” and “where do they exist” are already solved in that case. This should also remove the dep on git.

That would be better. The thing is, the build system isn’t allowed to execute the binaries it generated (and even if so this wouldn’t support cross compiling and be extremely ugly). So this would need an alternative way of extracting the arguments for the tools than running them with --help. Parsing C++ code is kind of nasty, so i’m not sure how.

BrandonOdiwuor commented at 2:32 pm on November 6, 2024: contributor

Concept ACK Adding --skip-missing-binaries as an optional flag seems reasonable.

in contrib/devtools/gen-manpages.py:28 in 7060d64bf7 outdated

23+parser.add_argument(
24+    "-s",
25+    "--skip-missing-binaries",
26+    action="store_true",
27+    default=False,
28+    help="skip generation for binaries that are not found",

rkrux commented at 11:55 am on November 8, 2024:

nit: for verbosity

0help="skip generation for binaries that are not found in the build path",

andremralves commented at 4:11 am on November 14, 2024:

Good suggestion. Thanks!

in contrib/devtools/gen-manpages.py:72 in f0806188dc outdated

67@@ -68,6 +68,10 @@
68 
69     versions.append((abspath, verstr, copyright))
70 
71+if not versions:
72+    print(f'No binaries found in {builddir}. Set BUILDDIR to specify the correct build path.')

rkrux commented at 12:08 pm on November 8, 2024:

0Set BUILDDIR to specify the correct build path.

Is the intent to make the user explicitly set BUILDDIR? Not aware of the usual use cases but as a user, my preference is to just run the script to generate the man pages w/o needing to set an env var. I’d be ok with this error as well.

0print(f'No binaries found in {builddir}. Please specify the correct build path.')

andremralves commented at 5:11 am on November 14, 2024:

The most common use case is running the script without setting BUILDDIR, in which case the default builddir, /build, is used. However, if no binaries are found in /build we remind the user to set BUILDDIR to specify the correct path.

rkrux commented at 8:58 am on November 14, 2024:

if no binaries are found in /build

This could also be because the user might not have built the binaries yet even in the case of default build path.

A more verbose message could be:

0'No binaries found in {builddir}. Please ensure the binaries are present in {builddir}, or set another build path using the BUILDDIR env variable.'

andremralves commented at 9:46 pm on November 21, 2024:

This could also be because the user might not have built the binaries yet even in the case of default build path.

Yes, I haven’t added it to the message because I though it would be too verbose, but I can reconsider it.

andremralves commented at 9:51 pm on November 21, 2024:

I’ve updated it, thanks for the suggestion.

rkrux approved

rkrux commented at 12:13 pm on November 8, 2024: none

Coming here from #31255, putting my two cents in.

I had to generate man pages while reviewing a PR and found that the script threw error even in the default case due to the new build process that moved the binaries in /build, so a strong ACK for 0194a638d31fef7e8193c50843c11d2c2ee212a3 because IMO the script should work in the default case w/o needing to set the BUILDDDIR.

As I read from @laanwj’s comment that the main intent of this script is to generate man pages for a release, I’d assume all the binaries would be found because they are part of the release? Having said this, the new --skip-missing-binaries arg could be useful in the cases when not everyone will have all the binaries. Eg: I didn’t use to have the GUI earlier.

All in all ACK for f0806188dcde531a401bdfeaaed8c04801f4b5f0

gen-manpages: implement --skip-missing-binaries

With --skip-missing-binaries, instead of stopping the execution of
gen-manpages.py when a binary is not found, continue generating
manpages for the available binaries and skip the missing ones.

299e2220e9

andremralves force-pushed on Nov 14, 2024

andremralves force-pushed on Nov 21, 2024

laanwj approved

laanwj commented at 11:43 am on November 22, 2024: member

Code review ACK 4bbd28baf33382231f4f1dab20681c05f9915af2

DrahtBot requested review from rkrux on Nov 22, 2024

DrahtBot requested review from willcl-ark on Nov 22, 2024

DrahtBot requested review from BrandonOdiwuor on Nov 22, 2024

rkrux approved

rkrux commented at 12:15 pm on November 22, 2024: none

re-ACK 4bbd28baf33382231f4f1dab20681c05f9915af2

Minimal range diff.

 01:  7060d64bf7 ! 1:  299e2220e9 gen-manpages: implement --skip-missing-binaries
 1    @@ contrib/devtools/gen-manpages.py: BINARIES = [
 2     +    "--skip-missing-binaries",
 3     +    action="store_true",
 4     +    default=False,
 5    -+    help="skip generation for binaries that are not found",
 6    ++    help="skip generation for binaries that are not found in the build path",
 7     +)
 8     +args = parser.parse_args()
 9     +
102:  0194a638d3 = 2:  c32104f544 gen-manpages: Update default build path
113:  f0806188dc ! 3:  4bbd28baf3 gen-manpages: Prompt error if no binaries are found
12    @@ contrib/devtools/gen-manpages.py: for relpath in BINARIES:
13          versions.append((abspath, verstr, copyright))
14
15     +if not versions:
16    -+    print(f'No binaries found in {builddir}. Set BUILDDIR to specify the correct build path.')
17    ++    print(f'No binaries found in {builddir}. Please ensure the binaries are present in {builddir}, or set another build path using the BUILDDIR env variable.')
18     +    sys.exit(1)
19     +
20      if any(verstr.endswith('-dirty') for (_, verstr, _) in versions):

willcl-ark approved

willcl-ark commented at 1:54 pm on November 25, 2024: member

ACK 4bbd28baf33382231f4f1dab20681c05f9915af2

Much better to add this behind a default-false flag

gen-manpages: Prompt error if no binaries are found ee6185372f

in contrib/devtools/gen-manpages.py:43 in c32104f544 outdated

39@@ -40,7 +40,7 @@
40     topdir = r.stdout.rstrip()
41 
42 # Get input and output directories.
43-builddir = os.getenv('BUILDDIR', topdir)
44+builddir = os.getenv('BUILDDIR', os.path.join(topdir, 'build'))

achow101 commented at 6:39 pm on November 26, 2024:

In c32104f5449bbea89a3fde5b99c97f7de6914829 “gen-manpages: Update default build path”

The documentation states that this script can work from any directory for in-tree builds, but this change would break that (although we do not currently do in-tree builds). It further states that BUILDDIR should be set for out-of-tree builds. Either the documentation should be updated, or this default should not be changed. I prefer the latter.

andremralves commented at 0:41 am on November 27, 2024:

Hmm, okay. I think it’s better to remove it from this PR and maybe address the subject in another issue so it doesn’t interfere with the other commits.

So, the PR will include only:

The –skip-missing-binaries option
The error prompt to inform the user if no binaries are found in the build directory path.

laanwj commented at 8:24 am on November 27, 2024:

Yes, it’s unrelated to the scope of the PR anyway.

andremralves force-pushed on Nov 27, 2024

laanwj approved

laanwj commented at 8:26 am on November 27, 2024: member

re-ACK ee6185372fc317d3948690997117e42f6b79a5ff

DrahtBot requested review from rkrux on Nov 27, 2024

DrahtBot requested review from willcl-ark on Nov 27, 2024

achow101 commented at 5:33 pm on November 27, 2024: member

ACK ee6185372fc317d3948690997117e42f6b79a5ff

achow101 merged this on Nov 27, 2024

achow101 closed this on Nov 27, 2024

contrib: skip missing binaries in gen-manpages #30986

Code Coverage & Benchmarks

Reviews

Conflicts