ci: Pass down $MAKEJOBS to test_runner.py, other improvements #16739

pull MarcoFalke wants to merge 2 commits into bitcoin:master from MarcoFalke:1908-ciScratch changing 4 files +8 −4

MarcoFalke commented at 8:03 PM on August 27, 2019: member
Some changes to the ci system:
- Pass down MAKEJOBS to the test_runner, instead of falling back to the default of 4. Passing it down avoids OOM on weak machines and allows better use of resources on beefy machines.
- Move CCACHE_DIR to ./ci/scratch/ subfolder: ccache is executed with root permissions inside the docker, so the cache files are created with root as owner. So it might be wise to not put them in the $HOME of the host
- Use the scratch dir as prefix for the test runner, as opposed to /tmp/, which is often a ramdisk and thus leads to OOM on the host when either a lot of tests are run in parallel or when a lot of tests fail and the datadirs are not cleaned.
ci: Pass down $MAKEJOBS to test_runner.py fa60583d23
ci: Move CCACHE_DIR and test_runner tmp dir into ./ci/scratch/ fa27372e65
MarcoFalke added the label Tests on Aug 27, 2019
DrahtBot commented at 10:32 PM on August 27, 2019: member


The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.



Conflicts

Reviewers, this pull request conflicts with the following ones:
- #16367 (Multiprocess build support by ryanofsky)
- #12134 (Build previous releases and run functional tests by Sjors)
- #10102 ([experimental] Multiprocess bitcoin by ryanofsky)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
in ci/test/04_install.sh:9 in fa27372e65
```
   5 | @@ -6,6 +6,7 @@
   6 |  
   7 |  export LC_ALL=C.UTF-8
   8 |  
   9 | +mkdir -p "${BASE_SCRATCH_DIR}"
```
promag commented at 11:05 PM on August 27, 2019:

Is this really necessary? On my system:

$ ls /tmp/this ls: /tmp/this: No such file or directory $ CCACHE_DIR=/tmp/this/doesnt/exits ccache find /tmp/this /tmp/this /tmp/this/doesnt /tmp/this/doesnt/exits /tmp/this/doesnt/exits/9 /tmp/this/doesnt/exits/9/stats /tmp/this/doesnt/exits/ccache.conf
MarcoFalke commented at 11:11 PM on August 27, 2019:

Yes, this is needed when the ci runner sets the ccache dir to not reside in the scratch dir. E.g. our cirrus ci: https://github.com/bitcoin/bitcoin/blob/a7be1cc92be4946c4f042bccd3a1b007657f3241/.cirrus.yml#L14

promag commented at 11:14 PM on August 27, 2019:

Oh right, when CCACHE_DIR is unrelated to BASE_SCRATCH_DIR, thanks.
fanquake approved
fanquake commented at 5:19 AM on August 29, 2019: member

ACK fa27372e65359f12bea9ac0680d6bcb96680dda8
laanwj approved
laanwj commented at 10:52 AM on August 29, 2019: member

ACK fa27372e65359f12bea9ac0680d6bcb96680dda8
laanwj referenced this in commit 1f77c5c3e7 on Aug 29, 2019
laanwj merged this on Aug 29, 2019
laanwj closed this on Aug 29, 2019
MarcoFalke deleted the branch on Aug 29, 2019
kittywhiskers referenced this in commit 7811034811 on Dec 13, 2021
kittywhiskers referenced this in commit fc93cdc1ee on Dec 13, 2021
DrahtBot locked this on Dec 16, 2021

Contributors

Labels

ci: Pass down $MAKEJOBS to test_runner.py, other improvements #16739

Conflicts