This avoids the possibility of flooding /tmp during long fuzzing campaigns by allowing an environment variable to be set that causes the test_common_Bitcoin Core directory to be cleared after each fuzz run. Because libFuzzer campaigns seem to produce less data in /tmp, I believe the possibility may not have been noticed before. A 4 hour Eclipser run produced nearly 350GB in test_common_Bitcoin Core. Tested locally.
fuzz: Add environment option to keep /tmp/ clean #22472
pull agroce wants to merge 11 commits into bitcoin:master from agroce:master changing 1 files +6 −0-
agroce commented at 4:30 PM on July 16, 2021: contributor
-
c07998c0d1
Add environment option to keep /tmp/ clean
This avoids the possibility of flooding /tmp during long fuzzing campaigns
- DrahtBot added the label Tests on Jul 16, 2021
-
agroce commented at 6:03 PM on July 16, 2021: contributor
Looks as if I need to add
-lstdc++fssomewhere, perhaps, to build in all envs. Anyone know where? -
Use boost to avoid adding new link 8e03c48a5b
-
Type for 2nd arg to boost::filesystem::remove_all d13ce4e81d
-
Remove extra semicolon 35323b08d3
-
Include boost error codes 5c31c13708
-
Just call, don't add dependency d0beba5ae2
-
maflcko commented at 8:34 AM on July 17, 2021: member
Is this only an issue with Eclipser?
If not, can you please provide a list of the largest files/folders in that directory?
Locally, I am seeing that the size stays constant, as it should be:
$ FUZZ=process_message_filteradd ./src/test/fuzz/fuzz$ du -s /tmp/test_common_Bitcoin\ Core/6d8ae75a417046414e8a64bb8398d1382b6801a363c1cd6c08512de1f63fdc38/ 17412 /tmp/test_common_Bitcoin Core/6d8ae75a417046414e8a64bb8398d1382b6801a363c1cd6c08512de1f63fdc38/ -
maflcko commented at 8:35 AM on July 17, 2021: member
OSS-Fuzz seems to be running into the same issue: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=35027
-
maflcko commented at 11:46 AM on July 17, 2021: member
Also, if you want you can add Eclipser instructions to https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md so that it will be easier to get started with it in the future.
-
agroce commented at 1:53 PM on July 17, 2021: contributor
I haven't tried with AFL. It may be only under rare circumstances? But not in libFuzzer (or an AFL persist loop?)? I'll show you a bad run later. You end up with many (very many) entries with different hashes in the '/tmp/test_common_Bitcoin Core/` dir. Are you saying multiple fuzz processes should use the same one? Maybe if one is launched while another is still somehow live?
Yes, I plan to add an Eclipser guide once I've hashed out some issues.
-
agroce commented at 2:11 PM on July 17, 2021: contributor
It looks like:
~/bitcoin# du -h /tmp 18M /tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest 18M /tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc 18M /tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest/wallets 18M /tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest 18M /tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d 18M /tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest/wallets 18M /tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest 18M /tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f 18M /tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest/wallets 18M /tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest 18M /tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07 18M /tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest 18M /tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5 18M /tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest/wallets 18M /tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest 18M /tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30 18M /tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest/wallets 18M /tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest 18M /tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9 18M /tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest/wallets 18M /tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest 18M /tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901 18M /tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest/wallets 18M /tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest 18M /tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f 18M /tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest/wallets 18M /tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest 18M /tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18 18M /tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest/wallets 18M /tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest 18M /tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f 18M /tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest/wallets 18M /tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest 18M /tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81 18M /tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest/wallets 18M /tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest 18M /tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5 18M /tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest/wallets 18M /tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest 18M /tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0 18M /tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest 18M /tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b 18M /tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest/wallets 18M /tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest 18M /tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347 18M /tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest/wallets 18M /tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest 18M /tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583 18M /tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest 18M /tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3 18M /tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest/wallets 18M /tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest 18M /tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e 18M /tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest/wallets 18M /tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest 18M /tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d 18M /tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest 18M /tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617 18M /tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest 18M /tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e 18M /tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest 18M /tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf 18M /tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest 18M /tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba 18M /tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest/wallets 18M /tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest 18M /tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797 18M /tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest 18M /tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08 18M /tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest/wallets 18M /tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest 18M /tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f 18M /tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest 18M /tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91 18M /tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest 18M /tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d 18M /tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest/wallets 18M /tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest 18M /tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787 4.0K /tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest/wallets 12K /tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest 16K /tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0 18M /tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest/wallets 18M /tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest 18M /tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30 18M /tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest/wallets 18M /tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest 18M /tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8 18M /tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest/wallets 18M /tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest 18M /tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57 18M /tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest/wallets 18M /tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest 18M /tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574 18M /tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest/wallets 18M /tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest 18M /tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34 18M /tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest 18M /tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89 18M /tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest/wallets 18M /tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest 18M /tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d 18M /tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest/wallets 18M /tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest 18M /tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248 18M /tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest/wallets 18M /tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest 18M /tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda 18M /tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest/wallets 18M /tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest 18M /tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2 18M /tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest/wallets 18M /tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest 18M /tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3 18M /tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest/wallets 18M /tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest 18M /tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22 18M /tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest/wallets 18M /tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest 18M /tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f 18M /tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest/wallets 18M /tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest 18M /tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7 18M /tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest/wallets 18M /tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest 18M /tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc 18M /tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest/wallets 18M /tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest 18M /tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534 18M /tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest/blocks 4.0K /tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest/wallets 18M /tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest 18M /tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e ... ~/bitcoin# du -h -d1 /tmp 326G /tmp/test_common_Bitcoin Core 8.0K /tmp/tmpvt4junxw 40K /tmp/dotnet-installer 8.0K /tmp/NuGetScratch 326G /tmp -
agroce commented at 2:12 PM on July 17, 2021: contributor
It seems to accumulate over time -- a single fuzz run on process_message doesn't create a new entry if one exists, or add usage.
-
agroce commented at 2:13 PM on July 17, 2021: contributor
Presumably depend on inputs to process_message, but hours of libFuzzer don't seem to cause the problem for me.
-
agroce commented at 2:18 PM on July 17, 2021: contributor
And yes, at a guess, adding and setting this will avoid what OSS-Fuzz is running into, as well.
-
maflcko commented at 7:20 AM on July 18, 2021: member
Are there steps to reproduce?
Each process should get a new directory, but the directory is cleaned once the process exits normally: https://github.com/bitcoin/bitcoin/blob/4371e635d68251202f94353aa3124d74c78f7ec9/src/test/util/setup_common.cpp#L130
-
maflcko commented at 7:22 AM on July 18, 2021: member
So there will be plenty of leftover directories if you CTRL+C out of the process, but then the fix here wouldn't improve that either.
-
agroce commented at 9:19 AM on July 18, 2021: contributor
Well, the processes are running under a fuzzer's control. Some exit abnormally for reasons not clear, that aren't flagged as crashes. For Eclipser these may be QEMU failures.
Because the last process may leave a non-cleaned-up directory, the next one (Eclipser/AFL may run millions) will clean itup so they don't slowly accumulate over time.
Running any fuzzer that uses a new process per run or a fork will probably do this, is the way to reproduce it. I can give Eclipser install/run instructions if you want...
-
agroce commented at 10:01 AM on July 18, 2021: contributor
At a guess, what happens is that every 1000 or so runs die without cleaning up but without crashing. The next run will clean up, with this set. It's not ideal, and we should figure out the underlying solution, but it makes process-based fuzzing feasible on limited space systems.
-
agroce commented at 6:35 PM on July 18, 2021: contributor
I'll add a full guide later, but I think you can reliably reproduce by installing Eclipser 1.x:
> git clone https://github.com/SoftSec-KAIST/Eclipser.git > git checkout v1.x > ...following the instructions to add dependencies, install .net (I used v2.2 iirc, but it should not matter), found here: https://github.com/SoftSec-KAIST/Eclipser/tree/v1.x
and then fuzzing by, e.g.:
> FUZZ=process_message dotnet /root/Eclipser/build/Eclipser.dll fuzz -p src/test/fuzz/fuzz -v 1 -t 14400 -o eclipser_4h --src stdin -i /root/qa-assets/fuzz_seed_corpus/process_messageit will take a while (you can probably seed with something less than the full QA assets for process message, but I haven't tried that myself), but eventually data will start piling up in /tmp. The most likely culprit is QEMU bugs that terminate the process, but are not counted as crashes (some definitely happen during corpus replay).
-
agroce commented at 4:37 PM on July 19, 2021: contributor
Confirmed that running AFL without setting this:
- Produces many dirs in
/tmp/test_common_Bitcoin Core - These are due to certain inputs, not crashes or timeouts of some sort, since I used plain google AFL, which terminates if a seed input dies unusually, and this was during seed execution.
The rate was 273MB, over 17 directories that did not get cleaned up, over about 6K seed inputs (some from the QA assets, some discovered locally by Eclipser/libFuzzer). So whatever causes no cleanup is quite rare. Do we want to find out what inputs cause this? I can zip them up.
- Produces many dirs in
-
maflcko commented at 4:53 PM on July 19, 2021: member
Good find that this doesn't reproduce with libFuzzer. OSS-Fuzz reports also "Fuzzing Engine: afl".
OSS-Fuzz has tagged them with "UNREPRODUCIBLE", so I doubt the issue is with the seeds. Do you have the stdout/stderr from afl when the process exited abnormally?
-
agroce commented at 5:11 PM on July 19, 2021: contributor
Nothing ever exits abnormally. I killed afl/eclipser after a while, and observed the extra dirs in /tmp. Eventually, enough of these will cause src/test/fuzz/fuzz to abort with a "no space" message, producing spurious UNREPRODUCIBLE "crashes." The only way to diagnose I see is to check which inputs cause the problem (assuming it is deterministic). The UNREPRODUCIBLE is because the "failing" tests aren't the problem, the problem is earlier non-crashes that exhaust storage space.
-
maflcko commented at 5:31 PM on July 19, 2021: member
Well, something has to exit abnormally, unless I am misunderstanding something.
The datadir is created in the init function: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/process_messages.cpp#L26
The
staticshould keep the object around until after the process exits themainfunction normally. If the datadir is still around when it shouldn't, it implies the process didn't exit normally?There are at least two options:
- Something is wrong with our
mainfunction: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/fuzz.cpp#L103 . This seems unlikely, because IIRC OSS-Fuzz doesn't use themainfunction, but provides a wrapper around the LLVM* functions: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/fuzz.cpp#L87 (Upstream file should be https://github.com/AFLplusplus/AFLplusplus/blob/32a0d6ac31554a47dca591f8978982758fb87677/utils/aflpp_driver/aflpp_driver.c ) - Something is wrong in afl
- Something else
- Something is wrong with our
-
agroce commented at 5:39 PM on July 19, 2021: contributor
It's not just afl, though; it and eclipser are using very different approaches (source instrumentation via compilation, vs. QEMU on an un-instrumented binary).
I should have some useful data soon. I agree it's a mystery!
-
agroce commented at 5:55 PM on July 19, 2021: contributor
What's weird is any kind of abort/crash should stop afl corpus replay. Eclipser does have some QEMU failures during corpu replay, but those can't affect afl...
I'm scanning the QA assets now. But it's run 1700 without seeing any "cookie crumbs" in /tmp, so maybe something does depend on the fuzzing process. I'll also scan the (much larger) local corpus, in case it IS a deterministic problem with inputs, but not present in the QA asset seeds.
-
agroce commented at 6:12 PM on July 19, 2021: contributor
5500 without anything being left in /tmp. Hypothesis:
- nothing is wrong in bitcoin code
- the fuzzers here (unlike libfuzzer) cause some non-crash early exits of some sort (QEMU failures will explain Eclipser, maybe afl does some kind of silent timeout/fork-server business I don't know about during corpus replay), and those prevent cleanup
- over time this accumulates junk causing spurious failures like OSS-Fuzz sees, and I see
-
maflcko commented at 6:14 PM on July 19, 2021: member
Maybe this doesn't happen during replay, but only while searching for new fuzz inputs?
-
agroce commented at 6:39 PM on July 19, 2021: contributor
it happens during corpus replay, though, with afl, I know.
-
agroce commented at 6:42 PM on July 19, 2021: contributor
Ok, no crumbs in /tmp for standalone runs with no instrumentation. Checking files I generated.
-
agroce commented at 7:44 PM on July 19, 2021: contributor
Ok, 13K of my corpus inputs checked, and nothing. It's an artifact of the fuzzers, so I'm not sure there's a way to avoid this other than something like this PR, and setting the env variable for OSS-Fuzz. Something has to clean it up, and I'm not sure an external watchdog can even work for OSS-Fuzz. However, OSS-Fuzz may be, as you point out, skipping fuzz.cpp's main and running AFL on the libFuzzer harness basically, so something deeper may be needed there.
-
agroce commented at 8:58 PM on July 19, 2021: contributor
Nothing. I'll look to see if I can make this fix also fix the OSS-Fuzz issue, inside the libFuzzer signature function (just looked, and that is how afl/honggfuzz hook there, also). I'll see if I can find a way with minimal performance impact on libFuzzer, somehow (I think a static guard to do it once per run will work for that).
-
Try to fix OSS-Fuzz issues also 24ad8174da
-
Just do the remove_all in the common initialize de32caef5b
- fanquake referenced this in commit d542603c5a on Jul 20, 2021
-
Fails banman, so back to last solution 6344eb698c
-
agroce commented at 5:38 PM on July 20, 2021: contributor
@MarcoFalke do you have any idea why trying to remove the old dirs at the start of initialization for libFuzzer breaks thatt banman assertion? Is there initialization before that initialization, and I'm destroying it?
-
maflcko commented at 5:45 PM on July 20, 2021: member
Your solution only works when a single process is run on the machine. This assumption is violated when several fuzz targets are running or the same target in several processes.
-
maflcko commented at 5:46 PM on July 20, 2021: member
I think we only have two possible solutions here:
- Get rid of all disk acess (might take some time to realize)
- Fix the underlying bug
-
agroce commented at 6:02 PM on July 20, 2021: contributor
Oh I see, of course, how stupid of me.
My guess is the "underlying bug" is fuzzers producing terminations that 1) aren't crashes but 2) avoid cleanup code execution. I think anything QEMU based is going to do that occasionally, and looks like AFL also does it, even during corpus replay.
One idea, I don't know where the files come from, but could they be unlinked once opened? Depending on the access pattern that could work (since the handles will stay valid, but the files will disappear at process termination). But it won't work if they are reopened, rather than just referenced through a one-time handle.
-
agroce commented at 6:03 PM on July 20, 2021: contributor
No disk access will likely improve throughput, at least in some situations, anyway, even over /tmp.
-
Clarify how much this fixes 60afee48c3
-
agroce commented at 6:49 PM on July 20, 2021: contributor
What about using
tmpfile-- is that feasible? -
maflcko commented at 9:21 AM on July 21, 2021: member
If the program terminates abnormally, it is implementation-defined if these temporary files are deleted.
- sidhujag referenced this in commit 8d8f3f538c on Jul 23, 2021
-
maflcko commented at 10:04 AM on July 24, 2021: member
I haven't been able to reproduce, though the following diff should be able to reproduce the out-of-disk crash faster (even with an unlimited disk):
diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp index 5334c4623..5d8478632 100644 --- a/src/test/util/setup_common.cpp +++ b/src/test/util/setup_common.cpp @@ -90,6 +90,7 @@ BasicTestingSetup::BasicTestingSetup(const std::string& chainName, const std::ve }, extra_args); util::ThreadRename("test"); + Assert(!fs::exists(fs::temp_directory_path() / "test_common_" PACKAGE_NAME)); fs::create_directories(m_path_root); m_args.ForceSetArg("-datadir", m_path_root.string()); gArgs.ForceSetArg("-datadir", m_path_root.string()); -
agroce commented at 2:41 PM on July 24, 2021: contributor
Interesting. What fuzzer have you been trying to reproduce under? This should do it:
> rm -rf /tmp/test_common_Bitcoin* > git clone https://github.com/google/AFL.git > cd AFL > sudo make install > cd bitcoin > CC=afl-clang CXX=afl-clang++ ./configure --enable-fuzz > make clean; make -j 5 > FUZZ=process_message afl-fuzz -i qa-assets/fuzz_seed_corpus/process_message -o fuzz_afl -m 500 -t 30000 -- src/test/fuzz/fuzzgive it a while (maybe an hour or two?), and /tmp should contain more than one leftover in
test_common_Bitcoin Core. If that doesn't work, maybe it isn't a necessary result of afl/eclipser and there is some solution/environment aspect... -
maflcko commented at 9:29 AM on July 25, 2021: member
google/AFL is unmaintained and also not used by OSS-Fuzz, so I am using AFLplusplus (See https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md#quickstart-guide-1).
Though the fuzzers compiled with
afl-clang-ltocan't run the process_message harness (all inputs will time out).When I use the historic afl that starts a new process for each input, I can't observe the crash.
-
agroce commented at 9:52 AM on July 25, 2021: contributor
It will take quite some time depending on free space for a crash. Do you see the leftover files in /tmp?
-
maflcko commented at 10:24 AM on July 25, 2021: member
With the "historic afl" (new process each time), I had to adjust the patch for an early crash:
diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp index 5334c4623..01e969b49 100644 --- a/src/test/util/setup_common.cpp +++ b/src/test/util/setup_common.cpp @@ -90,6 +90,7 @@ BasicTestingSetup::BasicTestingSetup(const std::string& chainName, const std::ve }, extra_args); util::ThreadRename("test"); + Assert(fs::is_empty(fs::temp_directory_path() / "test_common_" PACKAGE_NAME)); fs::create_directories(m_path_root); m_args.ForceSetArg("-datadir", m_path_root.string()); gArgs.ForceSetArg("-datadir", m_path_root.string());It didn't crash after 12 hours.
-
agroce commented at 12:30 PM on July 25, 2021: contributor
Hmm, sorry, missed that. Ok, that's interesting. OSS-Fuzz seems to have seen these spurious space-triggered crashes, too. That's presumably with aflplusplus?
Is there some difference in the environments? I guess if so the QEMU failures for Eclipser might not always cause this problem, conceivably, if we can isolate the issue.
Probably unrelated: do you know why aflplusplus can't handle process_message? The lto instrumentation should generally be lower overhead/faster. The throughput on process_message with 2.57b is not great (3/sec or so) but aflplusplus times out every test even with crazy multi-second limits.
-
maflcko commented at 2:35 PM on July 25, 2021: member
do you know why aflplusplus can't handle process_message?
The "every input times out" issue also happens with the historic google/afl in llvm_mode. Though it doesn't happen with the aflpp_driver, otherwise OSS-Fuzz would have reported the issue, I presume?
The timeout won't happen if I comment out
SyncWithValidationInterfaceQueue. -
maflcko commented at 3:40 PM on July 25, 2021: member
Haven't been able to reproduce the "/tmp/ fills up" issue with
afl-clang-fast/++from AFL++, yet. -
maflcko commented at 1:10 PM on July 26, 2021: member
Ok, so the root of the problem seems to be that
forkcan't copy threads. Using another thread afterforkis probably UB?! Unfortunately there doesn't seem to be a way in AFL(pp) to disablefork, even for theaflpp_driver. So the solution might be to remove all threads, which would likely also fix #22551 .There is also libafl (https://github.com/AFLplusplus/LibAFL), which might enable an experience closer to libFuzzer than AFL itself.
-
agroce commented at 1:53 PM on July 26, 2021: contributor
One threadsafe(-ish) solution I thought of:
Instead of 'remove_all', crawl the dir and (even with no env variable, just unconditionally to avoid changes for OSS-Fuzz?) remove any subdirs in test_common... that are older than half an hour. They'll never accumulate to the point of causing space issues, and no other fuzzer should have an input running anywhere near that long.
-
maflcko commented at 4:15 PM on July 26, 2021: member
I used the following diff to remove all threads, but it didn't help
diff --git a/src/scheduler.cpp b/src/scheduler.cpp index 02ada969a4..73d666db3d 100644 --- a/src/scheduler.cpp +++ b/src/scheduler.cpp @@ -72,11 +72,7 @@ void CScheduler::serviceQueue() void CScheduler::schedule(CScheduler::Function f, std::chrono::system_clock::time_point t) { - { - LOCK(newTaskMutex); - taskQueue.insert(std::make_pair(t, f)); - } - newTaskScheduled.notify_one(); + f(); } void CScheduler::MockForward(std::chrono::seconds delta_seconds) diff --git a/src/scheduler.h b/src/scheduler.h index 9eec8c0fa0..135a6e2594 100644 --- a/src/scheduler.h +++ b/src/scheduler.h @@ -46,7 +46,7 @@ public: /** Call f once after the delta has passed */ void scheduleFromNow(Function f, std::chrono::milliseconds delta) { - schedule(std::move(f), std::chrono::system_clock::now() + delta); + // NOOP } /** diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp index 5334c4623e..57eeb91008 100644 --- a/src/test/util/setup_common.cpp +++ b/src/test/util/setup_common.cpp @@ -138,7 +138,6 @@ ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::ve // We have to run a scheduler thread to prevent ActivateBestChain // from blocking due to queue overrun. m_node.scheduler = std::make_unique<CScheduler>(); - m_node.scheduler->m_service_thread = std::thread(util::TraceThread, "scheduler", [&] { m_node.scheduler->serviceQueue(); }); GetMainSignals().RegisterBackgroundSignalScheduler(*m_node.scheduler); m_node.fee_estimator = std::make_unique<CBlockPolicyEstimator>(); @@ -147,10 +146,7 @@ ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::ve m_node.chainman = std::make_unique<ChainstateManager>(); m_node.chainman->m_blockman.m_block_tree_db = std::make_unique<CBlockTreeDB>(1 << 20, true); - // Start script-checking threads. Set g_parallel_script_checks to true so they are used. - constexpr int script_check_threads = 2; - StartScriptCheckWorkerThreads(script_check_threads); - g_parallel_script_checks = true; + g_parallel_script_checks = false; } ChainTestingSetup::~ChainTestingSetup() diff --git a/src/util/thread.cpp b/src/util/thread.cpp index 14be668685..0c7dcec9b3 100644 --- a/src/util/thread.cpp +++ b/src/util/thread.cpp @@ -12,6 +12,7 @@ void util::TraceThread(const char* thread_name, std::function<void()> thread_func) { + assert(false); util::ThreadRename(thread_name); try { LogPrintf("%s thread start\n", thread_name); -
agroce commented at 5:49 PM on July 26, 2021: contributor
What do you think of simply automatically killing "old" subdirs of
test_common_Bitcoin Corein process-based fuzzing (do nothing to libFuzzer target based fuzzing)? -
maflcko commented at 7:00 AM on July 27, 2021: member
separate-process-based fuzzing should be unaffected. See comments #22472 (comment) (no crash after 12 hours with my patch) and your screenshot in #22551 (also no crash?, disk not filled up?)
With fork-based fuzzing I am seeing a background noise of crashes (though unrelated to the disk space issue).
Preparing the affected fuzz targets for
forkmight also magically fix the other issues, I suspect. -
maflcko commented at 8:54 AM on July 27, 2021: member
cc @practicalswift Any ideas?
-
agroce commented at 1:36 PM on July 27, 2021: contributor
Sorry, I didn't mean true separate process-based (which might also see this problem, depending on the cause). I meant anything that's not doing libFuzzer "test is a function call" approach. My big afl run isn't crashing because it's compiled with my PR and the environment variable is set to TRUE. :)
For me, afl or Eclipser in docker ubuntu 20.04 both reliably end up filling /tmp and failing most tests due to that, without the patch, so I'm running with the patch.
-
practicalswift commented at 10:40 AM on July 30, 2021: contributor
Concept ACK on working around this issue (if we cannot fix the root cause).
Instead of removing the entire tree
/tmp/test_common_Bitcoin Core/on exit would it be possible to remove only the specific sub-directory used during the fuzzing session (say/tmp/test_common_Bitcoin Core/6d8a[…]dc38/)? -
agroce commented at 2:15 PM on July 30, 2021: contributor
The trick is it has to remove not on exit (since failing to do so as it should is the problem) but on entry, when it's the "last" one it needs to kill, whose name is unknown.
-
practicalswift commented at 11:08 PM on July 30, 2021: contributor
The trick is it has to remove not on exit (since failing to do so as it should is the problem) but on entry, when it's the "last" one it needs to kill, whose name is unknown.
Oh, of course! I misunderstood the problem. Sorry! :)
-
agroce commented at 11:13 PM on August 4, 2021: contributor
Another idea:
If this isn't causing much issue on OSS-Fuzz, perhaps a stopgap for Eclipser etc. fuzzing would be to remove code inside fuzz and add a script users can launch in background to occasionally clean up old leftover directories?
-
maflcko commented at 5:47 AM on August 5, 2021: member
I'd still prefer to fix the underlying issue over a temporary workaround.
In #22472 (comment) I removed all threads. I wonder if I have to remove all locks too for
forkto work properly in afl? -
agroce commented at 6:23 AM on August 5, 2021: contributor
I am concerned QEMU failures are going to always produce leftovers, for Eclipser...
-
Merge branch 'bitcoin:master' into master 971aab70b0
-
DrahtBot commented at 6:15 AM on January 8, 2022: contributor
<!--e57a25ab6845829454e8d69fc972939a-->
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
<!--174a7506f384e20aa4161008e828411d-->
Conflicts
Reviewers, this pull request conflicts with the following ones:
- #20744 (Use std::filesystem. Remove Boost Filesystem & System by fanquake)
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
- DrahtBot added the label Needs rebase on Feb 3, 2022
-
DrahtBot commented at 4:11 PM on February 3, 2022: contributor
<!--cf906140f33d8803c4a75a2196329ecb-->
🐙 This pull request conflicts with the target branch and needs rebase.
<sub>Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".</sub>
-
fanquake commented at 1:35 PM on April 26, 2022: member
What is the status of this? Needs to be reworked to remove
boost::filesystem. -
agroce commented at 4:44 PM on April 26, 2022: contributor
@MarcoFalke wanted a root cause fix; I haven't been actively fuzzing bitcoin core lately, so am not sure if that ever happened? I assume not. I suspect for some fuzzers, it may be needed even with root-cause fixes, for some abnormal exits and long fuzzer runs.
-
maflcko commented at 8:40 AM on April 29, 2022: member
Yeah, I'd prefer to fix the underlying issue. In the meantime, I think it is fine to provide a temporary opt-in workaround.
-
maflcko commented at 8:40 AM on April 29, 2022: member
Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits
-
agroce commented at 5:16 PM on April 29, 2022: contributor
@MarcoFalke is it ready for final squash? Looks like boost::filesystem also needs to go?
-
maflcko commented at 5:45 PM on April 29, 2022: member
Yes, if you want to work on this again, it needs:
- Rebase
- Squash
- boost::fs -> <fs.h>
- fanquake added the label Up for grabs on Aug 15, 2022
- fanquake closed this on Aug 15, 2022
- bitcoin locked this on Aug 15, 2023
- maflcko removed the label Up for grabs on Dec 8, 2023
