fuzz: Add environment option to keep /tmp/ clean

agroce commented at 4:30 pm on July 16, 2021: contributor

This avoids the possibility of flooding /tmp during long fuzzing campaigns by allowing an environment variable to be set that causes the test_common_Bitcoin Core directory to be cleared after each fuzz run. Because libFuzzer campaigns seem to produce less data in /tmp, I believe the possibility may not have been noticed before. A 4 hour Eclipser run produced nearly 350GB in test_common_Bitcoin Core. Tested locally.

Add environment option to keep /tmp/ clean

This avoids the possibility of flooding /tmp during long fuzzing campaigns

c07998c0d1

DrahtBot added the label Tests on Jul 16, 2021

agroce commented at 6:03 pm on July 16, 2021: contributor

Looks as if I need to add -lstdc++fs somewhere, perhaps, to build in all envs. Anyone know where?

Use boost to avoid adding new link 8e03c48a5b

Type for 2nd arg to boost::filesystem::remove_all d13ce4e81d

Remove extra semicolon 35323b08d3

Include boost error codes 5c31c13708

Just call, don't add dependency d0beba5ae2

maflcko commented at 8:34 am on July 17, 2021: member

Is this only an issue with Eclipser?

If not, can you please provide a list of the largest files/folders in that directory?

Locally, I am seeing that the size stays constant, as it should be:

0$ FUZZ=process_message_filteradd ./src/test/fuzz/fuzz

0$ du -s /tmp/test_common_Bitcoin\ Core/6d8ae75a417046414e8a64bb8398d1382b6801a363c1cd6c08512de1f63fdc38/
117412	/tmp/test_common_Bitcoin Core/6d8ae75a417046414e8a64bb8398d1382b6801a363c1cd6c08512de1f63fdc38/

maflcko commented at 8:35 am on July 17, 2021: member

OSS-Fuzz seems to be running into the same issue: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=35027

maflcko commented at 11:46 am on July 17, 2021: member

Also, if you want you can add Eclipser instructions to https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md so that it will be easier to get started with it in the future.

agroce commented at 1:53 pm on July 17, 2021: contributor

I haven’t tried with AFL. It may be only under rare circumstances? But not in libFuzzer (or an AFL persist loop?)? I’ll show you a bad run later. You end up with many (very many) entries with different hashes in the ‘/tmp/test_common_Bitcoin Core/` dir. Are you saying multiple fuzz processes should use the same one? Maybe if one is launched while another is still somehow live?

Yes, I plan to add an Eclipser guide once I’ve hashed out some issues.

agroce commented at 2:11 pm on July 17, 2021: contributor

It looks like:

  0~/bitcoin# du -h /tmp
  118M	/tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest/blocks
  24.0K	/tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest/wallets
  318M	/tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc/regtest
  418M	/tmp/test_common_Bitcoin Core/0012b9a513fd0b28db3df0331f02fcbf6a834201f7ef68173f1b8daa75dfadcc
  518M	/tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest/blocks
  64.0K	/tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest/wallets
  718M	/tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d/regtest
  818M	/tmp/test_common_Bitcoin Core/00136fcb430abc4c8b5f733922ff8c7f3f9f07b7ff8f1af9248ea46f42ac480d
  918M	/tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest/blocks
 104.0K	/tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest/wallets
 1118M	/tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f/regtest
 1218M	/tmp/test_common_Bitcoin Core/004b9bf13ce846f7c49ab4ec9d76cc8a1275d30d2307af9f673f983bdba6a25f
 1318M	/tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest/blocks
 144.0K	/tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest/wallets
 1518M	/tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07/regtest
 1618M	/tmp/test_common_Bitcoin Core/007014aa13890609624a46381771511a56017cd17d6ed2eda53d39f3dc162a07
 1718M	/tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest/blocks
 184.0K	/tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest/wallets
 1918M	/tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5/regtest
 2018M	/tmp/test_common_Bitcoin Core/0084f7ff992472f448db8771be0f6c2aaa0d3b2fe7704a59163eaa80be4dd0e5
 2118M	/tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest/blocks
 224.0K	/tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest/wallets
 2318M	/tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30/regtest
 2418M	/tmp/test_common_Bitcoin Core/009a1fbaf78c7971cb5f4e231dfb8dbccc6bea685454e8fd72aa6c1c37f09a30
 2518M	/tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest/blocks
 264.0K	/tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest/wallets
 2718M	/tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9/regtest
 2818M	/tmp/test_common_Bitcoin Core/00bf7dba03bcd4d33c981f7349c7bb25b70af10027b7eaf60ae69868c05c05f9
 2918M	/tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest/blocks
 304.0K	/tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest/wallets
 3118M	/tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901/regtest
 3218M	/tmp/test_common_Bitcoin Core/00efc6af9fed06d50b3584ca164a4ae31d84dfd591054109a618d80a8d087901
 3318M	/tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest/blocks
 344.0K	/tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest/wallets
 3518M	/tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f/regtest
 3618M	/tmp/test_common_Bitcoin Core/00f924f4db99aad9b3e730b92984755b22fd2881e21a672d2349d139ec0a209f
 3718M	/tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest/blocks
 384.0K	/tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest/wallets
 3918M	/tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18/regtest
 4018M	/tmp/test_common_Bitcoin Core/010f0a77122057c7b8ccbf4c2c78244d866162dfa647f4c8b2b1cf839069ed18
 4118M	/tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest/blocks
 424.0K	/tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest/wallets
 4318M	/tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f/regtest
 4418M	/tmp/test_common_Bitcoin Core/01326a4a91ac1c0d2ec916a87e2ccb54f4222a258b7757d4dbb966127887f71f
 4518M	/tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest/blocks
 464.0K	/tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest/wallets
 4718M	/tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81/regtest
 4818M	/tmp/test_common_Bitcoin Core/01461b4db8cad01b43ed76de808eb0ad1ca81986e3d1e1e6c0254c7d5299de81
 4918M	/tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest/blocks
 504.0K	/tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest/wallets
 5118M	/tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5/regtest
 5218M	/tmp/test_common_Bitcoin Core/017f522e765ed75f30fba1fb5f4cc95f2a978fbc51dd0d4c631c16bf3e8506d5
 5318M	/tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest/blocks
 544.0K	/tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest/wallets
 5518M	/tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0/regtest
 5618M	/tmp/test_common_Bitcoin Core/020744d0ac61a2f956ce1bfffcf906f8dcce080ab761ecb6f4df9ddbc8b499a0
 5718M	/tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest/blocks
 584.0K	/tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest/wallets
 5918M	/tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b/regtest
 6018M	/tmp/test_common_Bitcoin Core/02375b66bd7f1ae717699d929e8d818e4c1c87b1419cbd6044eaa15a49f97f3b
 6118M	/tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest/blocks
 624.0K	/tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest/wallets
 6318M	/tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347/regtest
 6418M	/tmp/test_common_Bitcoin Core/026308149c4187072f081b8b1fe0de6b4e120e274b033d7d637851b9dcc02347
 6518M	/tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest/blocks
 664.0K	/tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest/wallets
 6718M	/tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583/regtest
 6818M	/tmp/test_common_Bitcoin Core/026c2d651b7395790c0111e6b6999763bdca7cd9402e55c2cc7f8be988e3f583
 6918M	/tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest/blocks
 704.0K	/tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest/wallets
 7118M	/tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3/regtest
 7218M	/tmp/test_common_Bitcoin Core/02703398296ec9012301cd5c217323b61a9ab42c752d69e675b645c0aa1afda3
 7318M	/tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest/blocks
 744.0K	/tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest/wallets
 7518M	/tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e/regtest
 7618M	/tmp/test_common_Bitcoin Core/027cd8a04ec8b9f8ef90c9f0bc1167d401fa1e6a9736252f74ca920255d2d03e
 7718M	/tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest/blocks
 784.0K	/tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest/wallets
 7918M	/tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d/regtest
 8018M	/tmp/test_common_Bitcoin Core/028a59f89f636c5dff8796b3dbfb54982c27796c3eb7f82761d9d984528bc77d
 8118M	/tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest/blocks
 824.0K	/tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest/wallets
 8318M	/tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617/regtest
 8418M	/tmp/test_common_Bitcoin Core/02c110bb57586463e1fa98a55a6c13ce0c99339c812f790f357a1e03c6d20617
 8518M	/tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest/blocks
 864.0K	/tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest/wallets
 8718M	/tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e/regtest
 8818M	/tmp/test_common_Bitcoin Core/02c55573d63949a93d76c35e8b836cec6cb2e5842d415c17a5b889bf6d20514e
 8918M	/tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest/blocks
 904.0K	/tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest/wallets
 9118M	/tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf/regtest
 9218M	/tmp/test_common_Bitcoin Core/02ca57b860892a9169c2b71b16e021726f6b6ebc1676d7c56b04a92bf1e34bdf
 9318M	/tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest/blocks
 944.0K	/tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest/wallets
 9518M	/tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba/regtest
 9618M	/tmp/test_common_Bitcoin Core/02d8afdf5d670c28b8f256169f4ab3ae2cc614b5b1028894ec9a0ca52789eaba
 9718M	/tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest/blocks
 984.0K	/tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest/wallets
 9918M	/tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797/regtest
10018M	/tmp/test_common_Bitcoin Core/02f58f3b3d46aa02010a9a95cd4e59a7873debf6880b0eea44411a4593a57797
10118M	/tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest/blocks
1024.0K	/tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest/wallets
10318M	/tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08/regtest
10418M	/tmp/test_common_Bitcoin Core/0346c38159421293b2b0cdbea2aa2b19a0397cff6cc0789e099d197eea940d08
10518M	/tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest/blocks
1064.0K	/tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest/wallets
10718M	/tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f/regtest
10818M	/tmp/test_common_Bitcoin Core/03494a505107c333456a2779d0515ef7af29b35374339b32c8ebe06a534dfe7f
10918M	/tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest/blocks
1104.0K	/tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest/wallets
11118M	/tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91/regtest
11218M	/tmp/test_common_Bitcoin Core/0357a644acacfb504b59f29bd2e8d03ef1187485644383ac05ab87f6ec538f91
11318M	/tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest/blocks
1144.0K	/tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest/wallets
11518M	/tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d/regtest
11618M	/tmp/test_common_Bitcoin Core/0357fd717c29b365b6dbed007c35ccb363331f2c3cf253402ac9d9100c12806d
11718M	/tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest/blocks
1184.0K	/tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest/wallets
11918M	/tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787/regtest
12018M	/tmp/test_common_Bitcoin Core/036a58de05ccdc6bc51df8cef6bb1daefc241d6c8eb73b8cd4168e23f989f787
1214.0K	/tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest/blocks
1224.0K	/tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest/wallets
12312K	/tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0/regtest
12416K	/tmp/test_common_Bitcoin Core/037bdfa9e72b35850a1399e4948340ab62ab8b3a376e4e77cb088a94992ce7f0
12518M	/tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest/blocks
1264.0K	/tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest/wallets
12718M	/tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30/regtest
12818M	/tmp/test_common_Bitcoin Core/037c148c066ca97fb246f02b9c54fda55fdaa74f9f4bbe6a05856a3f49eb6c30
12918M	/tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest/blocks
1304.0K	/tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest/wallets
13118M	/tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8/regtest
13218M	/tmp/test_common_Bitcoin Core/038e84ed36fadc2a5d6d5a43913e618ca18ef234ef53584c82d09a59b255f1f8
13318M	/tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest/blocks
1344.0K	/tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest/wallets
13518M	/tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57/regtest
13618M	/tmp/test_common_Bitcoin Core/03a4aceded82b0a68f244d598788c9e2b89c3207dab506e088f8a53c606feb57
13718M	/tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest/blocks
1384.0K	/tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest/wallets
13918M	/tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574/regtest
14018M	/tmp/test_common_Bitcoin Core/03d8b8aa6a52197efeb7f699ddaffddc0f67cca923546560e13f94cb7a493574
14118M	/tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest/blocks
1424.0K	/tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest/wallets
14318M	/tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34/regtest
14418M	/tmp/test_common_Bitcoin Core/03fd0e61b4d88a358609f3488e5eff4cca91b13e304bc93f8f9307c2ef0b9e34
14518M	/tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest/blocks
1464.0K	/tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest/wallets
14718M	/tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89/regtest
14818M	/tmp/test_common_Bitcoin Core/0426712d172dd81a2f22ec9ac54746e40e261bde378b8ea4ce731fb3f0355c89
14918M	/tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest/blocks
1504.0K	/tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest/wallets
15118M	/tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d/regtest
15218M	/tmp/test_common_Bitcoin Core/0453b4e042c331219f9bcc35980b793f6d08a2fc4630a92ea86dd5ca3a33a58d
15318M	/tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest/blocks
1544.0K	/tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest/wallets
15518M	/tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248/regtest
15618M	/tmp/test_common_Bitcoin Core/09a199770b4189e958979a2f03f1195f55243377f48986a5fb29080586954248
15718M	/tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest/blocks
1584.0K	/tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest/wallets
15918M	/tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda/regtest
16018M	/tmp/test_common_Bitcoin Core/35fffd27d8e296288e918226425647fb665dadc2d98c58663554904aa4bd5eda
16118M	/tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest/blocks
1624.0K	/tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest/wallets
16318M	/tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2/regtest
16418M	/tmp/test_common_Bitcoin Core/634c11f7b43be88e026ea2912840f24f2951f1dee29610b5acda2b7d453316e2
16518M	/tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest/blocks
1664.0K	/tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest/wallets
16718M	/tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3/regtest
16818M	/tmp/test_common_Bitcoin Core/90a0a72854d6a7300470b036de39970feb5c8e30c5800003ec71430e28b213b3
16918M	/tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest/blocks
1704.0K	/tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest/wallets
17118M	/tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22/regtest
17218M	/tmp/test_common_Bitcoin Core/be1068f3df670372b27cd4d040d15788aeaf691777ad172248b4d414420f7e22
17318M	/tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest/blocks
1744.0K	/tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest/wallets
17518M	/tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f/regtest
17618M	/tmp/test_common_Bitcoin Core/e9e458e2672b64ea2e1712c9994d398eb9f2d75ecd4a07e427eb73dfba13022f
17718M	/tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest/blocks
1784.0K	/tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest/wallets
17918M	/tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7/regtest
18018M	/tmp/test_common_Bitcoin Core/045e2c0d83fd90a0c29f821e591ac0a4567760a62bb11e0041a1e2e42bda71d7
18118M	/tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest/blocks
1824.0K	/tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest/wallets
18318M	/tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc/regtest
18418M	/tmp/test_common_Bitcoin Core/04a3aedf347c800471d038dd1d89c64c8ac5e7116dd453fd2b2b79a8edb7cffc
18518M	/tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest/blocks
1864.0K	/tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest/wallets
18718M	/tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534/regtest
18818M	/tmp/test_common_Bitcoin Core/04a4770db443843483092fb662f93d2eac5ec7352264d58443b2df5893434534
18918M	/tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest/blocks
1904.0K	/tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest/wallets
19118M	/tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e/regtest
19218M	/tmp/test_common_Bitcoin Core/04e1f4a542d1943faeaf6dddfe7bb1bb981be7769112f866a3cd06db39a1dc7e
193...
194~/bitcoin# du -h -d1 /tmp
195326G	/tmp/test_common_Bitcoin Core
1968.0K	/tmp/tmpvt4junxw
19740K	/tmp/dotnet-installer
1988.0K	/tmp/NuGetScratch
199326G	/tmp

agroce commented at 2:12 pm on July 17, 2021: contributor

It seems to accumulate over time – a single fuzz run on process_message doesn’t create a new entry if one exists, or add usage.

agroce commented at 2:13 pm on July 17, 2021: contributor

Presumably depend on inputs to process_message, but hours of libFuzzer don’t seem to cause the problem for me.

agroce commented at 2:18 pm on July 17, 2021: contributor

And yes, at a guess, adding and setting this will avoid what OSS-Fuzz is running into, as well.

maflcko commented at 7:20 am on July 18, 2021: member

Are there steps to reproduce?

Each process should get a new directory, but the directory is cleaned once the process exits normally: https://github.com/bitcoin/bitcoin/blob/4371e635d68251202f94353aa3124d74c78f7ec9/src/test/util/setup_common.cpp#L130

maflcko commented at 7:22 am on July 18, 2021: member

So there will be plenty of leftover directories if you CTRL+C out of the process, but then the fix here wouldn’t improve that either.

agroce commented at 9:19 am on July 18, 2021: contributor

Well, the processes are running under a fuzzer’s control. Some exit abnormally for reasons not clear, that aren’t flagged as crashes. For Eclipser these may be QEMU failures.

Because the last process may leave a non-cleaned-up directory, the next one (Eclipser/AFL may run millions) will clean itup so they don’t slowly accumulate over time.

Running any fuzzer that uses a new process per run or a fork will probably do this, is the way to reproduce it. I can give Eclipser install/run instructions if you want…

agroce commented at 10:01 am on July 18, 2021: contributor

At a guess, what happens is that every 1000 or so runs die without cleaning up but without crashing. The next run will clean up, with this set. It’s not ideal, and we should figure out the underlying solution, but it makes process-based fuzzing feasible on limited space systems.

agroce commented at 6:35 pm on July 18, 2021: contributor

I’ll add a full guide later, but I think you can reliably reproduce by installing Eclipser 1.x:

0> git clone https://github.com/SoftSec-KAIST/Eclipser.git
1> git checkout v1.x
2> ...

following the instructions to add dependencies, install .net (I used v2.2 iirc, but it should not matter), found here: https://github.com/SoftSec-KAIST/Eclipser/tree/v1.x

and then fuzzing by, e.g.:

0> FUZZ=process_message dotnet /root/Eclipser/build/Eclipser.dll fuzz -p src/test/fuzz/fuzz -v 1 -t 14400 -o eclipser_4h --src stdin -i /root/qa-assets/fuzz_seed_corpus/process_message

it will take a while (you can probably seed with something less than the full QA assets for process message, but I haven’t tried that myself), but eventually data will start piling up in /tmp. The most likely culprit is QEMU bugs that terminate the process, but are not counted as crashes (some definitely happen during corpus replay).

agroce commented at 4:37 pm on July 19, 2021: contributor

Confirmed that running AFL without setting this:

Produces many dirs in /tmp/test_common_Bitcoin Core
These are due to certain inputs, not crashes or timeouts of some sort, since I used plain google AFL, which terminates if a seed input dies unusually, and this was during seed execution.

The rate was 273MB, over 17 directories that did not get cleaned up, over about 6K seed inputs (some from the QA assets, some discovered locally by Eclipser/libFuzzer). So whatever causes no cleanup is quite rare. Do we want to find out what inputs cause this? I can zip them up.

maflcko commented at 4:53 pm on July 19, 2021: member

Good find that this doesn’t reproduce with libFuzzer. OSS-Fuzz reports also “Fuzzing Engine: afl”.

OSS-Fuzz has tagged them with “UNREPRODUCIBLE”, so I doubt the issue is with the seeds. Do you have the stdout/stderr from afl when the process exited abnormally?

agroce commented at 5:11 pm on July 19, 2021: contributor

Nothing ever exits abnormally. I killed afl/eclipser after a while, and observed the extra dirs in /tmp. Eventually, enough of these will cause src/test/fuzz/fuzz to abort with a “no space” message, producing spurious UNREPRODUCIBLE “crashes.” The only way to diagnose I see is to check which inputs cause the problem (assuming it is deterministic). The UNREPRODUCIBLE is because the “failing” tests aren’t the problem, the problem is earlier non-crashes that exhaust storage space.

maflcko commented at 5:31 pm on July 19, 2021: member

Well, something has to exit abnormally, unless I am misunderstanding something.

The datadir is created in the init function: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/process_messages.cpp#L26

The static should keep the object around until after the process exits the main function normally. If the datadir is still around when it shouldn’t, it implies the process didn’t exit normally?

There are at least two options:

Something is wrong with our main function: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/fuzz.cpp#L103 . This seems unlikely, because IIRC OSS-Fuzz doesn’t use the main function, but provides a wrapper around the LLVM* functions: https://github.com/bitcoin/bitcoin/blob/54e31742d208eb98ce706aaa6bbd4b023f42c3a5/src/test/fuzz/fuzz.cpp#L87 (Upstream file should be https://github.com/AFLplusplus/AFLplusplus/blob/32a0d6ac31554a47dca591f8978982758fb87677/utils/aflpp_driver/aflpp_driver.c )
Something is wrong in afl
Something else

agroce commented at 5:39 pm on July 19, 2021: contributor

It’s not just afl, though; it and eclipser are using very different approaches (source instrumentation via compilation, vs. QEMU on an un-instrumented binary).

I should have some useful data soon. I agree it’s a mystery!

agroce commented at 5:55 pm on July 19, 2021: contributor

What’s weird is any kind of abort/crash should stop afl corpus replay. Eclipser does have some QEMU failures during corpu replay, but those can’t affect afl…

I’m scanning the QA assets now. But it’s run 1700 without seeing any “cookie crumbs” in /tmp, so maybe something does depend on the fuzzing process. I’ll also scan the (much larger) local corpus, in case it IS a deterministic problem with inputs, but not present in the QA asset seeds.

agroce commented at 6:12 pm on July 19, 2021: contributor

5500 without anything being left in /tmp. Hypothesis:

nothing is wrong in bitcoin code
the fuzzers here (unlike libfuzzer) cause some non-crash early exits of some sort (QEMU failures will explain Eclipser, maybe afl does some kind of silent timeout/fork-server business I don’t know about during corpus replay), and those prevent cleanup
over time this accumulates junk causing spurious failures like OSS-Fuzz sees, and I see

maflcko commented at 6:14 pm on July 19, 2021: member

Maybe this doesn’t happen during replay, but only while searching for new fuzz inputs?

agroce commented at 6:39 pm on July 19, 2021: contributor

it happens during corpus replay, though, with afl, I know.

agroce commented at 6:42 pm on July 19, 2021: contributor

Ok, no crumbs in /tmp for standalone runs with no instrumentation. Checking files I generated.

agroce commented at 7:44 pm on July 19, 2021: contributor

Ok, 13K of my corpus inputs checked, and nothing. It’s an artifact of the fuzzers, so I’m not sure there’s a way to avoid this other than something like this PR, and setting the env variable for OSS-Fuzz. Something has to clean it up, and I’m not sure an external watchdog can even work for OSS-Fuzz. However, OSS-Fuzz may be, as you point out, skipping fuzz.cpp’s main and running AFL on the libFuzzer harness basically, so something deeper may be needed there.

agroce commented at 8:58 pm on July 19, 2021: contributor

Nothing. I’ll look to see if I can make this fix also fix the OSS-Fuzz issue, inside the libFuzzer signature function (just looked, and that is how afl/honggfuzz hook there, also). I’ll see if I can find a way with minimal performance impact on libFuzzer, somehow (I think a static guard to do it once per run will work for that).

Try to fix OSS-Fuzz issues also 24ad8174da

Just do the remove_all in the common initialize de32caef5b

fanquake referenced this in commit d542603c5a on Jul 20, 2021

Fails banman, so back to last solution 6344eb698c

agroce commented at 5:38 pm on July 20, 2021: contributor

@MarcoFalke do you have any idea why trying to remove the old dirs at the start of initialization for libFuzzer breaks thatt banman assertion? Is there initialization before that initialization, and I’m destroying it?

maflcko commented at 5:45 pm on July 20, 2021: member

Your solution only works when a single process is run on the machine. This assumption is violated when several fuzz targets are running or the same target in several processes.

maflcko commented at 5:46 pm on July 20, 2021: member

I think we only have two possible solutions here:

Get rid of all disk acess (might take some time to realize)
Fix the underlying bug

agroce commented at 6:02 pm on July 20, 2021: contributor

Oh I see, of course, how stupid of me.

My guess is the “underlying bug” is fuzzers producing terminations that 1) aren’t crashes but 2) avoid cleanup code execution. I think anything QEMU based is going to do that occasionally, and looks like AFL also does it, even during corpus replay.

One idea, I don’t know where the files come from, but could they be unlinked once opened? Depending on the access pattern that could work (since the handles will stay valid, but the files will disappear at process termination). But it won’t work if they are reopened, rather than just referenced through a one-time handle.

agroce commented at 6:03 pm on July 20, 2021: contributor

No disk access will likely improve throughput, at least in some situations, anyway, even over /tmp.

Clarify how much this fixes 60afee48c3

agroce commented at 6:49 pm on July 20, 2021: contributor

What about using tmpfile – is that feasible?

maflcko commented at 9:21 am on July 21, 2021: member

If the program terminates abnormally, it is implementation-defined if these temporary files are deleted.

https://en.cppreference.com/w/cpp/io/c/tmpfile

sidhujag referenced this in commit 8d8f3f538c on Jul 23, 2021

maflcko commented at 10:04 am on July 24, 2021: member

I haven’t been able to reproduce, though the following diff should be able to reproduce the out-of-disk crash faster (even with an unlimited disk):

 0diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp
 1index 5334c4623..5d8478632 100644
 2--- a/src/test/util/setup_common.cpp
 3+++ b/src/test/util/setup_common.cpp
 4@@ -90,6 +90,7 @@ BasicTestingSetup::BasicTestingSetup(const std::string& chainName, const std::ve
 5         },
 6         extra_args);
 7     util::ThreadRename("test");
 8+    Assert(!fs::exists(fs::temp_directory_path() / "test_common_" PACKAGE_NAME));
 9     fs::create_directories(m_path_root);
10     m_args.ForceSetArg("-datadir", m_path_root.string());
11     gArgs.ForceSetArg("-datadir", m_path_root.string());

agroce commented at 2:41 pm on July 24, 2021: contributor

Interesting. What fuzzer have you been trying to reproduce under? This should do it:

0> rm -rf /tmp/test_common_Bitcoin*
1> git clone https://github.com/google/AFL.git
2> cd AFL
3> sudo make install
4> cd bitcoin
5> CC=afl-clang CXX=afl-clang++ ./configure --enable-fuzz
6> make clean; make -j 5
7> FUZZ=process_message afl-fuzz -i qa-assets/fuzz_seed_corpus/process_message -o fuzz_afl -m 500 -t 30000 -- src/test/fuzz/fuzz

give it a while (maybe an hour or two?), and /tmp should contain more than one leftover in test_common_Bitcoin Core. If that doesn’t work, maybe it isn’t a necessary result of afl/eclipser and there is some solution/environment aspect…

maflcko commented at 9:29 am on July 25, 2021: member

google/AFL is unmaintained and also not used by OSS-Fuzz, so I am using AFLplusplus (See https://github.com/bitcoin/bitcoin/blob/master/doc/fuzzing.md#quickstart-guide-1).

Though the fuzzers compiled with afl-clang-lto can’t run the process_message harness (all inputs will time out).

When I use the historic afl that starts a new process for each input, I can’t observe the crash.

agroce commented at 9:52 am on July 25, 2021: contributor

It will take quite some time depending on free space for a crash. Do you see the leftover files in /tmp?

maflcko commented at 10:24 am on July 25, 2021: member

With the “historic afl” (new process each time), I had to adjust the patch for an early crash:

 0diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp
 1index 5334c4623..01e969b49 100644
 2--- a/src/test/util/setup_common.cpp
 3+++ b/src/test/util/setup_common.cpp
 4@@ -90,6 +90,7 @@ BasicTestingSetup::BasicTestingSetup(const std::string& chainName, const std::ve
 5         },
 6         extra_args);
 7     util::ThreadRename("test");
 8+    Assert(fs::is_empty(fs::temp_directory_path() / "test_common_" PACKAGE_NAME));
 9     fs::create_directories(m_path_root);
10     m_args.ForceSetArg("-datadir", m_path_root.string());
11     gArgs.ForceSetArg("-datadir", m_path_root.string());

It didn’t crash after 12 hours.

agroce commented at 12:30 pm on July 25, 2021: contributor

Hmm, sorry, missed that. Ok, that’s interesting. OSS-Fuzz seems to have seen these spurious space-triggered crashes, too. That’s presumably with aflplusplus?

Is there some difference in the environments? I guess if so the QEMU failures for Eclipser might not always cause this problem, conceivably, if we can isolate the issue.

Probably unrelated: do you know why aflplusplus can’t handle process_message? The lto instrumentation should generally be lower overhead/faster. The throughput on process_message with 2.57b is not great (3/sec or so) but aflplusplus times out every test even with crazy multi-second limits.

maflcko commented at 2:35 pm on July 25, 2021: member

do you know why aflplusplus can’t handle process_message?

The “every input times out” issue also happens with the historic google/afl in llvm_mode. Though it doesn’t happen with the aflpp_driver, otherwise OSS-Fuzz would have reported the issue, I presume?

The timeout won’t happen if I comment out SyncWithValidationInterfaceQueue.

maflcko commented at 3:40 pm on July 25, 2021: member

Haven’t been able to reproduce the “/tmp/ fills up” issue with afl-clang-fast/++ from AFL++, yet.

maflcko commented at 1:10 pm on July 26, 2021: member

Ok, so the root of the problem seems to be that fork can’t copy threads. Using another thread after fork is probably UB?! Unfortunately there doesn’t seem to be a way in AFL(pp) to disable fork, even for the aflpp_driver. So the solution might be to remove all threads, which would likely also fix #22551 .

There is also libafl (https://github.com/AFLplusplus/LibAFL), which might enable an experience closer to libFuzzer than AFL itself.

agroce commented at 1:53 pm on July 26, 2021: contributor

One threadsafe(-ish) solution I thought of:

Instead of ‘remove_all’, crawl the dir and (even with no env variable, just unconditionally to avoid changes for OSS-Fuzz?) remove any subdirs in test_common… that are older than half an hour. They’ll never accumulate to the point of causing space issues, and no other fuzzer should have an input running anywhere near that long.

maflcko commented at 4:15 pm on July 26, 2021: member

I used the following diff to remove all threads, but it didn’t help

 0diff --git a/src/scheduler.cpp b/src/scheduler.cpp
 1index 02ada969a4..73d666db3d 100644
 2--- a/src/scheduler.cpp
 3+++ b/src/scheduler.cpp
 4@@ -72,11 +72,7 @@ void CScheduler::serviceQueue()
 5 
 6 void CScheduler::schedule(CScheduler::Function f, std::chrono::system_clock::time_point t)
 7 {
 8-    {
 9-        LOCK(newTaskMutex);
10-        taskQueue.insert(std::make_pair(t, f));
11-    }
12-    newTaskScheduled.notify_one();
13+    f();
14 }
15 
16 void CScheduler::MockForward(std::chrono::seconds delta_seconds)
17diff --git a/src/scheduler.h b/src/scheduler.h
18index 9eec8c0fa0..135a6e2594 100644
19--- a/src/scheduler.h
20+++ b/src/scheduler.h
21@@ -46,7 +46,7 @@ public:
22     /** Call f once after the delta has passed */
23     void scheduleFromNow(Function f, std::chrono::milliseconds delta)
24     {
25-        schedule(std::move(f), std::chrono::system_clock::now() + delta);
26+        // NOOP
27     }
28 
29     /**
30diff --git a/src/test/util/setup_common.cpp b/src/test/util/setup_common.cpp
31index 5334c4623e..57eeb91008 100644
32--- a/src/test/util/setup_common.cpp
33+++ b/src/test/util/setup_common.cpp
34@@ -138,7 +138,6 @@ ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::ve
35     // We have to run a scheduler thread to prevent ActivateBestChain
36     // from blocking due to queue overrun.
37     m_node.scheduler = std::make_unique<CScheduler>();
38-    m_node.scheduler->m_service_thread = std::thread(util::TraceThread, "scheduler", [&] { m_node.scheduler->serviceQueue(); });
39     GetMainSignals().RegisterBackgroundSignalScheduler(*m_node.scheduler);
40 
41     m_node.fee_estimator = std::make_unique<CBlockPolicyEstimator>();
42@@ -147,10 +146,7 @@ ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::ve
43     m_node.chainman = std::make_unique<ChainstateManager>();
44     m_node.chainman->m_blockman.m_block_tree_db = std::make_unique<CBlockTreeDB>(1 << 20, true);
45 
46-    // Start script-checking threads. Set g_parallel_script_checks to true so they are used.
47-    constexpr int script_check_threads = 2;
48-    StartScriptCheckWorkerThreads(script_check_threads);
49-    g_parallel_script_checks = true;
50+    g_parallel_script_checks = false;
51 }
52 
53 ChainTestingSetup::~ChainTestingSetup()
54diff --git a/src/util/thread.cpp b/src/util/thread.cpp
55index 14be668685..0c7dcec9b3 100644
56--- a/src/util/thread.cpp
57+++ b/src/util/thread.cpp
58@@ -12,6 +12,7 @@
59 
60 void util::TraceThread(const char* thread_name, std::function<void()> thread_func)
61 {
62+    assert(false);
63     util::ThreadRename(thread_name);
64     try {
65         LogPrintf("%s thread start\n", thread_name);

agroce commented at 5:49 pm on July 26, 2021: contributor

What do you think of simply automatically killing “old” subdirs of test_common_Bitcoin Core in process-based fuzzing (do nothing to libFuzzer target based fuzzing)?

maflcko commented at 7:00 am on July 27, 2021: member

separate-process-based fuzzing should be unaffected. See comments #22472 (comment) (no crash after 12 hours with my patch) and your screenshot in #22551 (also no crash?, disk not filled up?)

With fork-based fuzzing I am seeing a background noise of crashes (though unrelated to the disk space issue).

Preparing the affected fuzz targets for fork might also magically fix the other issues, I suspect.

maflcko commented at 8:54 am on July 27, 2021: member

cc @practicalswift Any ideas?

agroce commented at 1:36 pm on July 27, 2021: contributor

Sorry, I didn’t mean true separate process-based (which might also see this problem, depending on the cause). I meant anything that’s not doing libFuzzer “test is a function call” approach. My big afl run isn’t crashing because it’s compiled with my PR and the environment variable is set to TRUE. :)

For me, afl or Eclipser in docker ubuntu 20.04 both reliably end up filling /tmp and failing most tests due to that, without the patch, so I’m running with the patch.

practicalswift commented at 10:40 am on July 30, 2021: contributor

Concept ACK on working around this issue (if we cannot fix the root cause).

Instead of removing the entire tree /tmp/test_common_Bitcoin Core/ on exit would it be possible to remove only the specific sub-directory used during the fuzzing session (say /tmp/test_common_Bitcoin Core/6d8a[…]dc38/)?

agroce commented at 2:15 pm on July 30, 2021: contributor

The trick is it has to remove not on exit (since failing to do so as it should is the problem) but on entry, when it’s the “last” one it needs to kill, whose name is unknown.

practicalswift commented at 11:08 pm on July 30, 2021: contributor

The trick is it has to remove not on exit (since failing to do so as it should is the problem) but on entry, when it’s the “last” one it needs to kill, whose name is unknown.

Oh, of course! I misunderstood the problem. Sorry! :)

agroce commented at 11:13 pm on August 4, 2021: contributor

@MarcoFalke

Another idea:

If this isn’t causing much issue on OSS-Fuzz, perhaps a stopgap for Eclipser etc. fuzzing would be to remove code inside fuzz and add a script users can launch in background to occasionally clean up old leftover directories?

maflcko commented at 5:47 am on August 5, 2021: member

I’d still prefer to fix the underlying issue over a temporary workaround.

In #22472 (comment) I removed all threads. I wonder if I have to remove all locks too for fork to work properly in afl?

agroce commented at 6:23 am on August 5, 2021: contributor

I am concerned QEMU failures are going to always produce leftovers, for Eclipser…

Merge branch 'bitcoin:master' into master 971aab70b0

DrahtBot commented at 6:15 am on January 8, 2022: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#20744 (Use std::filesystem. Remove Boost Filesystem & System by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

DrahtBot added the label Needs rebase on Feb 3, 2022

DrahtBot commented at 4:11 pm on February 3, 2022: contributor

🐙 This pull request conflicts with the target branch and needs rebase.

Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a “draft”.

fanquake commented at 1:35 pm on April 26, 2022: member

What is the status of this? Needs to be reworked to remove boost::filesystem.

agroce commented at 4:44 pm on April 26, 2022: contributor

@MarcoFalke wanted a root cause fix; I haven’t been actively fuzzing bitcoin core lately, so am not sure if that ever happened? I assume not. I suspect for some fuzzers, it may be needed even with root-cause fixes, for some abnormal exits and long fuzzer runs.

maflcko commented at 8:40 am on April 29, 2022: member

Yeah, I’d prefer to fix the underlying issue. In the meantime, I think it is fine to provide a temporary opt-in workaround.

maflcko commented at 8:40 am on April 29, 2022: member

Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits

agroce commented at 5:16 pm on April 29, 2022: contributor

@MarcoFalke is it ready for final squash? Looks like boost::filesystem also needs to go?

maflcko commented at 5:45 pm on April 29, 2022: member

Yes, if you want to work on this again, it needs:

Rebase
Squash
boost::fs -> <fs.h>

fanquake added the label Up for grabs on Aug 15, 2022

fanquake closed this on Aug 15, 2022

bitcoin locked this on Aug 15, 2023

maflcko removed the label Up for grabs on Dec 8, 2023

fuzz: Add environment option to keep /tmp/ clean #22472

Conflicts