flatfile_tests fail on alpine linux

MarcoFalke commented at 1:59 am on March 20, 2019: member

File size checks fails and I haven’t figured out why, but next I’d look into how posix_fallocate works on alpine linux.

Log:

 0$ cat /etc/alpine-release 
 13.9.2
 2
 3$ git log --oneline -1
 4e45b7f20e Merge [#15618](/bitcoin-bitcoin/15618/): refactor: Remove unused function
 5
 6$ ./src/test/test_bitcoin -t flatfile_tests
 7Running 4 test cases...
 8Version using posix_fallocate
 9test/flatfile_tests.cpp(94): error: in "flatfile_tests/flatfile_allocate": check fs::file_size(seq.FileName(FlatFilePos(0, 0))) == 100 has failed [0 != 100]
10test/flatfile_tests.cpp(98): error: in "flatfile_tests/flatfile_allocate": check fs::file_size(seq.FileName(FlatFilePos(0, 99))) == 100 has failed [0 != 100]
11Version using posix_fallocate
12test/flatfile_tests.cpp(102): error: in "flatfile_tests/flatfile_allocate": check fs::file_size(seq.FileName(FlatFilePos(0, 99))) == 200 has failed [0 != 200]
13Version using posix_fallocate
14test/flatfile_tests.cpp(116): error: in "flatfile_tests/flatfile_flush": check fs::file_size(seq.FileName(FlatFilePos(0, 1))) == 100 has failed [0 != 100]
15
16*** 4 failures are detected in the test module "Bitcoin Core Test Suite"

MarcoFalke added the label Linux/Unix on Mar 20, 2019

MarcoFalke added the label Tests on Mar 20, 2019

MarcoFalke added the label Up for grabs on Mar 20, 2019

MarcoFalke added the label good first issue on Mar 21, 2019

lucayepa commented at 2:33 pm on March 22, 2019: contributor

I’ve tried to replicate it, but it seems to work.

Log:

 0$ cat /etc/alpine-release 
 13.9.2
 2
 3$ git log --oneline -1
 4e45b7f20e (HEAD) Merge [#15618](/bitcoin-bitcoin/15618/): refactor: Remove unused function
 5
 6$ ./src/test/test_bitcoin -t flatfile_tests
 7Running 4 test cases...
 8
 9*** No errors detected
10
11$ uname -srv
12Linux 4.19.26-0-virt [#1](/bitcoin-bitcoin/1/)-Alpine SMP Thu Feb 28 21:29:54 UTC 2019

MarcoFalke commented at 3:33 pm on March 22, 2019: member

Hmm, I was running alpine in podman.

0$ uname -srv
1Linux 4.20.14-200.fc29.x86_64 [#1](/bitcoin-bitcoin/1/) SMP Tue Mar 5 19:55:32 UTC 2019

lucayepa commented at 9:47 pm on March 22, 2019: contributor

Maybe it is related to filesystem. Are you using unionfs or aufs? They don’t support posix_fallocate. What do you get, if you try from command line:

0sudo apk add util-linux
1fallocate -l 10M filename

MarcoFalke commented at 0:18 am on March 23, 2019: member

Indeed

0fallocate -l 10M filename
1fallocate: fallocate failed: Not supported

lucayepa commented at 2:32 am on March 23, 2019: contributor

This is still strange, because, when there is no posix_fallocate, there is an emulation in glibc. Maybe some versions of glibc do not provide this emulation.

If the underlying filesystem does not support fallocate(2), then the operation is emulated with the following caveats:

The emulation is inefficient.

There is a race condition where concurrent writes from another thread or process could be overwritten with null bytes.

There is a race condition where concurrent file size increases by another thread or process could result in a file whose size is smaller than expected.

If fd has been opened with the O_APPEND or O_WRONLY flags, the function will fail with the error EBADF. In general, the emulation is not MT-safe. On Linux, applications may use fallocate(2) if they cannot tolerate the emulation caveats. In general, this is only recommended if the application plans to terminate the operation if EOPNOTSUPP is returned, otherwise the application itself will need to implement a fallback with all the same problems as the emulation provided by glibc.

Since we open the files rb+, it should be fine.

lucayepa commented at 4:28 am on March 23, 2019: contributor

We can see the error on strace:

0$ strace test/test_bitcoin -t flatfile_tests 2>&1 | grep fallocate
1fallocate(3, 0, 0, 10000)               = -1 EOPNOTSUPP (Operation not supported)

MarcoFalke closed this on May 2, 2019

MarcoFalke referenced this in commit c4560a7dfe on May 2, 2019

sidhujag referenced this in commit b5fe08a951 on May 2, 2019

fanquake removed the label Up for grabs on May 8, 2019

PastaPastaPasta referenced this in commit b504998b77 on Jun 27, 2021

PastaPastaPasta referenced this in commit 7d35d71797 on Jun 28, 2021

PastaPastaPasta referenced this in commit a3d5e832a4 on Jun 29, 2021

PastaPastaPasta referenced this in commit c29a551053 on Jul 1, 2021

PastaPastaPasta referenced this in commit 02534d43d5 on Jul 1, 2021

PastaPastaPasta referenced this in commit 43a027d45e on Jul 8, 2021

PastaPastaPasta referenced this in commit 0199129fdc on Jul 10, 2021

MarcoFalke locked this on Dec 16, 2021

flatfile_tests fail on alpine linux #15624