Adding stop commands to bitcoind.service

McRemco commented at 0:02 am on July 22, 2018: none

When bitcoind (version 0.16.1) is started via systemd with the supplied bitcoind.service file stopping the service results in an unclean shutdown if for some reason it takes too long to write the cache to hdd and subsequently will cause a rolling forward when it is started the next time.

So maybe it is a good idea to add something like this to the bitcoind.service file

0# Commands to ensure a clean shutdown when stopping bitcoind
1ExecStop=bitcoin-cli -conf=/etc/bitcoin/bitcoin.conf stop
2TimeoutStopSec=600
3# don't send kill signal
4KillMode=none

luke-jr commented at 1:13 am on July 22, 2018: member

RPC stop shouldn’t be any faster than signal-based shutdown… I’d just do the timeout if needed.

McRemco commented at 4:42 pm on July 22, 2018: none

So if I understand it correct the kill signal grom systemd invokes the same routines the rpc command stop does, I was under the impression that that the kill signal was much more abrupt, hence the name.

sipa commented at 5:27 pm on July 22, 2018: member

@McRemco I’m unfamiliar with systemd, but typically such signal-based stopping first sends a SIGTERM, and only after a timeout sends a SIGKILL. SIGKILL cannot be intercepted and will immediately terminate the process, but SIGTERM can. SIGTERM invokes the same logic internally as the stop RPC command.

McRemco commented at 9:32 pm on July 22, 2018: none

@sipa, Thank you for clearing that out, I’m stil fairly new in the world of Linux but learning every day a little more.

greenaddress commented at 10:58 am on July 25, 2018: contributor

Perhaps is no longer the case but bitcoind in the past had instances where it was slow to shutdown, in the past using SIGTERM i remember i lost mempool/feeestimates and moved to call stop via rpc/cli to avoid the SIGKILL post SIGTERM timeout.

jonasschnelli commented at 3:45 pm on July 30, 2018: contributor

The shutdown can be “fast or slow” via RPC stop and via SIGTERM. It depends on caches being written and other tasks begin uncompleted. Example: if you sync the complete chain from the scratch with a dbcache of 6GB the shutdown (regardless wether you have use SIGTERM or RPC stop) will take a fairly long time since the the complete cache gets written during the shutdown procedure.

It would be possible to furthermore optimise for a quicker shutdown. See #12980 as an example.

jonasschnelli commented at 3:49 pm on July 30, 2018: contributor

It would probably be worth to figure out, what maximal execution time a process is given via systemd (or other application managers) before a SIGKILL happens. But in generally it might be complicated to never exceed this maximal timeout since it depends on disk speed, CPU, etc… Ideally, every time consuming process would respect the shutdown trigger (if possible).

setpill commented at 9:46 am on August 8, 2019: contributor

Simple fix would be to set TimeoutStopSec=infinity in the bitcoind.service file. Imho that has preference over setting a long timeout with KillMode=none since this can break systemd’s restart logic.

Another idea by @Sjors as discussed on IRC would be to switch from Type=forking to Type=notify which would allow bitcoind to request timeout extensions via EXTEND_TIMEOUT_USEC=; however this would involve (non-trivial) changes to the codebase to use sd_notify.

andrewtoth commented at 7:47 pm on August 8, 2019: contributor

#15218 would mitigate this issue.

setpill commented at 10:59 pm on August 8, 2019: contributor

I don’t believe it would; perhaps in specific cases but not all, if I understand the functioning correctly. It assumes an uninterrupted IBD.

andrewtoth commented at 11:06 pm on August 8, 2019: contributor

It would solve for the first 24 hours after IBD is finished. In cases where IBD doesn’t finish, then you are correct.

setpill commented at 11:00 am on August 9, 2019: contributor

Especially in the case where one tinkers with the config during IBD (e.g. to increase db cache in order to speed up IBD) and uses systemctl restart bitcoind, the current systemd service file causes unclean shutdowns, triggering a lengthy “Rolling forward” after the restart of the service (wasting much work that has been done and increasing how painful the IBD experience is). Imho both PRs do not conflict and complement each other :)

laanwj referenced this in commit 99cebc922c on Oct 8, 2019

sidhujag referenced this in commit a46e4acabc on Oct 8, 2019

adamjonas commented at 4:59 pm on May 6, 2020: member

Closed by #16569.

MarcoFalke closed this on May 8, 2020

MarcoFalke locked this on Feb 15, 2022

Adding stop commands to bitcoind.service #13736