contrib: Automation for Bitcoin Full Node Deployment

0xawaz commented at 4:19 pm on August 12, 2024: none

Please describe the feature you’d like to see added.

I followed the Bitcoin Full Node installation guide. While the guide is thorough, it lacks automation, which can lead to human errors and platform compatibility issues during manual deployment.

I propose adding an automation contrib for seamless deployment and maintenance of a Bitcoin full node. This will help eliminate manual errors, improve consistency across environments, simplify the overall process, and reduce the time and effort required for setup and maintenance.

Yes, this issue is related to the problem of manual setups and their failure risks. Manual setup is useful to understand the step-by-step setup process, but it is no longer reliable in production environments.

Describe the solution you’d like

Docker: Docker image for Bitcoin Core binaries. I started here. I would love feedback before converting it to a PR.
Ansible Playbooks: Ansible playbooks to setup instance, install dependencies, build docker image and run bitcoin core container. We want to support bare metal and cloud instances.
Monitoring System: Prometheus/Grafana monitoring system to monitor Bitcoin Node metrics and health.

Describe any alternatives you’ve considered

I automated the setup for myself, as the manual setup wasn’t convenient. I want to share it with the bitcoin community.

Please leave any additional context

I read contribution policy. I am not sure if this is the right repo for this contribution, if so, I am happy to start creating related PRs, if not, please direct me where I can share it with bitcoin community :)

0xawaz added the label Feature on Aug 12, 2024

kristapsk commented at 4:39 pm on August 12, 2024: contributor

3. Monitoring System: Prometheus/Grafana monitoring system to monitor Bitcoin Node metrics and health.

For monitoring I have made some scripts and templates for Zabbix monitoring system. https://github.com/zkSNACKs/zbx-bitcoin

maflcko commented at 4:54 pm on August 12, 2024: member

See https://github.com/bitcoin-core/packaging/issues/55 and #25681 about the previous docker discussions (there are more than that).

About the Ansible/monitoring: I’d doubt that the majority of users would want that. The ones that want it, will likely go with their in-house version anyway, so it seems unclear what the target audience would be. (The same applies to docker to a lesser extent)

maflcko added the label Brainstorming on Aug 12, 2024

maflcko added the label Scripts and tools on Aug 12, 2024

0xawaz commented at 5:55 pm on August 12, 2024: none

Thanks @maflcko for pointing out docker discussions I missed. I took some time to walk through all the discussions. It just got me more confused. I am genuinely curious to understand what’s happening. So I am asking few questions:

Docker
- Since we all agree that we need an official docker image for Bitcoin Core, what all PR attempts fail to consider?
- My understanding is Dockerfile maintenance is one of the main concerns, could you elaborate more? Do you need more hands in the team or do you need us to adapt to the Bitcoin Core build environment and CI/CD process?
- To be able to succeed where previous attempts failed, I will need to dig more in the code, CI/CD workflows, and tricky blind spots I need to consider … Will you be willing to provide me with more information about your concerns and experience with Dockerfile attempts integrations? (I would suggest a more elaborate and patient conversation, to be able to cover all that we need, let’s avoid prematurely aborting this conversation, communication can be tricky for some of us :D )
Ansible/Monitoring: I agree, tooling preferences/ideologies are a thing now, I am agnostic, I adapt, my questions:
- What tooling do you, Bitcoin core devs, and community support for system provisioning?
- What tooling for monitoring?

maflcko commented at 6:56 pm on August 12, 2024: member

Ansible/Monitoring: I agree, tooling preferences/ideologies are a thing now, I am agnostic, I adapt, my questions:
* What tooling do you, Bitcoin core devs, and community support for system provisioning?

* What tooling for monitoring?

Bitcoin Core is also agnostic, because it should work with any monitoring or provisioning solution out there out of the box with no further changes needed. (There are many out there, including graphana, or self-written ones)

The same holds for the dockerfiles. Since you followed the discussion I linked to, you’ve also seen the list of already existing (community) dockerfiles (at the time when I created the list).

I followed the Bitcoin Full Node installation guide. While the guide is thorough, it lacks automation, which can lead to human errors and platform compatibility issues during manual deployment.

I’d say the goal of the guide is to provide background information, not to offer an automation script. Obviously the guide is outdated and should be updated, however the bitcoin.org website is another project that is maintained elsewhere. (This project here is about the Bitcoin Core software).

0xawaz commented at 9:10 am on August 13, 2024: none

Ansible/Monitoring: I agree, tooling preferences/ideologies are a thing now, I am agnostic, I adapt, my questions:
0* What tooling do you, Bitcoin core devs, and community support for system provisioning?
1
2* What tooling for monitoring?
Bitcoin Core is also agnostic, because it should work with any monitoring or provisioning solution out there out of the box with no further changes needed. (There are many out there, including graphana, or self-written ones)

The same holds for the dockerfiles. Since you followed the discussion I linked to, you’ve also seen the list of already existing (community) dockerfiles (at the time when I created the list).

I followed the Bitcoin Full Node installation guide. While the guide is thorough, it lacks automation, which can lead to human errors and platform compatibility issues during manual deployment.

I’d say the goal of the guide is to provide background information, not to offer an automation script. Obviously the guide is outdated and should be updated, however the bitcoin.org website is another project that is maintained elsewhere. (This project here is about the Bitcoin Core software).

Great, if I understand correctly, the automation part and monitoring is out of scope in Bitcoin Core Software. You want to stay agnostic which is relevant. so, for that part, I can just suggest a solution in a separate repo and maintain it myself.

0xawaz commented at 9:13 am on August 13, 2024: none

Docker

Since we all agree that we need an official docker image for Bitcoin Core, what all PR attempts fail to consider?

My understanding is Dockerfile maintenance is one of the main concerns, could you elaborate more? Do you need more hands in the team or do you need us to adapt to the Bitcoin Core build environment and CI/CD process?

To be able to succeed where previous attempts failed, I will need to dig more in the code, CI/CD workflows, and tricky blind spots I need to consider … Will you be willing to provide me with more information about your concerns and experience with Dockerfile attempts integrations? (I would suggest a more elaborate and patient conversation, to be able to cover all that we need, let’s avoid prematurely aborting this conversation, communication can be tricky for some of us :D )

Regarding Dockerfile, I believe it can be considered part of Bitcoin Core Software, as a Bitcoin Core user I need it and I can see through the discussions that many need it, could someone enlighten me of what is needed to start accepting Dockerfile as part of Bitcoin Core Software? What are your requirements?

maflcko commented at 8:26 am on August 14, 2024: member

Regarding Dockerfile, I believe it can be considered part of Bitcoin Core Software, as a Bitcoin Core user I need it and I can see through the discussions that many need it, could someone enlighten me of what is needed to start accepting Dockerfile as part of Bitcoin Core Software? What are your requirements?

No one is holding anyone back from opening a pull request to add a container engine imagefile. Not sure how much actual interest is there, given that no one has done so far, or the previous attempts were seemingly abandoned?

In any case, I think everything has been said in all previous discussions of this topic over the past decade, so I don’t think it makes sense to re-hash everything every two years. I’d doubt that there are any points that have not been previously discussed, but if there are, feel free to ask.

0xawaz commented at 10:59 am on August 14, 2024: none

Great, let me review the discussions again to clear my confusion and find answers to my questions. Lazy me, I was hoping for a highlight of major requirements that weren’t met before, I guess I should do that highlight myself 😄

willcl-ark commented at 11:22 am on August 14, 2024: member

FWIW I am maintaing some (very unofficial, but up-to-date!) docker images using this repo: https://github.com/willcl-ark/bitcoin-core-docker, just to add to those already mentioned in the discussion over at https://github.com/bitcoin-core/packaging/issues/55

It has currently-supported tags, as well as an automatic nightly master build.

IMO this issue can probably be closed in this repo. I don’t think there is much appetite to maintain official docker images in this project and I think an issue in https://github.com/bitcoin-core/packaging might be more appropriate?

maflcko commented at 1:09 pm on August 14, 2024: member

@willcl-ark Maybe leave a new comment in the packaging issue 55 about your dockerfile?

I agree closing this for now. It may be best to continue discussion in https://github.com/bitcoin-core/packaging/issues/55 (yes it is closed, but discussion can continue).

And given that the discussion here seems about a user dockerfile, not a dev dockerfile, the packaging repo seems more appropriate.

maflcko closed this on Aug 14, 2024

contrib: Automation for Bitcoin Full Node Deployment #30638

Please describe the feature you’d like to see added.

Is your feature related to a problem, if so please describe it.

Describe the solution you’d like

Describe any alternatives you’ve considered

Please leave any additional context