rpc: allow writing UTXO set to a named pipe

theStack commented at 2:42 am on December 24, 2024: contributor

This PR slightly modifies the dumptxoutset RPC to allow writing the UTXO set dump into a named pipe, so that the output data can be consumed by another process, see #31373. Taking use of this with the utxo-to-sqlite.py tool (introduced in #27432), creating an UTXO set in SQLite3 format is possible on the fly. E.g. for signet:

 0$ mkfifo /tmp/utxo_fifo && ./build/bin/bitcoin-cli -signet dumptxoutset /tmp/utxo_fifo latest &
 1$ ./contrib/utxo-tools/utxo_to_sqlite.py /tmp/utxo_fifo ./utxo.sqlite
 2UTXO Snapshot for Signet at block hash 000000012711f0a4e741be4a22792982..., contains 61848352 coins
 31048576 coins converted [1.70%], 2.800s passed since start
 4....
 5....
 660817408 coins converted [98.33%], 159.598s passed since start
 7{
 8  "coins_written": 61848352,
 9  "base_hash": "000000012711f0a4e741be4a22792982370f51326db20fca955c7d45da97f768",
10  "base_height": 294305,
11  "path": "/tmp/utxo_fifo",
12  "txoutset_hash": "34ae7fe7af33f58d4b83e00ecfc3b9605d927f154e7a94401226922f8e3f534e",
13  "nchaintx": 28760852
14}
15TOTAL: 61848352 coins written to ./utxo.sqlite, snapshot height is 294305.

Note that the dumptxoutset RPC calculates an UTXO set hash as a first step before any data is emitted, so especially on mainnet it takes quite a while until the conversion starts and something is happening visibly.

DrahtBot commented at 2:42 am on December 24, 2024: contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31560.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	sedited, ajtowns
Concept ACK	naiyoma
Stale ACK	tdb3, virtu, rkrux, fjahr

If your review is incorrectly listed, please copy-paste <!–meta-tag:bot-skip–> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#34324 (Add CSV support to utxo_convert.py (formerly utxo_to_sqlite.py) by sipa)
#33477 (Rollback for dumptxoutset without invalidating blocks by fjahr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

“If a named pipe passed, write directly to it.” -> “If a named pipe is passed, write directly to it.” [missing verb “is” makes the conditional clause ungrammatical and could confuse readers]

2026-03-11 16:23:37

DrahtBot added the label RPC/REST/ZMQ on Dec 24, 2024

theStack force-pushed on Dec 24, 2024

DrahtBot added the label CI failed on Dec 24, 2024

DrahtBot commented at 2:52 am on December 24, 2024: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/34820327020

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

theStack force-pushed on Dec 24, 2024

DrahtBot removed the label CI failed on Dec 24, 2024

in contrib/README.md:54 in 95c861f368 outdated

49+
50+### [UTXO-to-SQLite](/contrib/utxo-tools/utxo_to_sqlite.py) ###
51+This script converts a compact-serialized UTXO set (as generated by Bitcoin Core with `dumptxoutset`)
52+to a SQLite3 database. The coins are stored in a table with the following schema:
53+```
54+CREATE TABLE utxos(txid TEXT, vout INT, value INT, coinbase INT, height INT, scriptpubkey TEXT)

tdb3 commented at 3:00 pm on December 24, 2024:

nit: To prevent maintaining the same info (table structure) in multiple files, maybe we can create a link here to the opening comment in utxo_to_sqlite.py (describing the table)?

theStack commented at 2:09 am on December 28, 2024:

Good idea, done in #27432 (referring now to the module docstring, which is also visible as --help output of the script).

in test/functional/tool_utxo_to_sqlite.py:124 in b5ca41cb06 outdated

131+            os.mkfifo(fifo_filename)
132+            output_direct_filename = os.path.join(self.options.tmpdir, "utxos_direct.sqlite")
133+            p = subprocess.Popen([sys.executable, utxo_to_sqlite_path, fifo_filename, output_direct_filename],
134+                                 stderr=subprocess.STDOUT)
135+            node.dumptxoutset(fifo_filename, "latest")
136+            p.wait()

tdb3 commented at 4:28 pm on December 24, 2024:

Rather than have this wait indefinitely, might be better to specify a timeout (e.g. CONVERSION_TIMEOUT = 60, p.wait(timeout=CONVERSION_TIMEOUT)). This could allow earlier failure detection (i.e. instead of relying on the longer CI timeout).

theStack commented at 2:10 am on December 28, 2024:

Added a fixed timeout of 10 seconds, which should be more than enough given the tiny regtest UTXO set. I felt that it’s not worth it to introduce a constant for that, but happy to add if others feel strongly.

in test/functional/tool_utxo_to_sqlite.py:112 in b5ca41cb06 outdated

119-            utxo_ser += CTxOut(value, bytes.fromhex(spk_hex)).serialize()
120-            muhash.insert(utxo_ser)
121-        con.close()
122-
123-        muhash_sqlite = muhash.digest()[::-1].hex()
124+        muhash_sqlite = calculate_muhash_from_sqlite_utxos(output_filename)

tdb3 commented at 4:30 pm on December 24, 2024:

Might be less churn to have commit d9a8a137b64f586422455318a9e757b1967a4f73 introduce this function instead of refactoring in this commit.

theStack commented at 2:10 am on December 28, 2024:

Great idea, done in #27432.

in test/functional/tool_utxo_to_sqlite.py:2 in d9a8a137b6 outdated

0@@ -0,0 +1,114 @@
1+#!/usr/bin/env python3
2+# Copyright (c) 2023 The Bitcoin Core developers

tdb3 commented at 4:34 pm on December 24, 2024:

2024-present

in contrib/utxo-tools/dump_to_sqlite.sh:2 in 43fff2e9da outdated

0@@ -0,0 +1,32 @@
1+#!/usr/bin/env bash
2+# Copyright (c) 2024 The Bitcoin Core developers

tdb3 commented at 4:36 pm on December 24, 2024:

2024-present

tdb3 commented at 5:43 pm on December 24, 2024: contributor

Approach ACK

Great feature!

Did some manual santiy testing on mainnet:

Used dumptxoutset to create a dump (with a node synced to block 876,186), utxo_to_sqlite.py to covert to a sqlite file, and opened/parsed in python. Conversion seemed successful, the correct number of coins were present in the table
Used dump_to_sqlite.sh to do the same with fifo (but with a node synced to block 200,000), then open/parsed in python. Conversion seemed successful, the correct number of coins were present in the table

Left a few relatively small comments. May circle back and review utxo_to_sqlite.py in more detail as time allows.

in test/functional/tool_utxo_to_sqlite.py:8 in 43fff2e9da outdated

0@@ -0,0 +1,131 @@
1+#!/usr/bin/env python3
2+# Copyright (c) 2023 The Bitcoin Core developers
3+# Distributed under the MIT software license, see the accompanying
4+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
5+"""Test utxo-to-sqlite conversion tool"""
6+import os
7+try:
8+    import sqlite3

romanz commented at 12:08 pm on December 25, 2024:

Is the import expected to fail on CI? If so, I am not sure that the test below can run successfully if sqlite3 is not available…

theStack commented at 2:14 am on December 28, 2024:

The idea is to skip a test rather than fail, if the sqlite3 module is not available (as far as I’m aware, the only supported distro where this could happen currently is FreeBSD). However, this indeed didn’t work as expected since I was using skip_if_no_sqlite rather than skip_if_no_py_sqlite3 (see #26882). Fixed in #27432.

theStack force-pushed on Dec 28, 2024

theStack commented at 2:24 am on December 28, 2024: contributor

@tdb3 @romanz: Thanks for your reviews, much appreciated! Note that the first two commits which introduce the utxo-to-sqlite.py tool (+test) are part of the base PR #27432, so further comments on those changes would better fit there in the future. I took all of your suggestions and updated #27432 and rebased this PR on top of that again accordingly.

tdb3 approved

tdb3 commented at 4:53 pm on December 28, 2024: contributor

ACK 59df8480be77a8e3618c9422536c4f8aed82467e

Also re-ran tests in #31560#pullrequestreview-2522042088

luke-jr commented at 5:59 am on January 7, 2025: member

Can we make the exists/is_fifo/open atomic somehow? Seems liable to have a race here someday…

theuni commented at 5:56 pm on January 8, 2025: member

Not opposed to this, but it seems like a good use-case for a kernel util :) @thecharlatan @stickies-v @josibake

luke-jr referenced this in commit 4c23b86973 on Jan 15, 2025

luke-jr commented at 4:45 pm on January 15, 2025: member

eg https://github.com/luke-jr/bitcoin/commit/56ee485533820d8c7f0e4e27acd712aa8b411286

theStack force-pushed on Jan 18, 2025

DrahtBot added the label CI failed on Jan 18, 2025

DrahtBot commented at 4:34 am on January 18, 2025: contributor

🚧 At least one of the CI tasks failed. Debug: https://github.com/bitcoin/bitcoin/runs/35809942510

Try to run the tests locally, according to the documentation. However, a CI failure may still happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being incompatible with the current code in the target branch). If so, make sure to rebase on the latest commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

theStack commented at 4:40 am on January 18, 2025: contributor

Can we make the exists/is_fifo/open atomic somehow? Seems liable to have a race here someday…

eg luke-jr@56ee485 @luke-jr: Thanks, applied this to https://github.com/bitcoin/bitcoin/pull/31560/commits/61cef65ed1bf43d7d48ed8257441e697d6c14171. Note that I had to introduce a fs::exists helper for file_status, as direct usage of std::filesystem::exists was prohibited by the linter (which failed with “Direct use of std::filesystem may be dangerous and buggy. Please include <util/fs.h> and use the fs:: namespace, which has unsafe filesystem functions marked as deleted.”).

DrahtBot removed the label CI failed on Jan 18, 2025

theStack force-pushed on Feb 15, 2025

theStack commented at 9:46 am on February 15, 2025: contributor

Rebased on master (now that #27432 has been merged 🎉).

DrahtBot added the label CI failed on Feb 15, 2025

DrahtBot removed the label CI failed on Feb 15, 2025

luke-jr referenced this in commit 7d3cb0e0fc on Feb 22, 2025

virtu commented at 2:39 pm on March 21, 2025: contributor

ACK 4c8e9b4f35be4104002ece15b6f72c360756f5d9

Tested on Linux and MacOS.

Nice work. I’m looking to integrating some UTXO sats on my website, so it’s nice to see people working on conveniently extracting this data.

DrahtBot requested review from tdb3 on Mar 21, 2025

yancyribbens commented at 7:01 pm on March 22, 2025: contributor

Interesting - I read the link above “name pipe” and it seems the main benefit to a named pipes is just that for performance, one can skip writing to disk and the churn of creating a new file. I’m not sure I see the perf benefit here in a named pipe though..

yancyribbens commented at 7:04 pm on March 22, 2025: contributor

I guess a follow comment is how big is the dump. If it’s very large then that would be a good motivation for a named pipe.

DrahtBot added the label CI failed on May 16, 2025

DrahtBot removed the label CI failed on May 17, 2025

luke-jr referenced this in commit 0393dc4a3e on Jun 6, 2025

luke-jr referenced this in commit 6d7ea40881 on Jun 6, 2025

DrahtBot added the label Needs rebase on Jul 3, 2025

theStack force-pushed on Jul 4, 2025

DrahtBot removed the label Needs rebase on Jul 4, 2025

theStack commented at 4:15 pm on July 4, 2025: contributor

Rebased on master (due to conflict with #29307, commit a69c4098b273b6db5d2212ba91cfc713c1634c5d). @yancyribbens: Sorry, I must have missed your comments back then.

Interesting - I read the link above “name pipe” and it seems the main benefit to a named pipes is just that for performance, one can skip writing to disk and the churn of creating a new file. I’m not sure I see the perf benefit here in a named pipe though..

Yes, feeding the data directly into the consuming process is obviously faster than first writing the whole dump to disk (which also takes additional temporary space) and then read it all from there again. Avoiding disk I/O would be possible in theory with any other IPC mechanism too, but named pipes have the advantage that the changes needed to the producer and consumer applications (in this case, bitcoind and utxo_to_sqlite.py) are minimal, as named pipes mostly behave like regular files.

I guess a follow comment is how big is the dump. If it’s very large then that would be a good motivation for a named pipe.

As of today on mainnet, at block height 903978, the UTXO set dump file has a size of ~8.95 GiB. (As a side note, it seems like the UTXO set size has decreased lately, I remember the dump being over 10 GiB already a few months ago.)

luke-jr referenced this in commit 565039aed0 on Jul 17, 2025

luke-jr referenced this in commit 2bb4447c7c on Jul 17, 2025

naiyoma commented at 6:50 pm on August 30, 2025: contributor

Concept ACK

briefly tested on regtest:

 0
 1./contrib/utxo-tools/dump_to_sqlite.sh "./build/src/bitcoin-cli -regtest" ~/utxos_03_regtest.sqlite3
 2UTXO Snapshot for Regtest at block hash 1859fa04bf13269212d2cbcc406c91b6..., contains 1358 coins
 3{
 4  "coins_written": 1358,
 5  "base_hash": "1859fa04bf13269212d2cbcc406c91b6d11decbc39b97de79ae75566f5b2e826",
 6  "base_height": 1338,
 7  "path": "/tmp/tmp.NbBipQ3mhb/utxos.fifo",
 8  "txoutset_hash": "3bdfcb5a529096de8e73b516005d4263b7924a6c5f81e0823e9ec904126bbf4d",
 9  "nchaintx": 1364
10}
11TOTAL: 1358 coins written to /home/ubuntu/utxos_03_regtest.sqlite3, snapshot height is 1338.

Verified UTXO count matches:

0sqlite3 ~/utxos_03_regtest.sqlite3 "SELECT COUNT(*) FROM utxos;"
11358

DrahtBot added the label Needs rebase on Oct 24, 2025

theStack force-pushed on Oct 24, 2025

theStack commented at 8:24 pm on October 24, 2025: contributor

Rebased on master (due to conflict with recently merged #32983, commit b63428ac9ce2c903670409b3e47b9f6730917ae8).

DrahtBot removed the label Needs rebase on Oct 24, 2025

in contrib/utxo-tools/dump_to_sqlite.sh:1 in ed02f67c58 outdated

rkrux commented at 11:30 am on November 5, 2025:

In ed02f67c5831d94720e55127216aa0d97f55e72a “contrib: add script dump_to_sqlite.sh for direct SQLite3 UTXO dump”

Nit: Can call the script dump_utxo_to_sqlite.sh for clarity even though it resides in utxo-tools/.

rkrux approved

rkrux commented at 11:32 am on November 5, 2025: contributor

lgtm ACK ed02f67c5831d94720e55127216aa0d97f55e72a

DrahtBot requested review from naiyoma on Nov 5, 2025

DrahtBot added the label Needs rebase on Feb 8, 2026

theStack force-pushed on Feb 9, 2026

DrahtBot removed the label Needs rebase on Feb 9, 2026

sedited requested review from rkrux on Mar 4, 2026

sedited approved

sedited commented at 8:29 am on March 4, 2026: contributor

ACK 32ecef55a479c98f252b9213732d67d4c5ef8b67

Edit: I’m not fully sold on the shell script. If you interrupt it, you have to manually do some cleanup, otherwise bitcoind will not shutdown. Maybe the last commit can just be dropped, seems easy enough to just start bitcoin-cli and the sqlite conversion script in a separate shell.

theStack force-pushed on Mar 4, 2026

theStack commented at 5:19 pm on March 4, 2026: contributor

Edit: I’m not fully sold on the shell script. If you interrupt it, you have to manually do some cleanup, otherwise bitcoind will not shutdown. Maybe the last commit can just be dropped, …

Sure, I’m happy to drop the shell script, since it’s more of a proof of concept and was put together rather quickly. If there’s demand for it, we can always introduce it in a future PR with proper error handling and cleanup.

…seems easy enough to just start bitcoin-cli and the sqlite conversion script in a separate shell.

In addition to those two steps, a named pipe has also be created first (mkfifo ...), but I agree this should still be simple enough.

sedited approved

sedited commented at 5:32 pm on March 4, 2026: contributor

Re-ACK 509d871fc00e3dd5a9c42e267da13a322b646d76

DrahtBot added the label CI failed on Mar 4, 2026

in src/rpc/blockchain.cpp:3088 in 509d871fc0 outdated

3085-    const fs::path temppath = path + ".incomplete";
3086+    const fs::path temppath = fs::is_fifo(path_info) ? // If a named pipe is passed, write directly to it
3087+        path : path + ".incomplete";
3088 
3089-    if (fs::exists(path)) {
3090+    if (fs::exists(path_info) && !fs::is_fifo(path_info)) {

rkrux commented at 10:00 am on March 5, 2026:

I don’t get how using status is particularly beneficial in avoiding race condition here. Even if it’s used, decreasing the number of operations between time to check and time to use can help because temppath is not needed until after the exists check.

 0diff --git a/src/rpc/blockchain.cpp b/src/rpc/blockchain.cpp
 1index 3e55ecab3a..c3a09ad40a 100644
 2--- a/src/rpc/blockchain.cpp
 3+++ b/src/rpc/blockchain.cpp
 4@@ -3080,11 +3080,6 @@ static RPCHelpMan dumptxoutset()
 5     const ArgsManager& args{EnsureAnyArgsman(request.context)};
 6     const fs::path path = fsbridge::AbsPathJoin(args.GetDataDirNet(), fs::u8path(self.Arg<std::string_view>("path")));
 7     const auto path_info{fs::status(path)};
 8-    // Write to a temporary path and then move into `path` on completion
 9-    // to avoid confusion due to an interruption.
10-    const fs::path temppath = fs::is_fifo(path_info) ? // If a named pipe is passed, write directly to it
11-        path : path + ".incomplete";
12-
13     if (fs::exists(path_info) && !fs::is_fifo(path_info)) {
14         throw JSONRPCError(
15             RPC_INVALID_PARAMETER,
16@@ -3092,6 +3087,11 @@ static RPCHelpMan dumptxoutset()
17             "move it out of the way first");
18     }
19 
20+    // Write to a temporary path and then move into `path` on completion
21+    // to avoid confusion due to an interruption.
22+    const fs::path temppath = fs::is_fifo(path_info) ? // If a named pipe is passed, write directly to it
23+        path : path + ".incomplete";
24+
25     FILE* file{fsbridge::fopen(temppath, "wb")};
26     AutoFile afile{file};

ajtowns commented at 3:05 pm on March 5, 2026:

The file’s information is grabbed at path_info{fs::status(path)} time, not when fs::is_fifo(path_info) is queried; I don’t think there’s any meaningful race condition here.

theStack commented at 6:43 pm on March 5, 2026:

I don’t get how using status is particularly beneficial in avoiding race condition here.

It’s been a while since I addressed this (see #31560 (comment) ff.), but I think the point was to only read the status once (at path_info{fs::status(path)}, as @ajtowns noted) and make all further decisions based on that, rather than access the file status repeatedly. Will leave as-is, unless you or other reviewers feel strongly about it.

rkrux commented at 10:07 am on March 6, 2026:

Yes, I had read that the status reads only once and helps in reducing syscalls. I was trying to understand how it helps in avoiding any race as that was mentioned in #31560 (comment).

The suggestion I shared seemed like a low hanging fruit and can be done if you retouch.

theStack commented at 4:20 pm on March 11, 2026:

The suggestion I shared seemed like a low hanging fruit and can be done if you retouch.

Left as-is to keep the diff minimal and not mix it with refactors. It’s true that the temppath variable is introduced a bit earlier than necessary, but I think it makes sense to have the accompanying comment (regular file vs. named pipe behaviour explainer) before the fs::exists check. Happy to review a refactoring follow-up.

rkrux commented at 10:00 am on March 5, 2026: contributor

Raised one point, rest looks fine.

DrahtBot requested review from rkrux on Mar 5, 2026

DrahtBot removed the label CI failed on Mar 5, 2026

in src/rpc/blockchain.cpp:3086 in 509d871fc0

3078@@ -3109,11 +3079,13 @@ static RPCHelpMan dumptxoutset()
3079 
3080     const ArgsManager& args{EnsureAnyArgsman(request.context)};
3081     const fs::path path = fsbridge::AbsPathJoin(args.GetDataDirNet(), fs::u8path(self.Arg<std::string_view>("path")));
3082+    const auto path_info{fs::status(path)};
3083     // Write to a temporary path and then move into `path` on completion
3084     // to avoid confusion due to an interruption.
3085-    const fs::path temppath = path + ".incomplete";
3086+    const fs::path temppath = fs::is_fifo(path_info) ? // If a named pipe is passed, write directly to it
3087+        path : path + ".incomplete";

ajtowns commented at 2:48 pm on March 5, 2026:

Seperating a ternary operator with a comment seems very strange, why not just:

0    // For regular files, write to a temporary path and then move into `path` on completion
1    // to avoid confusion due to an interruption. For a named pipe is passed, write directly to it
2    const fs::path temppath = fs::is_fifo(path_info) ? path : path + ".incomplete";

theStack commented at 6:43 pm on March 5, 2026:

Looks odd indeed, will take your suggestion if I have to retouch.

ajtowns commented at 3:12 pm on March 5, 2026: contributor

utACK 509d871fc00e3dd5a9c42e267da13a322b646d76

I don’t think this PR does “introduce dump_to_sqlite.sh” any more. Is there a way to tell if the streaming data you receive over the pipe is complete or not? Checking coins_count from the metadata equals the number of coins you actually received, I guess?

[EDIT: The PR title references the sh script still. I don’t think it makes sense to have a shell script to do that; the utxo_to_sqlite.py script should just make the fifo and invoke bitcoin-cli itself if we want to make things easier]

theStack renamed this:
~~rpc: allow writing UTXO set to a named pipe, introduce dump_to_sqlite.sh script~~
rpc: allow writing UTXO set to a named pipe
on Mar 5, 2026

theStack commented at 6:44 pm on March 5, 2026: contributor

Thanks for the reviews! Removed the shell script mentions from the PR title and the description, and updated the usage example (now being a “two liner”, including the pipe creation via mkfifo).

Is there a way to tell if the streaming data you receive over the pipe is complete or not? Checking coins_count from the metadata equals the number of coins you actually received, I guess?

Yes, I think that’s in fact the only way. Detecting this for a named pipe is not different from an actual file though (which could also be truncated and thus be incomplete): consumers should read for exactly the number of coins specified in the metadata (in the utxo_to_sqlite.py script, this is done with a for loop over num_utxos). If in the course of that any of the reads return zero bytes (indicating EOF), the data is incomplete [1]. If after the loop an extra read after returns not zero bytes, then there is unexpected extra data.

[1] I guess we could improve the utxo_to_sqlite.py with a read() wrapper that prints a nice “incomplete dump data” error message in that case, rather than continuing with an empty byte-string and likely running into an AssertionError or IndexError (or similar) at some point…

rkrux commented at 10:18 am on March 6, 2026: contributor

ACK 509d871fc00e3dd5a9c42e267da13a322b646d76

$ ./contrib/utxo-tools/utxo_to_sqlite.py /tmp/utxo_fifo ./utxo.dat

In PR description, should it not be s/utxo.dat/utxo.sqlite as it is the outfile?

theStack commented at 10:29 am on March 6, 2026: contributor

$ ./contrib/utxo-tools/utxo_to_sqlite.py /tmp/utxo_fifo ./utxo.dat

In PR description, should it not be s/utxo.dat/utxo.sqlite as it is the outfile?

Good point, fixed. (The example works with either file extension, but .sqlite is clearer indeed)

in src/rpc/blockchain.cpp:3172 in 23244283ac outdated

3168@@ -3167,7 +3169,7 @@ static RPCHelpMan dumptxoutset()
3169                                         path,
3170                                         temppath,
3171                                         node.rpc_interruption_point);
3172-    fs::rename(temppath, path);
3173+    if (!fs::is_fifo(path_info)) fs::rename(temppath, path);

fjahr commented at 11:08 am on March 6, 2026:

nit: Could use brackets

in test/functional/tool_utxo_to_sqlite.py:139 in 509d871fc0 outdated

135@@ -135,6 +136,19 @@ def run_test(self):
136             assert_equal(muhash_sqlite, muhash_compact_serialized)
137             self.log.info('')
138 
139+        if platform.system() != "Windows":  # FIFOs are not available on Windows

fjahr commented at 11:10 am on March 6, 2026:

nit: Could still run the test on windows too, to make sure this case falls back to the standard behavior like it should and doesn’t crash.

theStack commented at 4:20 pm on March 11, 2026:

Executing the same test with a regular file instead of a named pipe would very likely fail, as two processes would try to access that file concurrently. Let me know if I misinterpreted and you had something different in mind. If yes, it could be done in a follow-up (e.g. bundled with the proposed utxo_to_sqlite.py script change).

fjahr commented at 10:12 am on March 12, 2026:

Sorry for my superficial comment here. On the surface I just observed that there is a special handling for the windows case in the test (just ignoring it) but not in the RPC code and that felt off to me because I thought there might be a way to still hit this code path. But I was working on some assumptions that were slightly off and didn’t check them until now. After diving more deeply, here is what I found: There are named pipes in Windows, but the semantics are quite different so the comment on this line at least isn’t fully correct. But it would be impractical trying to cover this because mkfifo doesn’t support the windows semantics yet. This is seems fine because filesystem is_fifo does to not return true for these windows semantics either afaict (didn’t check all possible implementations). So the code falls back to using a windows system named pipe as a normal file. My idea when posting the comment was to cover this case because I didn’t think about that this might be working differently and annoying to implement (there seem to be ways but probably not worth it). So yeah, covering it in the functional test is harder than I thought and probably not worth the extra work.

However, maybe the follow-up should handle the case where a windows named pipe passed as the path arg is handled explicitly and a good error message is returned. Afaict this should be possible by checking (on windows only) if the path starts with \\<anything>\pipe\<name>. In that case we should tell the user that this isn’t supported on windows because otherwise the RPC will likely fail in some weird way later, when trying to move the file if not earlier when writing to it.

This is all fine for me to ignore for now because I don’t know how commonly these named pipes are used on windows so it seems like a pretty special case and the new release notes explicitly say already that this only supported on UNIX-like systems 👍

theStack commented at 3:07 pm on March 12, 2026:

Thanks for elaborating and providing the background information. Admittedly, my simplified assumption was “Python doesn’t provide os.mkfifo on Windows, therefore named pipes don’t exist at all there” :sweat_smile: In light of your findings, I agree it’s worthwhile to look into this:

However, maybe the follow-up should handle the case where a windows named pipe passed as the path arg is handled explicitly and a good error message is returned. Afaict this should be possible by checking (on windows only) if the path starts with \\pipe<name>. In that case we should tell the user that this isn’t supported on windows because otherwise the RPC will likely fail in some weird way later, when trying to move the file if not earlier when writing to it.

fjahr commented at 11:14 am on March 6, 2026: contributor

utACK 509d871fc00e3dd5a9c42e267da13a322b646d76

Agree with prior comments that utxo_to_sqlite.py should probably just use this automatically in a follow-up or otherwise it would be good to add a hint in the documentation.

sedited commented at 7:59 am on March 11, 2026: contributor

This could also get a small release note. I’m sure we’d re-ACK quickly if you also want to address the outstanding nits.

fanquake added the label Needs release note on Mar 11, 2026

rpc: support writing UTXO set dump (`dumptxoutset`) to a named pipe

This allows external tooling (e.g. converters) to consume the output
directly, rather than having to write the dump to disk first and then
read it from there again.

Co-authored-by: Luke Dashjr <luke-jr+git@utopios.org>

2e8072edbe

test: add test for utxo-to-sqlite conversion using named pipe 61a5460d0d

doc: add release note for #31560 (named pipe support for `dumptxoutset` RPC) b19caeea09

theStack force-pushed on Mar 11, 2026

theStack commented at 4:21 pm on March 11, 2026: contributor

Added a release note and addressed the nits #31560 (review) and #31560 (review). For the two nits that I didn’t address, see the comments above (https://github.com/bitcoin/bitcoin/pull/31560#discussion_r2919486878, #31560 (review)). Thanks for the detailed reviews!

sedited approved

sedited commented at 7:55 pm on March 11, 2026: contributor

Re-ACK b19caeea098f92a7f72aaeee49573358f4b153a3

DrahtBot requested review from rkrux on Mar 11, 2026

DrahtBot requested review from ajtowns on Mar 11, 2026

DrahtBot requested review from fjahr on Mar 11, 2026

ajtowns commented at 8:49 pm on March 11, 2026: contributor

utACK b19caeea098f92a7f72aaeee49573358f4b153a3

fanquake removed the label Needs release note on Mar 12, 2026

fanquake merged this on Mar 12, 2026

fanquake closed this on Mar 12, 2026

fanquake added this to the milestone 32.0 on Mar 12, 2026

fjahr commented at 10:12 am on March 12, 2026: contributor

utACK b19caeea098f92a7f72aaeee49573358f4b153a3

EDIT: Post-merge I guess ^^

theStack deleted the branch on Mar 12, 2026

rpc: allow writing UTXO set to a named pipe #31560

Code Coverage & Benchmarks

Reviews

Conflicts

LLM Linter (✨ experimental)