From: Eric Voskuil <eric@voskuil.org>
To: Bitcoin Development Mailing List <bitcoindev@googlegroups.com>
Subject: Re: [bitcoindev] Re: Reducing RAM requirements with dynamic dust
Date: Wed, 10 Dec 2025 12:44:33 -0800 (PST) [thread overview]
Message-ID: <97065ca0-662b-4a5d-90c5-7607db41ba2fn@googlegroups.com> (raw)
In-Reply-To: <CAJowKg+6==q=Wa4LRWRHdiOfcV6RPtYdCzVjY+JJrmEgbR6c2A@mail.gmail.com>
[-- Attachment #1.1: Type: text/plain, Size: 16411 bytes --]
Hi Erik,
It's a good idea, that's why we did it.
Libbitcoin is a set of libraries, libbitcoin-database being one of 4 that
make up node (system, network, database, and node). (libbitcoin-server is
an additional library that adds comprehensive set of client-server
interfaces, making the node useful.) libbitcoin-database is an
implementation of a simple query interface (defined as a c++ class) over a
backing store. The store is a templated collection of tables. The tables
are mmap-based head and body files used to construct multimaps, hashmaps,
arraymaps, arrays, and blobs. We mock the tables using
std::vector<uint8_t>, mock the store using the the mocked tables, and test
most of the query interface using the mock store. The structure is highly
relational, and surrogate keys are exposed to the caller for optimized
navigation.
This is an isolated and clear storage abstraction layer. All validation is
performed within the chain:: classes (e.g. block, header, tx, input,
script, etc.) defined in the base level libbitcoin-system. There is no
validator coupling to the store. The store retains chain objects, indexes,
and validation state for headers/blocks and txs as they progress through a
state machine. There is no utxo table, just natural relations between
objects, indexed and related. Validation correctness of course requires
store fidelity, but is totally decoupled from it. We validate blocks
concurrently, queueing up 50,000 blocks at a time by default (e.g. with 50k
threads we would validate all at the same time).
The store could be replaced with no impact to the query interface (as we
already do in testing). So it's not really accurate to imply that
libbitcoin's validation is tied to mmap or even append-only. Pruning could
be implemented in the existing model for example. The existing store could
be replaced with something simple and light like SQLite, or a full RDBMS.
We had some interns working the SQLite approach last summer. That would be
more specialized for low performance scenarios, where the custom database
targets ultra high performance.
With sufficient RAM there is never SSD access. The store can sync up and
just live in RAM, never touching a disk. Since it is append only, it's very
low impact on SSDs. As the store builds, no table body byte is ever
re-written. Table heads are hashmaps buckets, small and dynamic. It
performs live automated/manual snapshotting, automatic fault
detection/recovery on restart, automated disk-full pause/restart, and
supports hot backup. Query performance is phenomenal. A warm node can query
the full 5.2 million output Electrum query (very complex relations) in 15
seconds on my 2.1GHz workstation. But at the low end, an off-the-shelf
store is sufficient. A clear interface and swappable store makes a lot of
sense.
A large utxo set makes no difference in this design. There are no
operational problems associated with it. Given the recent fuss about it, I
figured it was worth putting some detail behind it here. This is not
theoretical, at this point we are only working on server (client-server
interfaces). The utxo set size is a complete non-issue.
e
On Wednesday, December 10, 2025 at 2:48:43 PM UTC-5 Erik Aronesty wrote:
> Core: Keep UTXO lookups in RAM as much as possible. This gives predictable
> validation under adversarial load and trades memory for protection from
> pathological disk latency.
> Libbitcoin: Represent history in append-only, memory-mapped files. Let the
> operating system decide what stays hot. Accept more disk I/O in return for
> simpler concurrency and no explicit UTXO map.
>
> To create 200 million outputs as an attack, the cost is roughly 48 BTC
> based on eight vb per output at low feerates.
>
> Given the operational problems that a very large UTXO set creates, it may
> be worthwhile to develop a clear storage abstraction layer.
>
> The goal would be to ensure that validation correctness does not depend on
> the transitional behavior of a specific storage engine.
>
> Once the validator has a well defined storage contract, alternate backends
> such as a libbitcoin style layout could exist as storage formats while
> still benefiting from the reliability of the rest of the Bitcoin
> implementation.
> (I would be interested in something like this.)
>
> On Wed, Dec 10, 2025 at 10:34 AM Eric Voskuil <er...@voskuil.org> wrote:
>
>> > Given the increasing RAM requirements, due to the increasing UTXO set
>>
>> I have been assured multiple times recently that this problem does not
>> exist in Bitcoin Core. I'm not sure about that, but I can say for sure that
>> this is not a problem in Libbitcoin and is therefore not problem with the
>> Bitcoin protocol.
>>
>> Best,
>> Eric
>>
>> On Wednesday, December 10, 2025 at 1:12:31 PM UTC-5 uuowwpevskfcordh
>> wrote:
>>
>>> Given the increasing RAM requirements, due to the increasing UTXO set, I
>>> suggest seeing the UTXO set size as a controlled variable. A feedback
>>> mechanism sets a dynamic dust level, below from which UTXOs are
>>> removed/discarded and thus freeing RAM.
>>>
>>> Below is an overview essay better expressed by grok, which can also be
>>> seen in here:
>>> https://hackmd.io/P-2lzGb8TiC86IOE3OGiYA?view
>>>
>>> # Enhancing Bitcoin's Scalability: A PID-Inspired Approach to Managing
>>> UTXO Set Growth
>>>
>>> ## Abstract
>>>
>>> Bitcoin’s UTXO set is currently an unbounded accumulator that risks
>>> long-term centralization as node RAM requirements grow without limit.
>>> Existing fee incentives have proven insufficient against sustained
>>> low-value output creation (e.g., inscriptions, tokenized assets, dust-heavy
>>> protocols). This article proposes a soft-fork mechanism that treats UTXO
>>> set size as a controlled variable: a slowly rising target size is defined,
>>> and a PID-style feedback controller, updated every difficulty epoch,
>>> dynamically raises a minimum-value floor beneath which old UTXOs become
>>> unspendable. The result is bounded, predictable growth of the UTXO set with
>>> ample warning periods, no hard caps on monetary use, and strong resistance
>>> to bloat attacks—all while remaining fully compatible with a soft-fork
>>> deployment.
>>>
>>> ## Introduction
>>>
>>> Bitcoin, the pioneering decentralized digital currency, operates as a
>>> complex dynamic system governed by consensus rules that ensure security,
>>> immutability, and permissionless participation. At its core, Bitcoin
>>> maintains a distributed ledger known as the blockchain, which records all
>>> transactions in a sequence of blocks. Each transaction involves inputs and
>>> outputs: inputs reference previously unspent outputs from prior
>>> transactions, while outputs create new spendable units called Unspent
>>> Transaction Outputs (UTXOs). The UTXO set represents the aggregate state of
>>> all currently spendable coins in the network, serving as a critical
>>> component for transaction validation and wallet management.
>>>
>>> As Bitcoin has evolved, the UTXO set has grown significantly, influenced
>>> by increasing adoption and diverse usage patterns. This growth, while
>>> indicative of the network's vitality, poses challenges to its long-term
>>> scalability and decentralization. Nodes—computers that validate and relay
>>> transactions—must store and process the entire UTXO set in memory for
>>> efficient operation, which can strain resources such as random access
>>> memory (RAM). Unchecked expansion could lead to higher barriers for running
>>> full nodes, potentially centralizing control among fewer, well-resourced
>>> participants. This article explores a proposed mechanism to address these
>>> concerns: an adaptive control system inspired by
>>> Proportional-Integral-Derivative (PID) feedback principles, designed to
>>> impose bounded growth on the UTXO set while preserving Bitcoin's
>>> foundational properties.
>>>
>>> ## Motivations for UTXO Set Management
>>>
>>> To appreciate the need for enhanced UTXO management, it is essential to
>>> understand the current status quo and its vulnerabilities. Bitcoin's design
>>> prioritizes efficiency and security through mechanisms like block size
>>> limits, which cap the amount of data added per block (approximately 1 MB
>>> base size, expandable to about 4 MB with Segregated Witness). These limits
>>> help control the overall blockchain size, ensuring predictable hardware
>>> requirements for storage on hard disk drives (HDDs). Similarly, the
>>> difficulty adjustment algorithm maintains a consistent block production
>>> rate of roughly one every 10 minutes by dynamically scaling the
>>> computational challenge for miners based on network hashrate.
>>>
>>> However, the UTXO set lacks comparable built-in constraints. It
>>> accumulates as users create new outputs—often in small
>>> denominations—without a mandatory mechanism to consolidate or prune them.
>>> This can result from various activities, including high-frequency
>>> microtransactions, the embedding of non-monetary data (such as through
>>> protocols like Ordinals, which inscribe arbitrary information onto
>>> satoshis, Bitcoin's smallest unit), or the anchoring of sidechain or
>>> tokenized assets that leverage Bitcoin's security but operate externally.
>>> While these innovations expand Bitcoin's utility, they can inadvertently
>>> increase the UTXO count disproportionately to their economic value,
>>> elevating transaction fees during congestion and raising operational costs
>>> for nodes.
>>>
>>> Existing incentives, such as transaction fees, partially mitigate this
>>> by encouraging users to consolidate low-value UTXOs to avoid higher costs.
>>> Yet, these market-driven forces are insufficient against sustained patterns
>>> of low-value output creation, particularly when driven by external systems
>>> that do not bear the full cost of network maintenance. Over time, this
>>> leads to bloat: as of recent estimates, the UTXO set exceeds several
>>> gigabytes when loaded into memory, complicating node synchronization and
>>> validation. Without intervention, projected growth could undermine
>>> decentralization, as fewer individuals or entities might afford to
>>> participate fully in the network.
>>>
>>> The motivation for reform, therefore, stems from a desire to balance
>>> innovation with sustainability. An ideal solution would allow gradual UTXO
>>> expansion to accommodate genuine monetary usage while introducing feedback
>>> to curb excessive accumulation, all without compromising Bitcoin's
>>> permissionless nature or requiring a hard fork that could fragment the
>>> community.
>>>
>>> ## A PID-Inspired Feedback Controller for UTXO Control
>>>
>>> Drawing from control theory, which studies how systems maintain desired
>>> behaviors through feedback, this proposal introduces a dynamic mechanism to
>>> regulate UTXO set size. PID controllers, widely used in engineering for
>>> processes like temperature regulation or autopilot systems, combine three
>>> terms: Proportional \(P\) for immediate response to errors, Integral (I)
>>> for correcting persistent deviations, and Derivative (D) for anticipating
>>> changes based on trends. In Bitcoin's context, we adapt this framework to
>>> compute a minimum value threshold—or "floor"—below which UTXOs become
>>> ineligible for spending, effectively deprecating them over time.
>>>
>>> Importantly, this deprecation rule constitutes a soft fork: it restricts
>>> the set of valid transactions to a subset of those previously allowed,
>>> without introducing new capabilities. Upgraded nodes enforce the rule,
>>> rejecting deprecated UTXOs as invalid inputs, ensuring backward
>>> compatibility and minimizing disruption.
>>>
>>> ### Mechanism Overview
>>>
>>> The controller activates at each difficulty adjustment epoch,
>>> approximately every two weeks (2016 blocks), aligning with Bitcoin's
>>> existing periodic recalibrations for predictability.
>>>
>>> 1. **Define a Target UTXO Size Trajectory**: Establish an increasing
>>> maximum target for the UTXO set size, denoted as $T(t)$, where $t$
>>> represents the epoch number. This could grow sublinearly with time or
>>> blockchain height to reflect organic adoption— for instance, $T(t) = U_0
>>> \cdot (1 + r)^t$, with $U_0$ as the initial size and $r$ a small annual
>>> growth rate (e.g., less than 1%). This permits expansion while preventing
>>> unbounded divergence.
>>>
>>> 2. **Measure the Error**: At each epoch, compute the error $e(t) = S(t)
>>> - T(t)$, where $S(t)$ is the current UTXO set size (measured by count or
>>> aggregate data footprint).
>>>
>>> 3. **Compute the Floor Adjustment**: Apply the PID formula to adjust the
>>> value floor $F(t)$:
>>>
>>> $$ \Delta F(t) = K_p \cdot e(t) + K_i \cdot \sum_{k=0}^{t} e(k) +
>>> K_d \cdot \frac{e(t) - e(t-1)}{\Delta t} $$
>>>
>>> Here, $K_p, K_i, K_d$ are tunable gains, selected conservatively
>>> through simulations to ensure stability (e.g., using root locus analysis to
>>> avoid oscillatory or unstable roots in the system's characteristic
>>> equation). The floor $F(t)$ increases only if $e(t) > 0$, applying to UTXOs
>>> below this threshold.
>>>
>>> 4. **Deprecation Process**: UTXOs with values below $F(t)$ are marked as
>>> unspendable in future transactions. To provide fairness and predictability,
>>> implement a grace period (e.g., 4-12 months) during which owners can
>>> consolidate affected UTXOs.
>>>
>>> 5. **Safeguards and Tuning**: Incorporate clamps on $\Delta F(t)$
>>> (similar to Bitcoin's difficulty adjustment limits) to prevent abrupt
>>> changes that might cause network congestion from mass consolidations.
>>> Periodic reviews of gains and targets via community governance would adapt
>>> the system to evolving conditions.
>>>
>>> ### Stability and Dynamics Considerations
>>>
>>> In dynamic systems terms, the UTXO set can be modeled as an integrator
>>> accumulating outputs minus expenditures. The PID controller introduces
>>> negative feedback to dampen this accumulation, driving $S(t)$ toward $T(t)$
>>> with minimal overshoot. Simulations would verify robustness against
>>> adversarial scenarios, such as deliberate UTXO spam, ensuring the system's
>>> poles remain in the stable region of the complex plane.
>>>
>>> Potential benefits include reduced node resource demands, lower fees
>>> during normal operation, and enhanced decentralization. Risks, such as
>>> over-deprecation affecting small holders, can be mitigated through careful
>>> gain selection.
>>>
>>> ## Conclusion
>>>
>>> This PID-inspired approach offers a principled path to managing
>>> Bitcoin's UTXO set, fostering sustainable growth while upholding core
>>> principles. By integrating feedback control, Bitcoin can evolve as a
>>> resilient dynamic system, better equipped for widespread adoption. Further
>>> research, including detailed modeling and community discourse, is essential
>>> to refine and potentially implement such enhancements.
>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "Bitcoin Development Mailing List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to bitcoindev+...@googlegroups.com.
>> To view this discussion visit
>> https://groups.google.com/d/msgid/bitcoindev/049858f4-28ba-4ffb-8966-f8bef09035b7n%40googlegroups.com
>> <https://groups.google.com/d/msgid/bitcoindev/049858f4-28ba-4ffb-8966-f8bef09035b7n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/97065ca0-662b-4a5d-90c5-7607db41ba2fn%40googlegroups.com.
[-- Attachment #1.2: Type: text/html, Size: 18917 bytes --]
next prev parent reply other threads:[~2025-12-10 20:52 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-12-06 16:08 [bitcoindev] " 'uuowwpevskfcordh' via Bitcoin Development Mailing List
2025-12-10 18:30 ` [bitcoindev] " Eric Voskuil
2025-12-10 19:33 ` Erik Aronesty
2025-12-10 20:44 ` Eric Voskuil [this message]
2025-12-12 22:22 ` 'uuowwpevskfcordh' via Bitcoin Development Mailing List
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=97065ca0-662b-4a5d-90c5-7607db41ba2fn@googlegroups.com \
--to=eric@voskuil.org \
--cc=bitcoindev@googlegroups.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox